I haven’t gotten to know many people outside the engineering team in Dexter, but they are great guys.

There’s lots of good stuff happening here, support for bazar, mercurial, git, trac, and other options on SourceForge itself, improved feeds, and other API’s for getting at SF data, etc. But I’m only peripherally aware of all that at the moment because I was hired to work on “totally new stuff” which is written in Python.

What I’m working on

Our first new project is a site called FossFor.Us, and it was the vision for this site, and the team that is working on this and other new stuff, that sold me on the coming to work for Sourceforge. It’s written in Django, and it’s been my first really large Django project, and while the experience has been pretty positive, there have been a number of things that have renewed my commitment to TurboGears development — but that’s a blog post for another day.

The backstory to the FossFor.Us site is that open source project hosting providers (Sourceforge and it’s recent competitors) have traditionally been pulled in two very different directions by two very different sets of users:

• developers of open source software
• and people who just want to.

And that tension has held us back in the past, we have to serve everybody with the same portal, and it ends up not serving either community as well as it should. But since developers are the most vocal users, it’s been the second class of user that’s been most neglected.

These people are just looking to get things done, and don’t care about the “project” part of open source software, they are, at least at first, only interested in the “product.” In many ways the Free and Open Source Software community has not served these people well.

Fossfor.us is in it’s first incarnation an attempt to create a window on the free software world, that’s just about finding and using software. But in a larger sense it’s an attempt to help us as a community to connect with potential users better.

I think connecting FOSS geeks and users is actually important

It’s important because people aren’t aware that there are free options, and are paying for software they can’t afford. There’s a prototypical user (based on a real person) that we talk about a lot, who’s a single mom, has an old laptop, and struggles week to week to pay her bills, but who bought Photoshop, because “that’s how you edit photos.” Her family could have used that money to more productive ends, but because she needed to edit photos, and didn’t know about the free alternatives all those opportunities are just lost.

Of course the same thing is true of small business owners, who could use free software to reduce their “overhead” costs, and actually spend money on creating things people love. Free software has the potential to lubricate the wheels of the economy, encourage entrepreneurial activity, and enrich people’s lives.

All of this is to say I think fossfor.us is a way to serve the world by making the product of all the open source developer’s labor more easily available and more accessible to real people. And when my mom actually used it to find some software a couple weeks ago, I knew we’d done something right.

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency.

Unfortunately multi-process and the JVM are currently two tastes that don’t taste great together. You can do it, but it’s not the kind of thing you want to do too much. So, the Jruby guys had a problem — Rail’s scalability story is only multi-process (rails core is NOT thread safe), and Java’s not so good that that….

Solution: Running “multiple isolated execution environments” in a single java process.

I think that’s a neat hack. The JRuby team is to be congratulated in making this work. It lets Rails mix multi-process concurrency with multi-threaded concurrency, if only on the JVM. But it’s likely to incur some memory bloat, so it’s probably not as good as it would be if Rails itself were to become threadsafe.

I’m not sure that the Jython folks have done anything like this. And I’m not sure they should. It’s a solution python folks don’t really have. Django used to have some thread-safety issues, but those have been worked out on some level. While the Django people aren’t promising anything about thread safety, it seems that there are enough people using it in a multi-threaded environment to notice if anything’s not working right.

At the same time, TurboGears has been threadsafe, from the beginning, as has Pylons, Zope, and many other python web dev tools. The point is, you have good web-framework options, without resorting to multiple python environments in one JVM.

Why you actually want multi-threaded execution…

In TurboGears we’ve found that the combination of both multi-threaded and multi-process concurrency works significantly better than either one would alone. This allows us to use threads to maximize the throughput of one process up to the point where python’s interpreter lock becomes the bottleneck, and use multi-processing to scale beyond that point, and to provide additional system redundancy.

A multi threaded system is particularly important for people who use Windows, which makes multi-process computing much more memory intensive than it needs to be. As my Grandma always said Windows “can’t fork worth a damn.” ;)

But, given how hard multi-threaded computing can be to get right TurboGears and related projects work hard to keep our threads isolated and not manipulate any shared resources across threads. So, really it’s kinda like shared-memory optimized micro-processes running inside larger OS level processes, and that makes multi-threaded applications a lot more reasonable to wrap your brain around. Once you start down the path of lock managment the non-deterministic character of the system can quickly overwhelm your brain.

As far as i can see, the same would be true for a Ruby web server in Ruby 1.9, where there is both OS level thread support and an interpreter lock.

I’m well aware of the fact that stackless, twisted, and Nginx have proved that there are other (asynchronous) methods that can easily outperform the multi-threaded+multi-process model throughput/concurrency per unit of server hardware. The async model requires thinking about the problem space pretty differently, so it’s not a drop in replacement, but for some problems async is definitely the way to go.

Anyway, hats off to the Jruby team, and here’s hoping that Rails itself becomes threadsafe at some point in the future.

Python get’s a very positive review:

Today you’d be nuts not to look seriously at PHP, Python, and Ruby.

So, the rise of the so-called scripting languages is one of the inflection points, but it’s not the only one.

He singles out web-framework development as one place where there’s a lot of stuff happening, and a lot of new “rails-like” frameworks are cropping up all the time. TurboGears will live or die in the context of a much larger web-development revolution, and we need to be prepared to make our way forward in the midst of that.

What comes after rails will not be a rails clone. It will learn the right lessons from rails, avoid the pitfalls of rails, but it will also need to carve out something new and better than rails. For RDBMS users, I think the key difference between TG and Rails is the power and flexibility of SQLAlchemy. We need to “sell” this better.

There are a lot of other revolutions coming according to Tim. And I do think we’re looking at big changes in terms of everything from programming language choice, to web-development tools, to end-user desktops, and data persistence mechanisms. We’re also just beginning to see what the world of high-end javascript and other “rich” internet applications is going to do to our view of end-user software.

He doesn’t even mention the rise of EC2 and the Google App Engine as sea-changes in the way we buy computational resources, and I think that’s going to have a huge impact.

In the end my prediction is that the way we develop applications will change more in the next 5 years than it did in the last 5, and it’s time to start getting our heads wrapped around these issues, or we’ll be left behind.

I’ve created a page on the Wiki for Sprint Organization tasks: http://docs.turbogears.org/SprintOrganization

If the sprint goes well it could get us very, very close to the point where we could reasonably do a TurboGears 2 technology preview release.

The main things we’ll need to do is sync up with the latest pylons, improve our tests, and do some basic doc organization work (migrate some information from the doc strings to the docs wiki, and create some pages which link to external docs). There’s lots of other things I want to do like improve out user authentication/authorization/registration support, or create a toolbox tool for helping create SQLAlchemy models. So, there’s tasks for anybody who can lend a hand.

If you’re available and want to help out (either attending physically or virtually), feel free to add your name to the wiki.

Likewise, if you can host a in-person sprint in another location, please feel free to edit the wiki with that information.

I loved the part about the database as an “implementation detail.” That sings to me.

–Daniel Waite

And, in spite of the poetic hyperbole of folks like Daniel, I think relational databases exist for good reasons and knowing about how they work is a good thing. Active record treats the database kinda like a big hash table in the sky, which is OK for some applications, but not so OK for others.

I find that large, complex apps have a way of growing in unexpected ways over time — and relational algebra is much better designed to handle unexpected requests for complex datasets than hash tables or even the most well designed objects. SQL, and relational algebra are implementation details in the same way that using a hammer is an implementation detail in building a tree house. Sure you can do without, but you will have a harder time, and will definitely end up building a different treehouse!

Don’t misunderstand me, Object’s are good things, and I sure as hell hope people understand them, use them, and make great software with them. But relational algebra is a good thing too. And it’s become clear that any really good database toolkit can’t forsake one in favor of the other, and that’s why SQLAlchemy, Storm, DejaVu, and Hibernate are all better tools than Rails’ Active Record.

As for Derik’s original article. In spite of all the brouhaha, Derek’s post basically says: “hey I liked Rails, and I learned a lot about good programming from it. And ultimately it was easier to get my site rewrite done in PHP because I learned rails first.”

I think it’s obvious that Rails could have done the job for Derek, but two things got in the way:

• everything was already in PHP so Rails wasn’t and incremental update, it was a rewrite.
• Active Record got in the way of using SQL as a relational algebra engine, rather than a glorified hash persistence mechanism.

TurboGears users should pay attention to both points. If you have a large complex app, you’ll be better off with an ORM that treats SQL as a full partner, which is why we’re migrating the defaults to SQLALchemy which does a fantastic job of making both relatinal algebra, and object oriented programming part of it’s core.

The second point is much harder to deal with, but if you can wrap existing application functionality in a larger TurboGears app, and migrate it piece by piece you’ll definitely have more success. I’ve been doing a lot of Web Service architecture stuff this year, and that’s proven to be a great way to connect existing resources to TurboGears applications.

At that conference one Ruby/Rails book author said something like; “I really like the vibe here. Everybody here is talking about making a difference, educating third world children, building better communities, and making the world a better place.” And he was setting this as a stark contrast to his experience of RailsConf where “everybody was talking about how much money they made.”

On one level it’s hard to fault the Rails conf people for this. Money does on some level validate what they’ve been doing, and it isgood to feel that kind of validation.

But, I think it’s fair to say that the Rails people have been too focused on proving something, and that has lead to a perception that many rails folks have a chip on their shoulder, or are totally focused on money as a metric of success. The extent that some elements in the Rails community have embraced this perception (as mentioned on Martin Fowler’s recent blog posting), is actually a bit frighting.

So, I think Chad is doing exactly the right thing by encouraging the Rails community to take a look at how they can improve their image. A couple days ago Chad posted a blog entry entitled “Change the world” and I hope that it makes an impact in the Rails community. I definitely see Chad Fowler many of the other members of the Rails community that I know personally as good guys, and I’d like to see the current negative stereotyping of the rails community come to an end.

I’d like to see the Rails community as a whole step up to the plate, and focus their attention and marketing efforts on new projects with altruistic goals. Turn the whold Buzz machine towards something that will really change people’s lives — replacing Java just isn’t a worthwhile goal when compared with helping to feed hungry people.

So, in the hope that a little good natured competition can spur things forward, I’d just like to mention that I think the Python community is beating the pants off of the rails community in the change the world department. Here are a couple of projects that I could think of off of the top of my head which the Python community is working on:

• Irrepressible.info — which is focused on fighting censorship
• the OLPC project — which aims to bring better educational materials to millions of underprivileged childeren around the world
• the Open Planning Project – which aims to provide tools for grass roots community improvement tools.

On another note, over the last 6 months or so Compound Thinking has been been working for Philantech on a project called PhilanTrack which takes a slightly different tack. The goal of the Philantrack project is to make it easier to manage the process of giving, receiving and reporting on grants.

The idea is that “greasing the wheels” of grant giving, and allowing non-profit organizations to focus more on doing the work, will make the nonprofits more efficient, more fun, and ultimately make the world a better place.

We may not think about it, but we geeks have a lot of individual power, and collectively we really can make the world a better place. If you’re working on a Rails or Python app that can change the world for the better, drop a comment here and let us all know.

Looks like Python and Ruby are both going to have a good number of talks. I’ll be talking about SQLAlchemy, which is the best object relational mapper I’ve ever seen. There’ll be talks about Test Driven Development in Python, Enterprise Architectural Patterns for Python developers, along with lots of cool talks about Lean Software Development, the side benefits of Test Driven development.

You can still submit a talk proposal before November 30th, and you’ll get free room and board. I think it would be great to see somebody talk about Dabo and Desktop application Development in Python, and they seem to be missing any talk about OSX/Cocoa stuff, which I’m sure is because they haven’t had any proposals yet.

It would also be nice to see a good cross platform development with Mono talk…

I’m really excited by the opportunity to get developers of all kinds together and talk about how to be productive and learn more about the strengths and weaknesses of the various tools/frameworks people are using.

My client is not an IT person and he knows nothing about programming…. I started sharing [Ruby on Rails] with him…. About ten minutes into my presentation he interrupted me and said, “So, this could save my IT department, couldn’t it?”

This is from Dave Thomas’ blog, where he is quoting an e-mail where Eric Knapp just gushes about the possibilities that Rails brings to the table. It seems nice enough, and everybody is well intentioned. But, I must admit I have my doubts. First of all, it sounds like a thousand non-technical managers getting sold on some new technology without taking time to understand the real life benefits and risks.

But it goes further than that, I doubt that Rails will ever save anybody’s IT department.

Can it make IT departments more productive? Absolutely. But when a department is broken that’s not usually just because they have bad tools, it’s because the staff either aren’t interested in finding better tools, or they are crippled by organizational red tape, or they are too afraid to tell management that the current tools suck.

“Whatever they say the problem is, it’s always a people problem.”
– Gerald Weinberg

Rails, TurboGears, Django, Plone, and even Zope and EJB 3 all offer the hope of more productive web development. Better tools are important. But a growing, healthy, team can discover, evaluate and when necessary the fit between these new tools and their current projects is the critical factor in long term success.

Rails may appear to be the world saving framework today. But it is not the end of the road, people are hard at work inventing better tools (for example, SQLAlchemy is poised to become a far better Dynamic Language based ORM than ActiveRecord), and if your team can’t evaluate new tools as they come they will end up stuck when Ruby on Rails is surpassed by other tools.

If you Rails is the end of the line, rather than just a remarkable jump forward –which will itself be left behind by new tools in the future — you’ve been fooled by the hype machine.

By the way, since I’ve noticed a few people in the Rails/Ruby community have a thin skin, this whole post should in no way be understood anti-rails. The problem is not the technology it is the marketing and the the uncontrolled hype which doesn’t admit the limitations of the technology.

But, in the same way that Rails won’t save your IT Department, TurboGears will not save you, nor will Zope, or EJB 3.

You have to save yourselves by becoming a place where software teams have the freedom to learn, grow and experiment with new tools.

When developing code you should always choose readability over convenience. Code will be read many, many more times than it is written…

–Practices of an Agile Developer by Venkat Subramaniam Andy Hunt

This quote from Ruby advocate and Rails book author Andy Hunt, explains why I work in Python even though Ruby has anonymous blocks and slightly less historical baggage than Python.

I recently taught a couple of free introductory classes, one for Python and another for Ruby. People with little experience in either language, could quickly read Python code. But with Ruby — and Rails in particular — there is so much heavy use of the “power features” of the language — like dynamic classes — that code readability was severely compromised. New programmers were quickly confused and lost. There were lots of “where does this happen” questions. Dave Heinemeier Hanson carries most of the blame here: Rails is built on his philosophy that values “convention always trumps configuration” and “less code is better” so highly that other critical considerations like the value of locality and orthogonality play in making code readable. In my view this is a serious mistake. After all, Code is read more often than it is written!

Wherever possible methods should be understandable on their own. No matter how powerful dynamic classes and objects are, using them means there will always becode elsewhere in your system that changes the behavior of the code you write — that means you have to know about and understand all of that elsewhere just to understand — and debug — the object in front of you.
For example, Rails has this nifty feature where Database classes map to table names that are the plural of the class name. This auto-pluralization like other Rails hacks is cute and fun at first, and it certainly makes the surface of the code shiny and pretty, but it also creates a layer of indirection and “magic” that obscures what is happening under the covers.

Of course, sometimes you have to make hard choices and it’s worth sacrificing code readability for other concerns. I’m not disputing that, I’m just saying the code you write — if it’s used and maintained — is going to be read many, many more times than it is written. So, you ought to optimize for the code reading experience, and more than any other language I’ve used, Python is designed just that.

There is a lot of interesting commentary on this movie floating around out there, so I thought it might be worth while to try to gather some of that commentary in one place.

Overall, I think the message of the video is: The “Dynamic Language Frameworks” are all demonstrably better than their Java counterparts for web based application development. These frameworks allow you to make small incremental changes to the user interface as you test your ideas with real users.

And this conclusion is exactly right — user interface design is hard, and it requires the ability to iterate and deploy incremental improvements.
But before I move on to other people’s commentary, let me say a couple of things from my own perspective:

1. Plone rocks for a certain set of problems, which it can solve out of the box.
2. Sean uses TurboGears 0.8. Had he used the newly released 0.9 he would have been able to get authentication and automatic CRUD. Someone made his application with something like 9 lines of TG 0.9 code! :)
3. Rails and Django also have internationalization frameworks now too (For those of you where this is important, I think Django and TurboGears do internationalization better than Rails because of Python’s Unicode support).
4. His application is pretty trivial — a real test for these applications would look better for the TurboGears framework.

On the web people have been saying (I’ll do my best to be complete, and get the attribution right):

• He’s very kind to Zope/Plone, if you step out of the Plone box then the cliff is indeed there but if you stay inside Plone everything tends to look like Plone and act like Plone and that isn’t that useful. That said, I’m sticking with Zope. — Simon Lucy
• This Lessig-style presentation is mostly good, especially if you want to compare J2EE to web frameworks in general… it clearly shows how using any of these frameworks will drastically decrease your development time and lines of code, and generally make your life a lot more fun. — Jeff Croft
• I should point out, though, that his version of Django was a little outdated, and his impressions of it had a few inaccuracies. Django’s got excellent i18n support, and for a while now it’s been able to use it without a database. Still, as long as someone’s not using J2EE, that’s a win for all of us dynamic-language freaks :) — Jacob Kaplan-Moss

My complaint (and it is a minor one) is that Sean seems to get a lot of things wrong about TurboGears, Django, and Rails, so it’s not very good at providing insight into which of these frameworks is going to be the best fit for your particular needs.

Most of the problems I have with his analysis stem from the fact that these three “dynamic language frameworks” are growing and evolving very quickly.

TurboGears, Rails, Django, Plone, and Zope 3 are building the future today. I’m most connected with the future of TurboGears, and I have to say that the changes that are coming present a lot of exciting possibilities. If this is any indication of what is happening in Django, Zope and Rails, then the future of Python web development frameworks looks very bright indeed.

The class is scheduled for the afternoons of Saturday, March 11 and Saturday, March 25.

If you would like to enroll in the class all you have to do is:

1. Go to Compound Thinking’s course site
2. Click on “Ruby on Rails”.
3. Create an account (if you don’t have one already!).
4. When prompted for the enrollment key, enter “rubymi-222″.

For now at least, this class is Ann Arbor–In Person only.

The first 25 people to sign up get in, so even if you sign up, you might not get a seat, since I have to close enrolment manually when the class gets full, but if you do enroll, and we don’t have room, I’ll let you know as soon as possible.

My quest for a unified explanation of MVC lead me back to the original SmallTalk implementation which was codified in Smalltalk-80. But that adds another layer of complexity to the problem of describing MVC, because Smalltalk-80 controllers did things that are now taken care of entirely outside of all of the modern MVC frameworks. For example, the controller was responsible for dealing with keypress and mouse click events — and doing something sensible with them. Now days the OS, and drivers, and window-managers take care of all of that, leaving a Smalltalk-80 controller without much to do.

And that, I think, is one of the key reasons that MVC has mutated so much over the last dozen or so years. The original architectural pattern was created to — among other things — solve the problem of where to put all the event-interpretation code that is necessary when your code is directly responsible for dealing with user input events. Now that we have mouse drivers, windows management systems, and a whole layer of GUI abstraction code between our users and the events there isn’t anywhere near as much need to keep “controller code” partitioned off in it’s own corner.

Another key component of the early version of MVC in Smalltalk that is different than what we find in “modern versions” is the observer relationship between Views and Models. In the original design views either poll, or subscribe to, models and automatically change what the user sees whenever the model is updated. But at least with web based MVC is that it is no longer feasible (well, at least until Ajax came along). So, as a sort of natural evolution, Controllers gained responsibility for sending status updates first to the model after a user event, and the also for telling the view to publish that updated data to the user.

This is natural, and sort of sensible, because much of the complexity of Smalltalk-80 style controllers had already been abstracted away, so it’s not really that painful to add a bit of “glue code” to the controller. So, anyway you can trace the history of this evolution through Objective-C, and NextStep, to Cocoa, WebObjects and Struts. With the controller being seen increasingly as “the glue that holds models and views together” in addition to its original purpose as “the code that processes user actions.”

OK, so given that this evolution has taken place, how is that reflected in all of the new “full-stack frameworks” we are seeing today.

I’m only deeply familiar with TurboGears and Rails, and both of them allow you to use model objects directly from your View code, so they are somewhat more like Smalltalk-80 than their older brother Struts. But, rails goes further by making it trivially easy to call Models from your view — everything is even magically imported s all you need to do is call the objects. TurboGears has so such magic, so if you want direct access to your model objects you have to import them (which is a single line of code). But it is often easier to just have your controller pass a dictionary of all the data you need into the view when it is called.

This is also a clear reflection of the Rails “convention over configuration” philosophy informing David’s view of MVC, while Python’s “Explicit is better than implicit” philosophy informs Kevin’s interpretation of the “same” design pattern. It’s hard to say which is better, but my experience was that it was really, really, really easy to do easy things in Rails, but then I had to figure out “how the magic works” before I felt comfortable venturing out from there. TurboGears, on the other hand took a little longer to get started, but then the learning curve was comparatively easier for moving on to more complex stuff, because everything was visible and I knew exactly where to look to find out more about how each of the pieces worked.

Over on Loud Thinking, David Hanson creator of Rails, seems pretty frustrated with Bruce Eckel’s recent blog post about Ruby, Rails, Python and Tate’s Beyond Java book.

“I’m losing track of the ill-conceived comparisons, but I do know what’s astoundingly clear: Bruce Eckel doesn’t like Ruby, he doesn’t like the attention its getting, and he doesn’t like people such as Bruce Tate fueling that attention.”

Bruce isn’t a huge fan of Ruby, but he doesn’t slam it either. So he does have some critical things to say, but on the other hand, here are a few quotes where he praises Ruby and Rails:

Clearly Ruby is making important contributions to the programming world.

Rails approach isn’t the ultimate solution; there will be plenty of other problems that we need to solve on the way to making web development easy. But it represents a fundamental restart in the thinking process.

He does however criticize Tate’s Beyond Java for announcing Ruby to the be the successor to Java without considering Python in any depth. And from what I can tell this is a reasonable critique of that book.

I know that a while ago on the Ruby list there where some quotes from one of Bruce’s posts which were not favorable to Ruby. And there might some bad blood from those quotes. But that post is no longer available in it’s entirety, and the only thing left is a few out of context quotes on the Ruby mailing list.

So, as far as I’m concerned the Ruby folks should get over it, and start engaging Bruce in a reasonable dialog. Especially since his current stance is that Ruby is nice, but he’s not yet convinced that there’s enough benefit to switching to let go of the advantages (and yes there are some!) of Python.

Over on Artema.com Bruce Eckel reviews Beyond Java and is somewhat critical of the way that Bruce Tate jumps on the Ruby bandwagon without considering the merits of other languages, and python in particular. I agree that a book like Beyond Java ought to be looking more deeply into the future alternatives before declaring a “winner.”

And while Ruby has a lot going for it (blocks), so does Python (mature libraries). I hope there is room for both in the post-java-as-a-tool-for-every-conceivable-job world, because I really like both languages.

Bruce also brings some much needed perspective to the Rails hype:

I’m sure we will find that the Rails approach isn’t the ultimate solution … But it represents a fundamental restart in the thinking process.

In other words, Rails is light-years better than what came before, but it isn’t at all likely to be better than what comes next.

And as far as what comes next in the Python community, Bruce has this to say:

And [Rails] has caused, in the Python community, attention to Django, the development of Subway (although I don’t know how that one is faring), and the creation of TurboGears, which seems like a very good solution because it builds on best-of-breed existing pieces using a Rails-inspired approach.

TurboGears is my pick of the Python litter. And interestingly enough as far as TurboGears goes, the “rails-inspired” approach is less about API than marketing. Sure both frameworks are “full-stack” and both frameworks are “Ajax-enabled” and both frameworks include and easy to use ORM. But the syntax and developer workflow for the two projects is still pretty different.

But as for marketing, the Kevin Dangoor creator of TurboGears, learned a lot from the Rails folks about intelligent use of screencasts, and the need for a good looking web-site. But I also happen to know that both Kevin and David read the Creating passionate Users blog, which is a good source of “marketing” ideas for framework developers.

I think the most critical piece of marketing advice a framework designer could read is “You can’t out-spend or out-teach.” I think the key to developing mindshare in “the coming framework wars” is not necessarily having the best framework with the cleanest code, and the most features. It will be which framework can grab a users attention long enough to teach them something significant, so they can actually start using the framework.