Compound Thinking

TG2 Status

Well, it’s official now, I have a new job with Predictix, doing open source TurboGears and Python web dev stuff. Predictix is very much invested in helping the TurboGears community to grow and thrive, and I’m proud to be working with their team. And I’m even more excited about the fact that they want me to do work on TurboGears 2 as part of my “real job.”

One of my main goals when looking for this job was to make sure that whoever I worked for was committed to growing and takeing care of the TG development community. And I couldn’t have asked for anything better. Working for Predictix will help me to polish up the good work that’s already been done to get us to a TG2 beta release, and they already have a lot of fantastic stuff that they would like to open source, which I’m really excited about.

TG2 is moving forward like crazy. In the last three weeks, we’ve had two sprints, both of which had several people working on docs, and on adding the last few features needed for the beta, and cleaning up the show-stopper bugs in our ticket system.

I’m a bit burned out by all the activity, but at the same time I’m very excited about where we are going. I think 2008 is shaping up to be a really busy year for the TG dev team. I see my job in the very short term as creating some stability and consistancy in the midst of the firestorm of new development that’s going on. So, my highest priority right now is getting a stable beta release out the door, and helping us to move forward the docs so that anybody who wants to try out TG2 has a stable base to work on.

My plan will be to do releases about once a month for the rest of the year (or until we have a TG2 final release), because there’s a lot going on, and I want to make that stuff available to people as soon as possible.

Some TurboGears Love

I just found this blog post about Paris Envies, and why Jon chose TurboGears for that project. I think we come out looking pretty good ;)

He was well aquanted with Ruby on Rails, and Django when he was introduced to TurboGears:

I was sceptical at first, being in love with Django at that time. TurboGears taught me a lot of things and without it I’m sure Paris Envies wouldn’t be where it is today.

His site is very cool and well designed:

http://www.parisenvies.com/

This site is a great example of what you can do with TurboGears, since it makes use of lots of TG features and add-ons. Widgets, JSON support, the user Registration module, and lots of other components are used. And the great thing is that he’s been able to rapidly add new features, and has been very happy with the flexibility of the TurboGears framework.

I made the Paris Envies mobile website in two days, no more. This included tests, integration of the WURFL mobile phone database (to get screen sizes) and Google Maps Static (I was using Yahoo Static Maps first, but google ones are much better ;)).

All in all, I would say this seems like a ringing endorsement of TurboGears.

I can code really faster with TurboGears than with any other framework, but when I say faster, I mean it.

And of course all this was done with TurboGears 1, which is still viable, still competitive, and still very much supported. Sure we’re working hard on TurboGears 2 to provide many more industrial strength solutions, and to make getting started even easier. But we’re also committed to maintaining and growing the TurboGears 1 platform at the same time. Unlike some other web-frameworks out there, we’re not abandoning 1.0 in favor of 2.0. Sure, we’ll eventually phase out TG1 support when people don’t want it anymore — but that’s a long way off, and don’t think shafting existing users is a path towards future success.

Threads, Processes, Rails, TurboGears, and Scalability

Threads may not be be best way, or the only way, to scale out your code. Multi-process solutions seem more and more attractive to me.

Unfortunately multi-process and the JVM are currently two tastes that don’t taste great together. You can do it, but it’s not the kind of thing you want to do too much. So, the Jruby guys had a problem — Rail’s scalability story is only multi-process (rails core is NOT thread safe), and Java’s not so good that that….

Solution: Running “multiple isolated execution environments” in a single java process.

I think that’s a neat hack. The JRuby team is to be congratulated in making this work. It lets Rails mix multi-process concurrency with multi-threaded concurrency, if only on the JVM. But it’s likely to incur some memory bloat, so it’s probably not as good as it would be if Rails itself were to become threadsafe.

I’m not sure that the Jython folks have done anything like this. And I’m not sure they should. It’s a solution python folks don’t really have. Django used to have some thread-safety issues, but those have been worked out on some level. While the Django people aren’t promising anything about thread safety, it seems that there are enough people using it in a multi-threaded environment to notice if anything’s not working right.

At the same time, TurboGears has been threadsafe, from the beginning, as has Pylons, Zope, and many other python web dev tools. The point is, you have good web-framework options, without resorting to multiple python environments in one JVM.

Why you actually want multi-threaded execution…

In TurboGears we’ve found that the combination of both multi-threaded and multi-process concurrency works significantly better than either one would alone. This allows us to use threads to maximize the throughput of one process up to the point where python’s interpreter lock becomes the bottleneck, and use multi-processing to scale beyond that point, and to provide additional system redundancy.

A multi threaded system is particularly important for people who use Windows, which makes multi-process computing much more memory intensive than it needs to be. As my Grandma always said Windows “can’t fork worth a damn.” ;)

But, given how hard multi-threaded computing can be to get right TurboGears and related projects work hard to keep our threads isolated and not manipulate any shared resources across threads. So, really it’s kinda like shared-memory optimized micro-processes running inside larger OS level processes, and that makes multi-threaded applications a lot more reasonable to wrap your brain around. Once you start down the path of lock managment the non-deterministic character of the system can quickly overwhelm your brain.

As far as i can see, the same would be true for a Ruby web server in Ruby 1.9, where there is both OS level thread support and an interpreter lock.

I’m well aware of the fact that stackless, twisted, and Nginx have proved that there are other (asynchronous) methods that can easily outperform the multi-threaded+multi-process model throughput/concurrency per unit of server hardware. The async model requires thinking about the problem space pretty differently, so it’s not a drop in replacement, but for some problems async is definitely the way to go.

Anyway, hats off to the Jruby team, and here’s hoping that Rails itself becomes threadsafe at some point in the future.

So many revolutions, so little time.

Tim Bray is blogging about “inflection points” in the uptake of various technologies.

Python get’s a very positive review:

Today you’d be nuts not to look seriously at PHP, Python, and Ruby.

So, the rise of the so-called scripting languages is one of the inflection points, but it’s not the only one.

He singles out web-framework development as one place where there’s a lot of stuff happening, and a lot of new “rails-like” frameworks are cropping up all the time. TurboGears will live or die in the context of a much larger web-development revolution, and we need to be prepared to make our way forward in the midst of that.

What comes after rails will not be a rails clone. It will learn the right lessons from rails, avoid the pitfalls of rails, but it will also need to carve out something new and better than rails. For RDBMS users, I think the key difference between TG and Rails is the power and flexibility of SQLAlchemy. We need to “sell” this better.

There are a lot of other revolutions coming according to Tim. And I do think we’re looking at big changes in terms of everything from programming language choice, to web-development tools, to end-user desktops, and data persistence mechanisms. We’re also just beginning to see what the world of high-end javascript and other “rich” internet applications is going to do to our view of end-user software.

He doesn’t even mention the rise of EC2 and the Google App Engine as sea-changes in the way we buy computational resources, and I think that’s going to have a huge impact.

In the end my prediction is that the way we develop applications will change more in the next 5 years than it did in the last 5, and it’s time to start getting our heads wrapped around these issues, or we’ll be left behind.

Authentication in TG2

Chris McDonough just posted a pretty extensive writeup of something he and Florent hacked up recently. Basically he helped us to put together some helpers that turn the repoze.who project into very similar authentication/authorization system to what we had in Turbogears 1.0.

The cool thing about this is that we get to share the stuff that makes sense to share, and yet maintain a backwards compatable API. We started a project (authority) which did the same thing, all on our own, but it’s much better to be able to share. ;) The Authority project turned out to provide a lot of useful bits and pieces which have since found homes, and that process will likely continue.

I’ve said before that I’m very impressed with the work that the Repoze folks have been doing, but I think this is a particularly good example of how it’s possible to collaborate in interesting new ways on top of a shared set of interfaces.

There’s a lot more to be done to make python web development work better, and I’m excited because we continue to move towards a world where innovating in one area doesn’t require building a whole new stack.

I think there are big benefits for users of having a healthy and diverse ecosystem.