Compound Thinking

Threads, Processes, Rails, TurboGears, and Scalability

Threads may not be be best way, or the only way, to scale out your code. Multi-process solutions seem more and more attractive to me.

Unfortunately multi-process and the JVM are currently two tastes that don’t taste great together. You can do it, but it’s not the kind of thing you want to do too much. So, the Jruby guys had a problem — Rail’s scalability story is only multi-process (rails core is NOT thread safe), and Java’s not so good that that….

Solution: Running “multiple isolated execution environments” in a single java process.

I think that’s a neat hack. The JRuby team is to be congratulated in making this work. It lets Rails mix multi-process concurrency with multi-threaded concurrency, if only on the JVM. But it’s likely to incur some memory bloat, so it’s probably not as good as it would be if Rails itself were to become threadsafe.

I’m not sure that the Jython folks have done anything like this. And I’m not sure they should. It’s a solution python folks don’t really have. Django used to have some thread-safety issues, but those have been worked out on some level. While the Django people aren’t promising anything about thread safety, it seems that there are enough people using it in a multi-threaded environment to notice if anything’s not working right.

At the same time, TurboGears has been threadsafe, from the beginning, as has Pylons, Zope, and many other python web dev tools. The point is, you have good web-framework options, without resorting to multiple python environments in one JVM.

Why you actually want multi-threaded execution…

In TurboGears we’ve found that the combination of both multi-threaded and multi-process concurrency works significantly better than either one would alone. This allows us to use threads to maximize the throughput of one process up to the point where python’s interpreter lock becomes the bottleneck, and use multi-processing to scale beyond that point, and to provide additional system redundancy.

A multi threaded system is particularly important for people who use Windows, which makes multi-process computing much more memory intensive than it needs to be. As my Grandma always said Windows “can’t fork worth a damn.” ;)

But, given how hard multi-threaded computing can be to get right TurboGears and related projects work hard to keep our threads isolated and not manipulate any shared resources across threads. So, really it’s kinda like shared-memory optimized micro-processes running inside larger OS level processes, and that makes multi-threaded applications a lot more reasonable to wrap your brain around. Once you start down the path of lock managment the non-deterministic character of the system can quickly overwhelm your brain.

As far as i can see, the same would be true for a Ruby web server in Ruby 1.9, where there is both OS level thread support and an interpreter lock.

I’m well aware of the fact that stackless, twisted, and Nginx have proved that there are other (asynchronous) methods that can easily outperform the multi-threaded+multi-process model throughput/concurrency per unit of server hardware. The async model requires thinking about the problem space pretty differently, so it’s not a drop in replacement, but for some problems async is definitely the way to go.

Anyway, hats off to the Jruby team, and here’s hoping that Rails itself becomes threadsafe at some point in the future.

So many revolutions, so little time.

Tim Bray is blogging about “inflection points” in the uptake of various technologies.

Python get’s a very positive review:

Today you’d be nuts not to look seriously at PHP, Python, and Ruby.

So, the rise of the so-called scripting languages is one of the inflection points, but it’s not the only one.

He singles out web-framework development as one place where there’s a lot of stuff happening, and a lot of new “rails-like” frameworks are cropping up all the time. TurboGears will live or die in the context of a much larger web-development revolution, and we need to be prepared to make our way forward in the midst of that.

What comes after rails will not be a rails clone. It will learn the right lessons from rails, avoid the pitfalls of rails, but it will also need to carve out something new and better than rails. For RDBMS users, I think the key difference between TG and Rails is the power and flexibility of SQLAlchemy. We need to “sell” this better.

There are a lot of other revolutions coming according to Tim. And I do think we’re looking at big changes in terms of everything from programming language choice, to web-development tools, to end-user desktops, and data persistence mechanisms. We’re also just beginning to see what the world of high-end javascript and other “rich” internet applications is going to do to our view of end-user software.

He doesn’t even mention the rise of EC2 and the Google App Engine as sea-changes in the way we buy computational resources, and I think that’s going to have a huge impact.

In the end my prediction is that the way we develop applications will change more in the next 5 years than it did in the last 5, and it’s time to start getting our heads wrapped around these issues, or we’ll be left behind.

Authentication in TG2

Chris McDonough just posted a pretty extensive writeup of something he and Florent hacked up recently. Basically he helped us to put together some helpers that turn the repoze.who project into very similar authentication/authorization system to what we had in Turbogears 1.0.

The cool thing about this is that we get to share the stuff that makes sense to share, and yet maintain a backwards compatable API. We started a project (authority) which did the same thing, all on our own, but it’s much better to be able to share. ;) The Authority project turned out to provide a lot of useful bits and pieces which have since found homes, and that process will likely continue.

I’ve said before that I’m very impressed with the work that the Repoze folks have been doing, but I think this is a particularly good example of how it’s possible to collaborate in interesting new ways on top of a shared set of interfaces.

There’s a lot more to be done to make python web development work better, and I’m excited because we continue to move towards a world where innovating in one area doesn’t require building a whole new stack.

I think there are big benefits for users of having a healthy and diverse ecosystem.

On Layoffs, “Jelled Teams,” and my new job status

I don’t think it’s possible to over emphasize the importance of developing teams in software companies. Software production is a group activity, and Brooks, Lister, Demarco and Weinberg all announced this same thing in various ways. And they have been saying it for a long time. The Mythical Man Month, The Peopleware Papers, and The Psychology of Computer were all written over a quarter century ago.

But, in spite of 30 years, and thousands of pages written, people still don’t get it.

Up until Monday I was working on a suite of fantastic applications for patient data and hospital management. I was part of a small, but very talented team of developers doing amazing things. Sure we had great tools like Python, Ext.js, TurboGears, etc. But the fundamental reason we were so successful is that we had a great team that worked together really well.

How do you know you’ve got a great team?

Chad Fowler’s very good (but very poorly titled) book My Job Went to India: and all I got was this book has a chapter explaining how before he became a programmer he was a musician, and someone gave him this powerful advice:

“Always try to be the worst player in the band.”

If you seek out people who are better than you, people who will push you — people who will make you grow just to keep up — you won’t have the option of stagnating. You either get better, or you get out. That’s how I felt on this team, and I know it’s how we all felt.

We knew each-other, knew our strengths and weaknesses, and we knew that together we could make things happen. We knew how to challenge one another, and if something went wrong, we would take action to fix it as a team.

Great teams are a huge asset

I can honestly say that it was the best team of people I’ve ever worked with, and we were producing products with real revenue attached — in other words, we were making money, and lots of it.

But as you can probably guess, after the acquisition came layoffs. And the team was chopped in half. Even though the team consisted of the best developers in the company, even though we were the most profitable part of the company, even though things were going amazingly well in our little corner of the world — we were still torn asunder by layoffs.

There may be some higher corporate logic behind this. And perhaps at some level it was the right thing to do, though I doubt it. I am convinced that Lister and Demarco are right is arguing (via peopleware) that a “jelled team” is an incredibly valuable resource for a company. They get stuff done, and get it done quickly, because they already know how to work together well. We cared about one another, cared about our work, and cared about our company, we were motivated, skilled, and we got things done. Loosing that is loosing a lot.

I got lucky

I was fortunate enough to be the newest member of the team, and was one of the folks that was let go. Why fortunate? Because I don’t have to sit around every day trying to get stuff done without the rest of the team, because I don’t have to be reminded every single day of what once was, but is no more.

And because I can take this opportunity to take a breath, look around and try to find a job which fits my long term goals. I don’t know exactly what I want to do yet, but I do know that I want to do something good for python, something good for the world, and something where I get to work with great people.

So if you’re interested, feel free to drop me an e-mail (mark.mchristensen@gmail.com).

Please hire these guys — they are amazing!

But more importantly there are some other great developers who are also looking for jobs. I’ve seen them perform miracles in C#, Java, Python, and any number of other languages. And I know them well enough to know that they would each be a huge asset to any company who’s looking to develop a great team.

One of the things you need the most in team development is someone who has a picture of how it could be better, of what’s possible, and the will to make it so. In addition to their technical skills, I know that’s something they all have. So, if you’re looking for one or more highly skilled software developers in the Atlanta area (or you’re willing to work with remote people) I can’t recommend them highly enough. Please send me e-mail if you’re interested and I’ll pass the info on to the right people.

Google Summer of Code meets TurboGears

The TurboGears community is proud to welcome the participation of six Google Summer of Code participants. These students have proposed very interesting projects, done a lot of research, and promise to make TurboGears and related projects better.

The goal of the Google Summer of Code project is to get students more involved in the Open Source community, and I’m very excited about getting to know this group of students better, and seeing them show their stuff. And of course I’m looking forward to all the things they promise to do, from building automated build and test systems, to creating a super-easy CRUD interface builder, and improving the performance of Genshi templates.

We also had a large number of student proposals which weren’t accepted for the GSoC project, some of which were very, very good. The competition was very tough this year, and I am hopeful that some of those students will get involved in the TurboGears community anyway.

• DBSprockets User Interface by Alberto Valverde González
• Documentation Production System by Bruno José de Moreaes Melo
• Genshi optimization & XPath rewrite by Marcin Kurczych
• ToscaWidgets Library for OpenLayers and jQuery by Sanjiv Singh
• Improving TurboGears2 support on Jython by Ariane Paola Gomes
• TurboGears Quality Assurance Initiative by Steven Mohr

As you can see the majority of these projects are about improving stuff that’s used in TurboGears, but are generally useful tools on their own. We’re committed to principle of re-usable components, and the GSoC proposals we received have shown that there are a lot of other people commited to that same approach.

I’m particularly interested in Stephen Mohr’s project, because I think having a reliable build and test system available so that we can easily determine if a change in one of our dependencies breaks things is critical in for a world of interconnected but independent components to be viable.

But most of all I’m very excited that Google chose us to participate and that we had such a great set of proposals to choose from. Thanks Google! and thanks everybody who wrote a proposal.