Archive for December, 2009

Premature optimization

We all know it’s bad. But, programming for performance in reasonable ways is good. So, what’s the difference?

Sometimes we think we know that a piece of code is important so we spend some time optimizing it. And in the end it’s less clear, and less maintainable, and it turns out that our bottlenecks are all elsewhere.

But, sometimes we do know where bottlenecks are going to be, we’ve learned from experience, and we know what needs to be done.

We know that architecture determines performance, and architecture isn’t easily bolted on at the end of the project.

So we have a conundrum. We shouldn’t optimize yet because we don’t know where the bottlenecks will be. We shouldn’t wait to optimize because we can’t easily retrofit a good architecture on a complex system.

Some of the conundrum is only apparent — there’s a difference between architectural problems that need to be set up front, and the kind of low level micro-optimization that obscures more than it helps. But, sometimes these conflicts are real — how do I know if I need a multi-process multi-consumer queue system for PDF generation before we build the system and benchmark it? If you don’t need it, that kind of extra architectural complexity just obscures the bit of code that actually solves the problem.

Solving the problem by going meta

Perhaps the problem really is that we’re dumb and optimize the wrong things at the wrong time. The solution to that problem is to get less dumb. Which means that we ought to spend time optimizing “learning”, both within our project processes, and across projects.

Codifying this learning is what the Patterns of Enterprise Application Architecture book was all about.

And I think it’s great as far as it goes, and if you haven’t read it you should buy it now.

But there are a lot of patterns that I can identify from my last half dozen projects that aren’t covered in PoEAA, so it would be great to see a next generation of books and blog posts that cover the modern architectural trade-offs that you have to make, something that covers some of the paterns of the web.

Scalability via in HTTP, etags, caching, and load balancing (the whole RESTful services argument), networked async processing patterns, etc. Scaling to the public web levels requires a whole different set of architectural principles than scaling to the old “enterprise” levels did, and that knowledge seems very much in flux.

It would be great if it also provided some advice for those of us who’ve moved into what Neil Ford has called the world of the Polyglot Programmer, patterns for coordinating activities across language barriers in a sensible way. That’s part of the nature of modern web systems too.

How do we expand Open Source?

So, one thing which keeps comming up in a bunch of different areas of my life is how we can expand the ethic of Open Source development.

People want TurboGears to do more than it does, they want other open source projects to grow, they want new open source projects in specific areas, and they want Open Source like activity in other professions like nursing or construction.

I definitely don’t have the answers. But I’ve had this conversation with a lot of folks over the last couple of months, and some of them had some great ideas.

So, in the spirit of opening up a larger conversation about these issues, here are a couple of thoughts distilled from all those conversations.

Institutionalizing Open Source Values

It is of course possible to create cultural institutions around which money can be channeled into Open Source development.

And all the legal mechanisms needed to structure those institutions in the right way are available today.

But the trick it seems is to create the institutions in such a way that money is delivered in small enough amounts that individuals remain in control. Money is powerfully persuasive, but one of the keys to the current success of open source is that collective action is always purely voluntary.

But at the same time the money needs to come in large enough amounts to make a difference. People need to be able to support lives and families on the work they do advancing various projects. To the extent that this is reliable income, we can remove competing priorities, and developers will be able to devote themselves more fully to projects that advance the common good.

So, the key to making all of this work is going to be the “bureaucracies” we create to manage the flow of money. They need to be tuned properly to the nature of the work, stable enough to provide a level of personal security, and perhaps above all they need to be financially transparent.

Creating the right kinds of organizational structures will help us channel the right amounts of money to the right people, and creating the wrong kinds will create perverse incentives that pollute the whole system.

Most of what’s been happening so far in this direction are ecosystems of companies built around open source offerings. This has worked pretty well, but it’s clear that there can be conflicts of interest, and the nature of commercial ownership leaves even the best run companies vulnerable to sudden changes (acquisition of small open source companies by huge proprietary competitors is already a fact of life).

But, what seems more interesting to me at this point is the number of foundations that are being are created for popular projects or groups of popular projects, etc.

These institutions will continue to grow, but they have the potential to change the way projects are run, so I expect a lot of fits and starts as we mature.

Open Source for other Professions

With that thought in mind perhaps lawyers, doctors, and other professions already have a form of the Open Source ethic, which has grown up around large institutions, and functions to spread knowledge and advance the state of the art of those groups. These institutions work to create new knowledge, train practitioners, and they seem to work pretty well.

If you haven’t caught on already I think it might be fair to say that this sub-section of these professions is called “academics.” ;)

Of course the university system isn’t perfect, and it’s taken hundreds of years to evolve to it’s current state, but I think it does provide some insight into how we might evolve larger institutional presences around open source, not in the next few years, but in the next few decades.

Rule Mongo with an Iron Fist

At we’ve been using MongoDB on various projects for the last six months or so. We finally re-factored out our MongoDB related code and created a new library. It’s battle tested on the project pages of, and it’s getting a workout in my new project (no details on that yet).

250px-Max_von_Sydow-Ming_the_MercilessOne thing we learned is that you have to enforce some stuff on the Python side of the world in order to make working with totally schema free data stores easier.

So, just like in Flash Gordan, we’ve got Ming to rule Mongo.

Rick’s blogged about it at length here:

And all I can say, is that I’m very happy with Ming as it is now, and it’s meeting our needs quite well. We are of course open to suggestions, and we’re hoping that we can open up more stuff as we go.

If you want to see the details, they can be found on the (of course) site:

If you’re looking for a larger exampel of Ming in actin Chris Perkins (coding machine that he is) and some other TurboGears devs are already looking at building a content management system with Ming and tg2:

If you use Ming, and like it, feel free to let us know.

Python Template languages (Part 1 — Django)

I’ve been thinking a lot about template engines in Python recently. Partly because’s new python code needed to choose a template language, and there were some questions about why we would choose one over the others.

But beyond that In the past few weeks used Genshi, Mako, Jinja, Django Templates, and Cheetah, and have been looking at, but not yet using out chameleon.genshi.

I figure all this promiscuous template library usage means that I should put my thoughts down somewhere. There are advantages and disadvantages of all these libraries, but I think that the choices are pretty clear once you know your constraints.

I’m not going to commit to covering them all in depth, but I’m going to try to put my thoughts about them down over the next few days.

For today let’s talk about the pros and cons of Django Templates. This is another post that has been developed over the last year or so, where typed stuff up while working on

Django made making easy in lots of ways. Want threaded comments? Add the existing app in a couple hours — Done! Want OpenID? Again add an app — Done!

But it also had frustrations, and one of the biggest for me was the template language.

Continue reading ‘Python Template languages (Part 1 — Django)’

Thinking about the Dip

I recently read Seth G’s book “the dip” which I’ve heard described variously as a book about choosing your battles, a book about quitting, or a book about mastery.

And it is about all those things. Because all those things revolve around a central idea:

sometimes things get harder before they get easier.

That “harder” is what Seth calls the dip.

And it is is difficult, but it’s also important because it makes being on the other side more valuable — after all not just anybody is willing to work their way through that dip.

For Seth, there’s obviously a lot about marketing, and reaching a wide audience. Obviously, it takes work to develop a new skill, grow a network of helpful people, or sell a new product. But given that our economy is “dipping” and that many of us have personal dips to work through, I’m finding Seth’s message very timely and relevant at the moment.

Getting through the dip is about the choices you make

If you’re like me and the people I know in southeast michigan, you are actually facing multiple “dips” at once. We are struggling our way through problems at work, financial problems, interpersonal problems, and we’re trying to figure out what to do about all those things.

One of the core ideas in “The Dip” is that you have to choose which battles you will fight, or you’ll loose them all. Some battles beg to be lost. You have to give up fighting on some fronts in order to win on others.

So, to go back to yesterday’s post, you have to say No to some things, in order to make it through any of the dips. You can’t have everything, you have to choose. And once you choose, you have to make it happen.