Compound Thinking

Conceptual Integrity, Django, Pylons and TurboGears

People keep asking me what I think of Adam Goma’s post about Pylons, Django and “Conceptual Integrity”. There have already been a couple of good posts from Ben, and Jonathan on the subject. And both of them have some interesting points.

But, I’ve been thinking about this a bit more, and I think it’s smart to start with a posture of humility. What can we learn from Adam’s critique? And how can we do what we do better?

It’s always hard not to just “react” to people who criticize your work, or the work of others who you respect. And I think Adam’s post does seem to have a slight odor of self-congratulatory rationalization about the way it singles out Pylons for critique, and has nothing but praise for Django. And that may be what Ben and Jonathan are reacting too. I think it’s highly unlikely that Pylons has 0% “conceptual integrity,” and Django has 100%. If the term means anything it has to be understood in terms of a sliding scale — filled with gray areas — in which every project lives. The Django project has a lot of components, and there are a lot of things working’s together, and some of them fit better than others.

But, this isn’t a critique of Django. It’s a defense of Pylons, and an explanation of what I see as the core concepts around which Pylons was built. Pylons relies on the WSGI specification, and that plus the idea of putting the user-template in control of the framework rather than the other way round, to provide the core structures around which everything revolves.

For example, WSGI encourages you to have multiple WSGI apps running together, so Pylons works to make it easy to maintain multiple application and request contexts in a single process without trouncing around on one another’s state. The Pylons framework is designed as a stack of WSGI middleware plus some helpers — which is all “put together” so to speak by the Pylons application template. This too is the result of the core conceptual structure provided by WSGI, which aims to create a large and varied world of application and middleware interoperability. But it’s also based on the other core conceptual structure in Pylons — putting the user code in control of the framework.

In Pylons it’s your application which sets up the framework stack, wires up the database connectivity, and sets up everything from the config, to preparing the template engine.

This provides lots of flexibility, and can be a very good thing because it lets you in on exactly what’s happening all through the framework. You don’t have to dive into the opaque depths of the framework to figure out how to add a new middleware component, or swap out one that Pylons provides — it’s all right there in the code generated when you started your new project.

But here’s where I think Adam has something to tell us. These core conceptual structures only work to provide “conceptual integrity” for people learning the framework if they are 1) easily visible, and 2) easily understood. You have to see the overarching structure, and see how they have impacted all the various choices that Pylons has made.

This is all very reliant to the TurboGears 2 project, as we’re working hard to bring together two systems together, in a way that maintains the conceptual integrity of TurboGears, and welds in a seamless way with many of the good ideas from Pylons. Every step of the way, we’re making choices about what we think will feel natural to TurboGears users, and what parts of Pylons to use, and what not to use. And part of what I personally am taking away from Adam’s post is that I have to work more to make the core concepts in TG2 clearer, more obvious, and to help explain how various components fit into that overall structure.

Documentation is important, and centralized control (like Django or Ruby on Rails) can make it easier to enforce the core concepts, and maintain conceptual integrity. That’s certainly another lesson to be learned.

But I’m convinced that conceptual integrity is possible even in a widely distributed development world.

After all, unix based OS’s have a lot of conceptual integrity as a system, but that any idea of central control has left the unix world decades ago. Instead it’s been bult up from lots of independent implementations of lots of different components working together, along clearly defined pathways (pipes, etc). And OS X and Ubuntu have been able to leverage that existing set of diverse open components in a way that results in clean, powerful, user oriented operating system experiences.

So, there’s evidence that you don’t have to to retreat from the open component model in order to achieve conceptual integrity — even when you’re working at a much, much larger software project scale than a simple web framework ;)

I think Adam’s essay, perhaps unintentionally, presents us with a false choice:

• either you can have an “open” component based system,
• or you can have conceptual integrity,

I want us to resist that false dichotomy by building great software that is both open, and conceptually integrated. It requires work, hard thinking about interfaces, and library choices, but it is possible, and I think ultimately more than worth the effort.

I do think the Unix analogy goes one step deeper. The idea of pipes is central to the way Unix tools have been built, and I think the equivalent in the python web framework world is WSGI. After all Ian Bicking tells us that WSGI is a series of tubes, and what are tubes but flexible pipes ;) More about that another day.