Compound Thinking

DjangoCon talk (part 3)

The third part of my DjangoCon talk was about how Django can become a more innovative place, or alternatively how it can stifle innovation in the future. This very much builds on parts one and two, which I blogged about over the last few days.

The “big idea” of this talk is that competition, and continual refinement are the two real drivers of design excellence, and both Django and Python are well served by having web frameworks that compete and that are continually undergoing a process of internal refinement and improvement.

Again, this notion is not unique to me, the book Innovation Happens Elsewhere fleshes out the specifics of how these two factors foster good design:

“Competition and collaboration contribute
to innovation and creativity in two distinct ways.
Competition works through diversity and selection;
collaboration works through refinement and improving.”

This means that we are best served by an python web ecosystem that’s both cooperative and competitive. Which is why I (a turbogears guy) am willing to take time out of my schedule to try to make Django a better framework.

And it’s why I’m so stuck on the notion of components, and reuse, because good component architecture helps us to collaborate at the level of the library, not just the level of the framework. If the file system abstraction code, or the Django orm, or whatever other components of django were reusable libraries, they would be candidates for cooperation, and continual refinement.

And if newforms (now forms) were a separate component, they could compete head-to-head with similar tools like ToscaWidgets.

But since they are locked up inside Django, that competition has to happen at the level of the framework. Which is fine, except that you loose the granularity, and specificity that comes from competing at the individual component level. For example, where’s the detailed comparison of ToscaWidgets vs Django’s new forms library? Nobody has written such a thing because nobody (except my friend Max) outside of Django considers the Django forms library to be a reusable component.

And at the same time that competition happens at the level of the framework, cooperation can’t really happen well at all, because Django’s no outside dependency policy precludes it in some important ways. Django users can use non-django components, and Django itself has borrowed ideas and even code from other projects (simplejson, etc). But there is no two-way flow of code, and no active cooperation with outside library developers.

So, now for the specific recommendations, and here since I have a bit more time, I’ll offer a few more specifics than I did in the talk itself:

1. Make the request object a proxy to the wsgi environ dict
2. Make django middleware and WSGI middleware more interchangable
3. Reconsider the no-outside dependency rule

Make the request object a proxy to the wsgi environ dict

In the past there has been some talk about making TurboGears, Pylons, and Django all share the same request/response object API’s. I’m not opposed to this, and the fact is that TG2 and Pylons both share WebOb as a request/response API now. But I think that WebOb has actually shown us that the critical thing is not that the Request/Response object API must always be the same, instead the key thing is that the WSGI Environ dictionary be the canonical representation of the data, so that it can be passed along easily to anything WSGI compliant.

Since WebOb is already pretty close to the Django request/response object API, it’s probably easiest to subclass the webob object and provide a backwards compatible API with the current Django implementation. But I think we’d all be almost as happy if Django re-implemented this, the key is to always have an up-to-date version of the environ that you can grab.

The big advantage of all this is that you can then use WSGI applications easily within a django view. For example Rum is a wsgi app that provides a django admin like interface for SQLAlchemy, and if the environ were up-to-date and available there’s no reason that RUM couldn’t be used inside a Django app.

WSGI Middleware and Django Middleware

There’s a lot of innovation happening in the WSGI middleware space, from web-based interactive debuggers, to memory-leak tracing tools, and web-based profilers, to an advanced transaction management system that handles two-phase commit, cross-database transactions, etc.

At the same time a lot of effort is going into making Django middleware that does the same thing, which isn’t nessisarily a valuable kind of effort — we should be trying to make better stuff than what’s there, not just duplicating existing stuff in a different framework.

And though I have not tested all this WSGI middleware with Django it should “just work.” So, in the short term anyone creating what might be widely reusable middleware should consider doing it as WSGI middleware rather than Django middleware because that will allow it to reach a wider audience.

But since WSGI middleware and Django middleware are essentially doing the same thing (pre and post processing a request) there’s no reason that we couldn’t write an adaptor that lets everybody use everybody elses stuff in a reasonable way, and while this isn’t particularly glamourous work it would significantly improve cooperation between python web develoers on another front.

The no-dependency rule

My understanding is that the no-outside dependency rule is designed to make installation easy, and a realization that easy_install is not always so easy. But this is changing, virtualenv, basketweaver, eggbasket, and other tools have made working around the limitations of easy_install a lot easier. You can now create your own package indexes, make reliable install scripts, etc.

There are also a number of people working in this space to make distributing multi-package installs easier. Ian Bicking’s new pyinstall package has a way to package tarballs with all the source distribution packages you want to ship.
And Alberto Valverde released the EggFreezer, which packages up all the source and binary eggs necessary to do an install into a single python install script.

Ian’s package skips binary eggs entirely, and something similar can be done using Kevin Dangoor’s Paver project. Here the idea would be to use paver to overide the “python setup.py install” command, to install not just django but all of it’s dependencies (which are conveniently packaged up in the same tarball as Django.