Django developer discovers WSGI…

What happens when django developers discover WSGI?

Mark Ramm wasn’t full of hot air when he talked about this at DjangoCon or blogged about it later. He was right, and I for one wish I had listened sooner.

Eric Florenzano

Good to know not everybody thinks I’m just a blowhard ;)

And I would also like to mention that what I asked Django developers to do was not to stop writing Django middleware — just to consider sometimes writing wsgi middleware, since it’s more widely reusable.

But, there are trade-offs everywhere, and if your middleware:

  1. isn’t helpful the the wider web world,
  2. or is way, way easier to write as Django middleware

Then don’t use wsgi to write it.

9 Responses to “Django developer discovers WSGI…”


  1. For me personally it’s not that I don’t know what WSGI is or what it can do, just that there’s a different conceptual setup when working with Django (see my comment on Armin’s recent blog post for a more detailed explanation: http://lucumr.pocoo.org/2009/1/6/opinionated-frameworks#comment-687).

    Also, a few of the things I’d theoretically use WSGI middleware for are things that are either forbidden or ambiguously covered by the spec, something which annoys me to no end :)

  2. James,

    I think you are missing my point entirely, and are thereby creating confusion rather than clarity.

    Django Apps are nothing like wsgi middleware — and of course you are aware of this because django has it’s own middleware.

    I would not write middleware to replace most apps. Nor would I try to write a django app to replace most middleare.

    Sure WSGI middleware can both act as an django app, and django middleware all in one reusable package that uses standard interfaces to interoperate with a whole variety of other apps.

    But that’s getting well beyond what I’m talking about. I’m talking about middleware like repoze.profile (linked above), Dozer (http://amix.dk/blog/viewEntry/19363), WebError (http://pypi.python.org/pypi/WebError/0.10.1). All of which are fantastically useful, and all of which can help Django developers debug and optimize their apps right now with no changes to their code or to Django.

    These kinds of things should not be written as django middleware, because that just locks them away in djangoland away from the rest of us. That’s all I’m saying.

  3. Well, let me expand a bit then, because a lot of this goes back to my
    “conceptual integrity” post from last year.

    And since you’ve mentioned it, let’s take debugging as an example:
    there seem to be several WSGI middlewares, like repoze.profile, whose
    main purpose is to show you what’s going on inside your application,
    but they have to do this from a distance, often using Python’s
    standard profiling facilities.

    But suppose I’m using SQLAlchemy as my ORM; simply seeing a profiler
    dump can be useful, but what I’d really *want* is something that knows
    about and integrates with SQLAlchemy’s logging facilities, which give
    me an incredibly detailed and relevant picture of what’s actually
    happening. Of course, now it’s not just a generic middleware anymore;
    even if I do it as a WSGI middleware, it’s still tied to the specifics
    of a particular component I’m using. If I switch away from that
    component I’ll need to switch my debugging tool, and if someone else
    likes my debugging tool but uses a different component, they’ll need
    to rewrite it to adapt.

    That situation — having debugging tools which, because they assume a
    specific component stack and take advantage of its specific APIs — is
    what you get with Django (which, just like SQLAlchemy, has debugging
    facilities you can tap into). And just as you’d have to swap or
    rewrite your debugging tools if they depended on SQLAlchemy and you
    switched away, you’d have swap or rewrite your debugging tools if they
    depended on the Django ORM and you switched away.

    All of which goes back to my original point last year about
    trade-offs: in return for using/assuming a certain component stack,
    you can often get easier/increased functionality or access.

    And I really think that conceptually this does come down to the
    different ideas of what constitutes an “application”; in WSGI it often
    seems that you’re not supposed to know (from the outside) how the
    application works internally (or you’re supposed to pretend you
    don’t), instead using the tools WSGI gives you and whatever
    action-at-a-distance inspection you can do with Python tools (or with
    out-of-band tricks like log files). In Django, by contrast, it’s
    perfectly normal for multiple applications to be a lot more
    promiscuous with each other, use application-level instead of gateway-
    or Python-level APIs, etc., etc.

    Which isn’t to say one approach is universally better than another, of
    course, just that there are different folks with different goals, and
    who are willing to make different sorts of trade-offs. Which is the
    way I’d like to characterize the whole Python web-dev world, since I
    think that’s the best way to summarize it :)

  4. Yea, middleware that knows about SQLALchemy and can display the queries generated in a page view, along with other debugging information would be good.

    And more importantly it would be good for both TurboGears and Pylons, as well as Werzerkrig apps, and WSGI apps that don’t use a framework at all.

    Django middleware that knew about SQLAlchemy and did the same thing would be entirely useless outside of django. And, since you still seem to be missing it, that’s the flipping point.

    As far as I can tell this has nothing to do with “conceptual integrity” or “what constitutes an app” or whether it makes sense to define interfaces and decouple the code on either side of that interface from one another. While those are all interesting topics, and points where I think we may end up disagreeing in substantial ways, they all move us away from my point.

    This is all about one thing — compatibility — which in this instance just means making your code usable to others.

    Sure there are trade-offs everywhere, but writing your SA debugger as WSGI middleware rather than django middleware seems to me an easy choice, it costs almost nothing and lets more people use the code. And that would still be true if Django used SQLALchemy rather than shipping their own ORM.

    Of course this is only meaningful if you consider helping others who don’t use Django to be a valuable goal. You’re right that this is not universal, sometimes nobody outside django will care about your middleware, and sometimes what you need to do will require messing with Django bits itself, and in both those cases it would be stupid to write WSGI midleware rather than Django middleware.

    And if it was easy to use the Django ORM outside of django, it would be equally useful for the Django querry debugger to be written as WSGI middlware, so that you could use it outside Django too.

    All of this is about Django either turning inward and focusing only on itself, or turning outward and spending time and effort to make the whole python web world better.

    Now, as for being able to make assumptions about what’s available, Django and TG are on the same side. We have middleware that assumes that SQLALchemy is there, that automatically wraps requests in a transaction. We have middleware that handles authentication/authorization using SQLAlchemy models defined in your application. We want to expose a full stack, and to make it easier to write reusable “site components” which means about the same thing as you do by “django apps.” So, you won’t get any argument from me on that front.

  5. Let me try one more time, because I think we’re talking past each other.

    Suppose I wanted to write a SQLAlchemy debug tool that would expose information about the DB activity in a given request/response cycle. I could do this as WSGI middleware, but only by not really doing “WSGI”; AFAICT I’d have to set up an out-of-band communication channel (a log file which SA knows about and writes to, and which the middleware knows about and reads from) to accomplish this, because WSGI doesn’t allow me to do anything else through its own interface.

    In fact, short of someone developing a dtrace equivalent for Python (which wouldn’t be a bad thing, mind you) and instrumenting all the components I’m using, pretty much anything of this sort will either have to invent some sort of side-channel communication WSGI doesn’t know about, or will be limited to less useful techniques like dumping profiler output.

    This is why I harp on the distinction between WSGI’s idea of an “application” and Django’s idea of an “application”; WSGI sets up an “application” to be, basically, a black box: environ mapping goes in, response iterable comes out, and that’s it. This is a useful thing, to be certain, but I don’t quite buy into it as a universal interface; there are still plenty of times when I want or need a more full-featured API, and I’m happy to make the trade-offs I need to get it.

    We seem to be in agreement on that being OK, so I’m kind of wondering what the problem is here; a lot of tools written by people who use Django don’t end up as generic WSGI components, true, but I think that’s precisely because the tools would either end up coupled to the Django stack anyway (by assuming particular components — the most popular debugging tool on this side, for example, actually reaches in and monkeypatches a few key classes inside Django), or because the authors want to use APIs or techniques that WSGI by itself doesn’t offer.

    In other words, I don’t think there’s an ignorance of WSGI, or of WSGI components (as implied by this post’s title), just an approach that doesn’t necessarily fit well into WSGI’s view of the world. And that’s not at all unique to Django; sooner or later pretty much everybody invents a plugin API so that they can deal with this sort of thing, and nobody seems to think that, say, Zine is breaking from the larger Python web community or locking things away in “Zine-land” because it has plugins (even when some of its plugins reinvent wheels already covered by existing WSGI middlewares).

    (Incidentally, you _can_ use the ORM on its own; you just have to call django.conf.settings.configure() first to supply the basic info it needs about your DB. You may also want to either import everything up-front, or also add the necessary settings to tell the ORM where all the models are, because reverse relationship descriptors don’t get set up until the related models are imported.)

  6. James,

    I think we’re making some progress now. TurboGears 2 and Pylons use something we’re calling “WSGI Components” which are not pure WSGI middleware in the sense you mean it. Routes for example has a pretty standard API to work with, but it’s (or something implementing that API) is definitely required for Pylons to work.

    Repoze.what handles all kinds of authorization stuff, and it provides both a middleware component that decorates the wsgi environ with information about the user, and her permissions, and a library of auth stuff that gets used in TG to work with user and permission stuff, and this works out great.

    So, we’re on the same page with that. And I do think that there’s a relative ignorance of the good stuff that’s available via WSGI middleware in the django community. And people are discovering it all the time.

    Equally, a notable python web developer outside the django community recently told me “there’s probably nothing worth steeling there” about Django, which betrays a clear lack of understanding of many of the good things in DJanog.

    Oh, and on the incidental front, I know that you can use the django ORM on it’s own, and I don’t think I’ve ever said otherwise. It’s just not terribly clean to do. And yes that means I think django.conf.settings.configure() is a hack, but at least it’s a nice hack, and I’m very happy that it was created.

  7. Also to be fair, there is some level of ignorance involved as well. Certainly on the part of James, which explains his arguments, but not everyone is so well-versed. I, for instance, came to Python by way of Django, and went a long time without learning about WSGI. Therefore, I developed a Django middleware for signing and validating cookies, which has absolutely nothing to do with Django, and could instead be implemented as WSGI middleware with no problems whatsoever. had I known about WSGI, I would’ve just done so, but I didn’t, so I wrote it for Django alone.

    Of course, I see other signed cookie WSGI middleware out there, so maybe mine isn’t even necessary in the grand scheme of things, but ignorance plays another role here. Lots of people are like me, coming to Python through Django, and may not be aware of WSGI middleware or how to find them. having Django-specific middleware that duplicates other WSGI middleware is an easy way to at least get the functionality in their hands without having to get into a discussion of the virtues of WSGI over other systems.

    I agree that we shouldn’t limit middleware to Django when it can just easily be written for WSGI generally, but I also see no problem with making middleware available both ways for ease of adoption.

  8. I think this is a very fruitful discussion. One thing that seems obvious is that James lives in a Django world and works on Django problems. I work in a Film Studio, and work on generic problems. Django gets in my way, WSGI helps. I am very familiar with the Django ORM, yet am very unhappy with the limitations it offers.

    Django is a great tool, just like many tools, ls, cat, sqlalchemy, etc. It just happens to be a walled garden built by people living in a walled garden, as such it will always be a certain kind of tool to its detriment or success. No amount of Jackass of the Year blog posts, claims of potential cures for cancer etc, will change this ever present fact.

  1. [...] lots of interesting discussion based on my last post about a Django developer discovering how rich the WSGI middleware ecosystem [...]

Comments are currently closed.