Compound Thinking

Coupling Django Style

I wrote the first draft of this a long time ago, and I skipped it because tempers seemed high in some places. It seems like things have calmed down, and I think the points are still 100% relevant.

For those reading this remember, I’ve used Django to build things like http://fossfor.us and I have done enough Django surgery to know what I’m talking about.

Django Developers have said, over and over again:

Django is tightly integrated and loosely coupled.

On the face of it, this statement is paradoxical (more techncally, it sounds like an antinomy to me) coupling and integration are generally seen as ends on a spectrum. Perhaps the Django catchphrase is more than just marketing speak, and actually means something.

If so perhaps they are thinking that they are more loosely coupled than some, and more tightly integrated than others. I suppose you could put opinionated frameworks like Ruby on Rails on one end, and free-form frameworks like Pylons on the other, and Django sits somewhere in between.

But even that seems an oversimplification, Rails is a huge community and Rails users have lots of options, there are alternative template engines galore, and many other components have plugins which replace or seriously modify their behavior.

Lest you think I’m manufacturing this from the air, here’s a quote from a django proponent discussing the idea of loose coupling in Django:

Developers coming from Ruby on Rails or other extremely opinionated frameworks may be used to following their framework’s best practices to avoid fighting against a framework which feels that it knows your project better than you do, but with Django you’ll be back in the driver’s seat.

– Will Larson on “loose coupling” in django

I think this is only true, if it’s true, by degree. It’s easier to do radical surgery to the framework in Django than it is in Struts, though I’m not all that convinced that Rails is harder to change than Django. But really, that’s beside the point. The fact of the matter is that framework surgery is much harder to do in Django than Pylons, and I think that’s not a wild claim, but a verifiable fact.

I wouldn’t recommend decoupling from the Django ORM without an extremely compelling reason. It is the most coupled of all the subparts of Django, and certainly not trivial to replace.

– Will Larson on using SQLAlchemy with Django (emphasis added)

Using SQLAlchemy in Django is not really hard (but at the same time it’s not all that easy either). But, it’s the kind of thing you wouldn’t want to do unless you really had a good reason.

Why? Because there are an awful lot of Django components that are “tightly integrated” with the django ORM.

But before everybody gets too mad at me, let me turn it around, and say that I think:

• it is not a bad thing to value integration above coupling.

Another Quote from Will’s blog, shows that he’s aware that loose coupling isn’t the single core value of Django:

Django places value on loose coupling, but it isn’t the sole design principle either.

– Will Larson (in a comment here)

I would say that even more strongly:

Django should not place loose coupling above developer productivity.

Adrian made a good point after my talk by suggesting that developers need to get things done, and to make sites that work now — not create software that is perfect by some abstract standard of design.

And I do think that some of the helpers that depend on the Django ORM are significant productivity wins. And, removing the django ORM dependency in all of them would be both very hard, and totally not worth it.

“Tightly Integrated” has value, and sometimes that value trumps “Loosely Coupled.” Zope is pretty tightly integrated into the ZODB. Many TurboGears 2 plugins are going to be pretty tightly integrated into SQLALchemy. Others will not, and I have encouraged some folks to rewrite their TG2 plugins to make them into pure WSGI apps that don’t depend on TG2 at all. Determining what exactly you will depend on, and how tightly you will be coupled to that dependency requires thought, and ultimately has consequences.

I’m hard on the Django folks here because I think the “tightly integrated/loosely coupled” buzz phrase is actually detrimental to understanding how the trade-off’s work.

And there are trade-offs and those trade-offs mean that there isn’t and will never be one perfect web-framework which somehow magically isn’t subject to the down-side of any of the constraints and design trade-offs that we all have to deal with every day.

Which brings me to the other major point I tried to make in my talk about django. Encapsulation, orthogality, or loose coupling is to some extent enforced by package boundaries. It’s not so much that you can’t be tightly coupled to the internals of an outside package, but that it feels wrong. And it feels wrong because when you start making libraries you start defining public interfaces, and making decisions about what’s internal and what’s external. And that makes monkeying with the internals feel icky. But if everything is all in one package, it’s a lot easier and less ‘icky” feeling to just grab some internal bit and do what you need to, since less thought has been given to what’s public and what’s private.

Morality and Software Development Leadership

Power, dominance, and responsibility are hot button issues which hover over and around every action leaders take like ghosts. And those who ignore them — who wield power without thought, who see only the ends, and ignore the means — put their projects at risk.

I think it’s a truism that there is no single barrier to IT project success more powerful than bad management. There are dozens, if not hundreds, of ways in which bad management directly and indirectly decreases productivity, and undermines even the the possibility of success.

Foucault, is the twentieth century master of uncovering hidden power relationships, and anybody who’s read his masterwork Discipline and Punish will learn a variety of tools to uncover hidden power relationships, and these tools can be immensely valuable in unpacking what’s really going on when a situation “just feels wrong.”

It’s hard to quantify, but knowing this stuff can definitely help you avoid political landmines, and create a safer work environment. But, I still haven’t figured out how to make Foucault palatable and understandable to the average IT manager. ;)

But Naked aggression is easy to spot

On the other hand, you don’t need a sophisticated set of tools for unpacking power relationships to see what’s wrong with some teams. For example, I once worked for a manager who held daily motivational meetings, in which he enumerated every small mistake that any of us could make that might make him look bad, and let each of us know in no uncertain terms that we would be “terminated” if we were seen to have made even the smallest mistake. If we didn’t answer the phone with the right phrase, or try to convince customers that they wanted more expensive products than they needed, or failed to live up to any of thirty different (and sometimes contradictory) arbitrary rules, we were told we would be fired instantly.

Doing the wrong thing may pay off now, but it almost always hurts you later.

I left after less than 6 months, by the time a year was up, only 2 of 20 of the original people on the team were left. But, the manager received a commendation for “running a tight ship.” Things were looking good. Then again, a year later he was fired, because his team just couldn’t keep up, and everyone on the team was so busy not screwing up that the had no time or energy left for the important things they were hired to do.

Dominance games, and pure aggressive pressure helped him meet his daily and weekly goals, but killed him in the long run.

Project managers need to learn to wield power with a light touch.

It’s inevitable that there are power-relationships involved in the context of an important project. If you’re doing things right people are passionate about the project, people want to do the right thing, and people don’t always agree about what the right thing is.

It can be tempting to jump into these disagreements and make decisions — and it can speed up progress in the short term. But be careful, just like the manager who made his short term goals, but lost in the long run, you may end up killing people’s passion, and cutting off discussion before critical information is revealed.

Sure, there are times when a strong good push can help people avoid prolonged and useless discussions, but it’s too easy to take advantage power relationships to avoid difficult but important discussions.

There’s lots more to be said on the subject of power relationships in software development, but I think one of the key things we have to understand is that computer nerds participate in power struggles too, they just do it naively and instinctively. And that leaves us easy prey for those who do it with knowledge, talent, and finesse. And those people are out there and in places where 2 years is a long time, they can get away with it for a surprisingly long time.

P.S. I wrote this post in 2007, almost published it in 2008, and am just now getting around to publishing it. So, please don’t think this is about anything happening to me personally right now, but it is something I keep seeing “around town” and something I think we need to understand better if we are going to stop falling prey to those who understand power politics better than we do and wield them more aggressively, and end up ruining many good projects.

Serving Developers *and* Users at SourceForge

My last post might have left some folks thinking that we’ve been focused entirely on “end user” experience at SourceForge and have been ignoring the developer side of the equasion, if that’s you, and you’ve felt a bit left out, there’s very good news. In the 7 months since I’ve been here, there have been a significant number of developer focused changes at sf.net, and there are more coming before OSCON.

Hosted Apps

One of the biggest changes is the new Hosted Apps system. We’re an Open Source hosting company, and we want to provide some of the best tools available for Open Source developers, so it only makes sense to use Open Source tools to do it. So, we now provide two dozen applications that you can install and use to help develop and manage your project, including trac, mediawiki, dotproject, a microbloging system, phpbb forums, an app for brainstorming ideas, and lots more.

From the perspective of an open source project maintainer, I think the best part of this is that I don’t have to manage them, do upgrades, backups, or worry about downtime — there are other people responsible for all that.

Have it your way

Part of the plan here is to make SourceForge more modular, and to let project managers use the tools that make sense for them and for their project. I think we’re the only open source hosting solution that provides svn, git, hg, bzr, and cvs source repositories. And with Trac, you’ve definitely got a far more full featured bug tracker than is available in most other open source project hosting.

One of the other advantages is that if you want new features in SourceForge, there’s now a clear and obvious way to do it. For example if you want some new ticket tracking feature you can add that feature to Trac, and once there’s a new release, it’ll be added to your sf.net project for free.

How the Consume side fits in

If you saw yesterday’s post, you’ll remember this diagram.

We’ve created the summary page based on the idea that projects have data in various places, in SourceForge developed apps, in hosted apps, on freshmeat, and out on the web in various places. In fact, all the the data on the sf.net project summary pages and download pages is fetched from the existing php apps via feeds and public API’s.

We still have lots of work to do, but all of this means that we’re changing the way SourceForge works to serve our developers better. We’re giving up on the assumption that we can provide the one right set of tools for all open source projects, and we’re also trying to leverage and improve existing open source solutions rather than reinvent the wheel.

To be fair, lots of this stuff wasn’t available when sourceforge got started, so back then we had to do some inventing, but the open solutions have passed us in lots of areas, and we’re taking advantage of that.

Developers need users too

And that brings us full circle, we want to grow project communities. In the end this means serving two sets of people well, and it means bringing those two communities together, serving developers means they get the tools they need to make better software, and serving downloaders and end users means the overall community grows. But the real growth happens when the line between developer and user begins to blur, with non-developers triaging bugs, writing documentation, doing translations, and sometimes even becoming developers themselves.

TurboGears on Sourceforge

No, we’re not moving the TG2 hosting to SourceForge. Instead, Sourceforge is now using TG2 to display the front pages, project pages, and download pages for all projects. This comes with a new look for SourceForge, but more than that it’s the first step in a fundamental rethinking of what SourceForge does. We’ve been given the opportunity to focus on improving the experience of users of SourceForge hosted software. We’re not ignoring developers, and lots of good stuff is going on for developers, but this latest update is all about the users. We wanted to make downloading software from SourceForge faster and easier, because this will help projects attract and maintain users. And more than 9 out of 10 of page views on SourceForge are by end-users, which means that the vast majority of page views on the SourceForge site are now going through TG2.

It’s all about users.

The big user experience wins come from some heuristics that we’ve developed to guess the best file to download. We’re analyzing your browser’s user agent for information about your operating system. And when the project has not told us about their preferred download, we’re analyzing file names and other data to get our best guess as to which file is the most relevant for a particular operating system.

This means we can generally give you a direct link to the right version of the file, right on the project page, so there’s no need to browse through a complex array of pages, and links just to finally get to the file you came to download in the first place.

Now with TG2 goodness.

Under the hood, there’s a new TurboGears 2 app powered by MongoDB. We’ve put this all together very quickly, and there have been a couple of rough spots here and there. Fortunately, none of the rough spots were in the TG2 or the TG2 stack.

But we did have a couple of rough patches. For example, we couldn’t use a mongo replica pair for the master database and have slaves on each node. So, we chose to try running the site all against a single replica pair rather than to do master-slave everywhere. This was compounded with a coding issue that, we were trying to pull tons more data out of MongoDB than we needed, and we ended up saturating a 2 gigabit network connection between the mongo master and the local slaves. Which if you think about it is kind of amazing, since MongoDB was still only using a few percent of the CPU on the box.

At this point though, all those issues are resolved and we are very confident that we’ll be able to add more user-facing improvements to SourceForge project pages and make the download side of things even better.

TurboGears 2.0.1 now available on pypi

Since the beginning (nearly) of TurboGears 2.x development we use a private index to store all our dependencies and point people to it for installation. This was nice because it helped us control our dependency tree and make sure installation of our unstable software was as easy as possible.

Since we had released a stable 2.0 version, followed by a 2.0.1 bugfix, we really were lacking a real pypi install mechanism. This is now a thing of the past! You can install a full blown TurboGears2 environment in one simple command:

easy_install tg.devtools

and this will fetch everything using normal lookups from the pypi. I hope this will help people out there.

At the same time I’d like to ask for help in that domain: If you are using TurboGears2 and find out that some dependency that needs compilation is not found in a compiled state (either on pypi or on our index), please help us:

1. By sending a compiled version of the egg to me (florent aide, you’ll find me on the mailing list easily)
2. By pointing us the missing dependency (send a mail in the turbogears or turbogears-trunk mailing list)
3. Or even better: try to become a maintainer for the specific dependency, that you know will always be missing for your architecture, and let us know about that great news. The whole open source community will become better, your karma (the real one, not the one you can admire on ohloh) will bump-up and you’ll be forever remembered on endor.

Now this leaves us with one more thing to do: finish the web site for turbogears that is currently in a poor state. If you would like to help us on the engine, please let us know in the mailing lists or via private message. If you want to help-out with content, please let us know also and we’ll be more than glad to give you an editor account and some directions to get some content online in this new website.

Cheers all,

Florent.