There’s definitely more to come in terms of releases and code, but I thought it’d be worthwhile to start with a quick run through of the tech stack and a bit of a description of what we’re doing.

Our first rule for libraries and tools on the new forge, was that we needed to use open source everywhere. Partly this is just because having the freedom to look at the code and modify it where we need fixes, makes it’s the easiest and best way to develop software. Partly it’s because we’re an open source code hosting platform, and we want to use what we promote. But perhaps most importantly, it means that we’re not prevented from sharing our work with others, or from inviting others to work with us in the future.

At the same time we had a company wide decision to standardize on the technology stack that we’d used in the “consume” project last year. So, we’re using:

• Python,
• TurboGears,
• MongoDB,
• and AMQP (RabbitMQ).

The combination of these means that we have:

• a huge number of libraries available to us,
• a web framework that we can turn into a plugin framework for projects and the tools they want,
• a schema-free database that lets us easily version documents to keep history on wiki pages, tickets, and other “artifacts” within the new forge
• a scalable system for handling asynchronous tasks, and propagating update notifications

The choice to use Python has been particularly valuable, since there are (literally) dozens of libraries that we were able to use to help us with everything from encrypted cookie sessions, and mongodb drivers, to markdown text processing, and syntax highlighting.

We’re still in the early days and have a lot more to do, but the goal is an open extensible, system that supports open source projects, and ultimately encourages more people do download and use a wider variety of open source applications.

The first thing we did was create some new pages for downloads. Recently we releases a new service designed just for open source project leaders who want to use sf.net as a directory and downloads service.

But, we’re also aware that one of the most important services we provide is project hosting. For the last several months a small group of us have been trying to bring sourceforge.net’s tools into 2010. And now we’re releasing an early preview of those new developer/community tools:

We have a long way still to go, but every long journey begins with a single step, and today’s step is allowing you to try the new forge, to create new projects at:

https://sourceforge.net/register

Where you can go to get a new project, with our new tracker, wiki, git, svn, and other tools. Projects can have subprojects, and links to other tools hosted off site, along with the many features that sf.net brings (free web hosting, hosted apps, etc).

But, why do all this?

In 1999 SourceForge was cool.

It provided all the tools that an open source project needed to get going, from cvs hosting, to bug tracking, and e-mail list support.

They pioneered free free software project hosting, and helped to transform the software development culture from one which barely new about free software or open source, to one where nearly everybody I know uses open license software. Oh sure, some of them might not know it, but they have it on their phones, in their TVs, their wireless routers — not to mention all the websites they use everyday that run on open source.

But, time passed.

More alternatives came out, more projects (including my own) started self hosting, and the landscape of open source software development changed. SourceForge.net took a long time coming out with support for new tools like svn, and then git.

Still, SourceForge has a special place in my heart. Partly it’s nostalgia, I suppose, but I still think:

• the core mission is still right
• and there is still a real need

We (Open Source developers) still need tools like git, mercurial, and svn hosting. We still need bug trackers and mailing lists. And in a meeting of other open source project leaders last fall, nearly every single one of them identified the time wasted integrating and administering these tools as one of their most important frustrations.

Not enough…

But, for many sourceforge.net and other free project hosting services were just not good enough, they weren’t scriptable, the weren’t extensible, their data wasn’t portable, and so they felt like they had to take on that cost.

And I fundamentally believe that open source projects live an die by communication, and that sourceforge.net can do something new by integrating the various kinds of “conversations” that happen around the project. We can integrate mailing lists and forums, we can integrate SCM and ticket trackers, etc.

New and improved

So, a couple of us have been quietly working on something new. The new forge is designed around a few core ideas:

• that data should be portable (every project gets their own database, which they can take with them if they want),
• that the open source community ought to be able to extend and enhance the tools they need,
• that integrating and cross linking the various kinds of conversations that open source projects need to have ought to be easier.

So, what we’re announcing today is more of a commitment to getting there on all these things, and a commitment to the “release early, release often” project management strategy.

So, expect us to take your feedback and make things better. Expect us to release lots of small fixes, and expect a few places where things are broken/incomplete because we value feedback more than polish at this point.

Meanwhile Gerry Weinberg, who wrote several books that I love, and gives lots of great advice, including the some of the best advice I’ve ever read about how to give advice, says “every problem is a people problem.”

So, which is it?

Are bad things that happen the result of bad processes, are they the result of things people do?

I’ve been party to a bit of discussion about this in the last month or two, and in the end it’s all pretty silly.

Processes are created by people, implemented by people, and are designed to accomplish the goals of people.

People run processes!

So, whenever something is broken, it’s people who will need to find the problem and fix it.

People can and do think of ways to improve processes everyday, but I’ll eat my shoe if you find a process that thinks of a way to improve people.

But there’s still a HUGE problem.

Modern companies seem to have a persistent failing — they look for people to blame when something goes wrong — and ignore the context in which those problems happened.

When something goes wrong, fire some people, and replace them with new people who make the same mistakes all over again.

Sometimes you “get lucky”.

The company might get lucky and find a person who’s able to raise awareness, reveal the larger contextual problems, and succeeded in spite of the fact that everything’s stacked against her.

More often than not though, the poor new guy doesn’t see the systematic pressures that caused everything to fall apart, at least not until it’s too late.

Sometimes replacing what’s broken isn’t enough.

Sometimes it’s the equivalent of a mechanic replacing your car’s engine several times in a row, because it keeps burning up — without ever checking to make sure oil is flowing normally, and the cooling system is working.

The easy way out.

It’s often easier to blame people because they don’t “control” them they way they do the context. This blame game is as old as the hills, but definitely not as pretty.

Help people fix processes

The solution is to ask people to look for the systematic pressures, give them the tools to find them, and to empower them to change the way work gets done.

In the end, people will improve the processes, if they believe they are allowed.

Sometimes a design isn’t working because you think you can’t change the one element that needs to be changed.

–Ryan (via svn)

The same thing is true when you are designing the processes by which work gets done.

Sometimes we think we know that a piece of code is important so we spend some time optimizing it. And in the end it’s less clear, and less maintainable, and it turns out that our bottlenecks are all elsewhere.

But, sometimes we do know where bottlenecks are going to be, we’ve learned from experience, and we know what needs to be done.

We know that architecture determines performance, and architecture isn’t easily bolted on at the end of the project.

So we have a conundrum. We shouldn’t optimize yet because we don’t know where the bottlenecks will be. We shouldn’t wait to optimize because we can’t easily retrofit a good architecture on a complex system.

Some of the conundrum is only apparent — there’s a difference between architectural problems that need to be set up front, and the kind of low level micro-optimization that obscures more than it helps. But, sometimes these conflicts are real — how do I know if I need a multi-process multi-consumer queue system for PDF generation before we build the system and benchmark it? If you don’t need it, that kind of extra architectural complexity just obscures the bit of code that actually solves the problem.

Solving the problem by going meta

Perhaps the problem really is that we’re dumb and optimize the wrong things at the wrong time. The solution to that problem is to get less dumb. Which means that we ought to spend time optimizing “learning”, both within our project processes, and across projects.

Codifying this learning is what the Patterns of Enterprise Application Architecture book was all about.

And I think it’s great as far as it goes, and if you haven’t read it you should buy it now.

But there are a lot of patterns that I can identify from my last half dozen projects that aren’t covered in PoEAA, so it would be great to see a next generation of books and blog posts that cover the modern architectural trade-offs that you have to make, something that covers some of the paterns of the web.

Scalability via in HTTP, etags, caching, and load balancing (the whole RESTful services argument), networked async processing patterns, etc. Scaling to the public web levels requires a whole different set of architectural principles than scaling to the old “enterprise” levels did, and that knowledge seems very much in flux.

It would be great if it also provided some advice for those of us who’ve moved into what Neil Ford has called the world of the Polyglot Programmer, patterns for coordinating activities across language barriers in a sensible way. That’s part of the nature of modern web systems too.

People want TurboGears to do more than it does, they want other open source projects to grow, they want new open source projects in specific areas, and they want Open Source like activity in other professions like nursing or construction.

I definitely don’t have the answers. But I’ve had this conversation with a lot of folks over the last couple of months, and some of them had some great ideas.

So, in the spirit of opening up a larger conversation about these issues, here are a couple of thoughts distilled from all those conversations.

Institutionalizing Open Source Values

It is of course possible to create cultural institutions around which money can be channeled into Open Source development.

And all the legal mechanisms needed to structure those institutions in the right way are available today.

But the trick it seems is to create the institutions in such a way that money is delivered in small enough amounts that individuals remain in control. Money is powerfully persuasive, but one of the keys to the current success of open source is that collective action is always purely voluntary.

But at the same time the money needs to come in large enough amounts to make a difference. People need to be able to support lives and families on the work they do advancing various projects. To the extent that this is reliable income, we can remove competing priorities, and developers will be able to devote themselves more fully to projects that advance the common good.

So, the key to making all of this work is going to be the “bureaucracies” we create to manage the flow of money. They need to be tuned properly to the nature of the work, stable enough to provide a level of personal security, and perhaps above all they need to be financially transparent.

Creating the right kinds of organizational structures will help us channel the right amounts of money to the right people, and creating the wrong kinds will create perverse incentives that pollute the whole system.

Most of what’s been happening so far in this direction are ecosystems of companies built around open source offerings. This has worked pretty well, but it’s clear that there can be conflicts of interest, and the nature of commercial ownership leaves even the best run companies vulnerable to sudden changes (acquisition of small open source companies by huge proprietary competitors is already a fact of life).

But, what seems more interesting to me at this point is the number of foundations that are being are created for popular projects or groups of popular projects, etc.

These institutions will continue to grow, but they have the potential to change the way projects are run, so I expect a lot of fits and starts as we mature.

Open Source for other Professions

With that thought in mind perhaps lawyers, doctors, and other professions already have a form of the Open Source ethic, which has grown up around large institutions, and functions to spread knowledge and advance the state of the art of those groups. These institutions work to create new knowledge, train practitioners, and they seem to work pretty well.

If you haven’t caught on already I think it might be fair to say that this sub-section of these professions is called “academics.” ;)

Of course the university system isn’t perfect, and it’s taken hundreds of years to evolve to it’s current state, but I think it does provide some insight into how we might evolve larger institutional presences around open source, not in the next few years, but in the next few decades.

One thing we learned is that you have to enforce some stuff on the Python side of the world in order to make working with totally schema free data stores easier.

So, just like in Flash Gordan, we’ve got Ming to rule Mongo.

Rick’s blogged about it at length here:

http://blog.pythonisito.com/

And all I can say, is that I’m very happy with Ming as it is now, and it’s meeting our needs quite well. We are of course open to suggestions, and we’re hoping that we can open up more stuff as we go.

If you want to see the details, they can be found on the (of course) sourceforge.net site:

• http://merciless.sourceforge.net/tour.html
• http://sourceforge.net/projects/merciless/

If you’re looking for a larger exampel of Ming in actin Chris Perkins (coding machine that he is) and some other TurboGears devs are already looking at building a content management system with Ming and tg2:

http://bitbucket.org/percious/c5t/

If you use Ming, and like it, feel free to let us know.

But beyond that In the past few weeks used Genshi, Mako, Jinja, Django Templates, and Cheetah, and have been looking at, but not yet using out chameleon.genshi.

I figure all this promiscuous template library usage means that I should put my thoughts down somewhere. There are advantages and disadvantages of all these libraries, but I think that the choices are pretty clear once you know your constraints.

I’m not going to commit to covering them all in depth, but I’m going to try to put my thoughts about them down over the next few days.

For today let’s talk about the pros and cons of Django Templates. This is another post that has been developed over the last year or so, where typed stuff up while working on fossfor.us.

Django made making fossfor.us easy in lots of ways. Want threaded comments? Add the existing app in a couple hours — Done! Want OpenID? Again add an app — Done!

But it also had frustrations, and one of the biggest for me was the template language.

First the good

Django templates operate in a sandboxed environment. This is good because it limits the ways that designers can blow-up your web-app. In fact this seems to be an underlying design decision of the Django framework. It’s the reason accessing non-existent variables just returns empty strings, because as the Djanogo book tells me “you wouldn’t want a designer to be able to take down your website with a typo.”

I can respect that, and I definitely see the benefits of sandboxing template code, particularly in places where end-users might be allowed to edit templates.

Easy to learn, and easy to use
They also provide a very simple syntax, that’s very well documented. This makes it easy for designers to figure out.

All of this is good.

But there are some negatives

Django developers seem to love them, but I feel like there are some significant drawbacks to using django templates.

Not trivial to use outside of django.

You can do it, but it’s not easy, and you end up with a lot of Django loaded that you don’t need. Fortunately there’s Jinja which is somewhat Django template like, but which isn’t at all dependent on Django itself. So, if you’re not using Django, I think it’s sane to recommend that you use Jinja templates. And tomorrow I’ll blog more about why I think that Jinja is just a better version of Django templates.

Won’t let you follow the fail early philosophy of development

When you use Django Templates in Django, I have two major issues. THe first is the decision to always treat undefined variables as variables that contain an empty string. Generally “fail fast” is a good engineering principle, because it’s helps you find problems sooner, and makes debugging much easier.

Of course the Django guys are smart, and the knew about this so they planned for it.

Well sort-of, apparently it’s easy enough to make invalid variables return something other than an empty string by setting the TEMPLATE_STRING_IF_INVALID config variable to something else, which gets displayed if you access an undefined variable in a template (say you make a typo. And this does makes it easier to find template errorsmore quickly.

But it’s not really “failing fast.”

I want an all out failure, not a small difference in the rendered page.

With that said, this as faar as it goes, I think it is good. TEMPLATE_STRING_IF_INVALID should be a configuraiton option, (though perhaps with a better name) and people should turn it on in development and off in production. Pylons does this with it’s c variable, and I think it’s very helpful.

But what I want is the exception, not just some random other text inserted, so then I looked into what it would take to force django template rendering to stop silently suppressing exceptions, and it’s not that hard on first glance.

The template renderer depends on a special attribute silent_variable_failure being set to True on the exception, so all we would have to do is just disable this checking when the fail_fast config value was set to True and we’d be good to go.

Seemed like a good idea, but in reality — not so good.

Django’s admin will totally break if you do such a thing, and it turns out that even setting the TEMPLATE_STRING_IF_INVALID parameter makes these pages render quite badly.

I ultimately found this tidbit in the Django Template Docs:

While TEMPLATE_STRING_IF_INVALID can be a useful debugging tool, it is a bad idea to turn it on as a ‘development default’.

Many templates, including those in the Admin site, rely upon the silence of the template system when a non-existent variable is encountered. If you assign a value other than ” to TEMPLATE_STRING_IF_INVALID, you will experience rendering problems with these templates and sites.

Generally, TEMPLATE_STRING_IF_INVALID should only be enabled in order to debug a specific template problem, then cleared once debugging is complete

Oops, that’s exactly what I don’t want. In practice this was not as bad as I thought it would be, but it was not good, and it made my life harder because the time between making a typo in a template and finding the problem increased, which also increased the dificulty of finding the cause of the problem.

My other frustration with Django Templates

Beyond that, the fact that Django template sandboxing excludes all expressions pushes lots and lots of work into filters and up into the “views” which other frameworks call “controllers” and means that django views mix presentation and action logic even more than is common in other MVC like web frameworks (Rails, Pylons, TurboGears, etc.).

Having a full featured experssion language like python in templates has been very useful to me in capturing presentation logic in the template.

And at the same time creating anything like a “macro” or a “template function” (a bit of reusable template code) requires some slightly complicated python gymnastics.

I’m certainly no MVC purist, but I think that in the new post web-2.0 world where applications have HTLM frontends, rich javascript front-ends, and even flex/silverlight clients, it makes a lot of sense to keep presentation logic out of the “view/controller” and let that focus on handling HTTP requests, and marshaling the data required by whatever presentation technology (html, html+javascript, flex, rich-client + JSON, or whatever) you happen to use.

And I really want to be able to have code-reuse in templates without jumping through hoops. That’s part of what a template language is for, so I really don’t like giving it up.

Conclusion

With all of that said, if the benifits of Django (reusable apps mainly) are important to your project, you should use Django, and if you are using Django you absolutely should use Dango Templates. They aren’t bad really. It’s just that I’m used to the other options that do it better.

So, ultimately if you’re not already using Django, I think you should consider one of the other options, as I think they all do better on template code reuse, and allowing fail fast behavior.

More Soon

Tomorrow I’ll try to write a bit more about Jinja, but I think this post pretty much sumarizes why I think that Django Template users outside Django proper should really just use Jinja2.

And it is about all those things. Because all those things revolve around a central idea:

sometimes things get harder before they get easier.

That “harder” is what Seth calls the dip.

And it is is difficult, but it’s also important because it makes being on the other side more valuable — after all not just anybody is willing to work their way through that dip.

For Seth, there’s obviously a lot about marketing, and reaching a wide audience. Obviously, it takes work to develop a new skill, grow a network of helpful people, or sell a new product. But given that our economy is “dipping” and that many of us have personal dips to work through, I’m finding Seth’s message very timely and relevant at the moment.

Getting through the dip is about the choices you make

If you’re like me and the people I know in southeast michigan, you are actually facing multiple “dips” at once. We are struggling our way through problems at work, financial problems, interpersonal problems, and we’re trying to figure out what to do about all those things.

One of the core ideas in “The Dip” is that you have to choose which battles you will fight, or you’ll loose them all. Some battles beg to be lost. You have to give up fighting on some fronts in order to win on others.

So, to go back to yesterday’s post, you have to say No to some things, in order to make it through any of the dips. You can’t have everything, you have to choose. And once you choose, you have to make it happen.

Where David Allen lead me wrong

The first insight I had is that Getting Things Done (GTD) has steared me wrong. This is hard to say because I think it’s a great book with a lot of great insights. In particular the strong admonition to get everything you need to do down on paper has changed the way I live and think. That list includes all the major and minor commitments I’ve made, and having it out of my head and in a system that I trust reduced my stress levels imensely.

But I found that I still have a lot of stress. And I found that I was still thrashing back and forth between projects without making the kind of decisive progress on any of them that I wanted to.

Why wasn’t the GTD process enough?

Because, as I eventually learned the GTD “inbox processing” strategy as described in the book is broken. You are supposed to choose between three options for each input: do it, deligate it, or defer it. But really there’s a key fourth option that is the really important one if you’re life is anything like mine.

What is that all important forth option?

• Don’t do it. Say no. Avoid adding another commitment to your list.

Wait, there’s science behind this!

This is really critical because like any queue that’s processed by a limited resource (in this case my time and attention) filling it too full actually causes the system to break down. This process of breakdown even has a technical name that feels exactly right, it’s called “thrashing.”

Every programmer and computer user knows what this is like. Open too many apps at once, and your machine grinds to a halt, data keeps getting swapped out to disk and the rate at which the machine can process information goes down exponentially.

Ok, so know I knew the name for my problem. Naming it is good, but it’s not the solution.

So, how do I stop thrashing about and get stuff done?

At some point, i’m not exactly sure where, I had a realization that in life, just like a manufacturing line, or a software resource issue, there are two keys to preventing thrashing.

1. Avoid too much multitasking. A system spends time switching between tasks and is less efficent when time-windows for work are too small, and task-switching happens too often.
2. When multiple commitments are being made that require real-time or near real-time responses, you have to keep the task-switching window short enough that high-priority tasks can be scheduled immediately. This, of course, creates a tension with the first principle which suggests larger time windows between task-switching.

This is where Getting Things Done’s todo list system provided me with a lot of help. It helped me reduce my task-switching costs, and that’s allowed me to get a better balance on these two pressures.

Letting the work FLOW

I think there’s a very clear analogy with what happened at Toyota in the early days. Manufacturers had die presses which took a long time to change, so they would run them for a long time. This was great in that it helped keep the machine working and limited the downtime. But it also meant that there were long delays in the system since the switching time and the time for the long runs added up. Toyota decided that since they couldn’t afford more machines, they needed to figure out ways to reduce the cost of switching, which reduced total cycle time, made them more responsive to bugs (quality problems) and helped them to get more done in less time.

Reducing the cost of switching made it possible for the system to run differently, rather than batching up lots of work and pushing it through, it Toyota discovered that you could pull what you needed through the system just in time. This same thing works for time management, when you aren’t overloaded you can be more responsive to today’s needs, and you avoid the inevitable mismatches that come with long delays between request and response.

What happens when there’s still too much to do?

But, to come back to my main point, even when you’ve done everything you can do to reduce the task-switching costs, you still can have significant thrashing problems when the resource (that’s you or me) gets overscheduled.

Most projects I’ve worked on ended up in this situation at some point, where working harder stopped producing results because of schedule pressure and resource contention.

This is why saying no is a critical skill.

If you limit the commitments you make, you can provide rapid turn around on the commitements you do make, and everything runs much more smoothly because you’re thrashing less, and wasting less time on task switching.

Beyond the basics, I’ve discovered that one really critical notion is to have enough slack in your system to handle emergencies. If you schedule the system full, any high-priority, high-urgency task that enters the system can break the whole process.

Without slack in the system emergencies snowball…

I know this from experience, as I’ve had my share of emergencies in the last couple of years, and when there was slack in the system things settled down quickly, and when there wasn’t I ended up with new emergencies caused by the first emergency delaying things just a little bit too much.

So, when you are tempted to take on another commitment, think about what would happen to your life if you had to take a week off to deal with a death in the family, or to help a sick relative. If you don’t see any path to recovery, perhaps you’re over-scheduling your most important resource — you.

When No, really means Yes

In the end I discovered the biggest irony of time management: it’s only when you say No to some things, that you have the ability to say Yes, and make real commitments.

When you say no to helping a friend move, or to visiting family, or whatever feels valuable to you you will feel terrible. But, if you don’t say no sometimes to at least some of these things, you’ll end up not being able to do any of those things anyway, because you’re always behind and never quite able to fulfill the commitments you make.

I learned the hard way, hopefully you don’t have to ;)

I agree, but I think that any discussion power politics is pretty risky, particularly since we use authority, power and force pretty interchangeably in many situations. And it’s hard to nail down exactly what people mean because there’s some confusion even at the level of what the words mean.

So, from my perspective (heavily influenced by Hannah Arendt here’s a reasonable definition of terms:

• Power: group action in service of some goal
• Authority: some kind of status “given” to an individual, which gives her influence over the actions of others.
• Force: the ability to make somebody do something they would not do on their own
• Violence: the act of causing of physical, social, or economic damage to an individual or group.

( * Arednt fans should note that I am using force differently than she does.)

Power and Authority in Open Source

Open Source projects have powerful people, in that they have people who somehow manage to get a group of people to work in common on some larger goal. And there’s often some sort of formal authority which resides in the BDFL or project leaders. Authority often centers around who has commit access and who can flip the commit bit. But there’s very little force or violence involved, and the authority that we give is always tentative and revokable in the case of an “unfriendly” fork of the community.

The fact that there’s often not much money going on reduces the chance of economic violence and removes the main way that developers encounter force. So, nobody can force you to work on stuff you don’t want to, and nobody can force you to stop working on the things you really want to see done.

Open source is Force/Violence free

At least that’s how it often seems. But I there can be significant social pressures to work on one thing rather than another, and many people often feel like they have a right to tell you what you ought to be working on.

None of this counts as force, but it does seem to indicate some imbalances in the system, where there are producers and consumers of open source and the producers seem to often feel like they can never do enough to please all of their consumers.

And because there’s a lack of resources going to the producers, it’s consumers are often frustrated by the slop pace of development. But all of that is another post for another day.

P.S. A Science Fiction book recommendation

Ursula K. LeGuin wrote a great book that I’ve been thinking a lot about when talking to people about the subject of the difference between “commercial” software culture, and the somewhat anarchistic Open Source culture. The book The Dispossessed, is one of my favorite Science Fiction novels, and and though it came long before “Open Source” or even “Free Software” even existed as such, I think it has a lot to say about the strengths and limitations of the “Free Software” culture.

For those reading this remember, I’ve used Django to build things like http://fossfor.us and I have done enough Django surgery to know what I’m talking about.

Django Developers have said, over and over again:

Django is tightly integrated and loosely coupled.

On the face of it, this statement is paradoxical (more techncally, it sounds like an antinomy to me) coupling and integration are generally seen as ends on a spectrum. Perhaps the Django catchphrase is more than just marketing speak, and actually means something.

If so perhaps they are thinking that they are more loosely coupled than some, and more tightly integrated than others. I suppose you could put opinionated frameworks like Ruby on Rails on one end, and free-form frameworks like Pylons on the other, and Django sits somewhere in between.

But even that seems an oversimplification, Rails is a huge community and Rails users have lots of options, there are alternative template engines galore, and many other components have plugins which replace or seriously modify their behavior.

Lest you think I’m manufacturing this from the air, here’s a quote from a django proponent discussing the idea of loose coupling in Django:

Developers coming from Ruby on Rails or other extremely opinionated frameworks may be used to following their framework’s best practices to avoid fighting against a framework which feels that it knows your project better than you do, but with Django you’ll be back in the driver’s seat.

– Will Larson on “loose coupling” in django

I think this is only true, if it’s true, by degree. It’s easier to do radical surgery to the framework in Django than it is in Struts, though I’m not all that convinced that Rails is harder to change than Django. But really, that’s beside the point. The fact of the matter is that framework surgery is much harder to do in Django than Pylons, and I think that’s not a wild claim, but a verifiable fact.

I wouldn’t recommend decoupling from the Django ORM without an extremely compelling reason. It is the most coupled of all the subparts of Django, and certainly not trivial to replace.

– Will Larson on using SQLAlchemy with Django (emphasis added)

Using SQLAlchemy in Django is not really hard (but at the same time it’s not all that easy either). But, it’s the kind of thing you wouldn’t want to do unless you really had a good reason.

Why? Because there are an awful lot of Django components that are “tightly integrated” with the django ORM.

But before everybody gets too mad at me, let me turn it around, and say that I think:

• it is not a bad thing to value integration above coupling.

Another Quote from Will’s blog, shows that he’s aware that loose coupling isn’t the single core value of Django:

Django places value on loose coupling, but it isn’t the sole design principle either.

– Will Larson (in a comment here)

I would say that even more strongly:

Django should not place loose coupling above developer productivity.

Adrian made a good point after my talk by suggesting that developers need to get things done, and to make sites that work now — not create software that is perfect by some abstract standard of design.

And I do think that some of the helpers that depend on the Django ORM are significant productivity wins. And, removing the django ORM dependency in all of them would be both very hard, and totally not worth it.

“Tightly Integrated” has value, and sometimes that value trumps “Loosely Coupled.” Zope is pretty tightly integrated into the ZODB. Many TurboGears 2 plugins are going to be pretty tightly integrated into SQLALchemy. Others will not, and I have encouraged some folks to rewrite their TG2 plugins to make them into pure WSGI apps that don’t depend on TG2 at all. Determining what exactly you will depend on, and how tightly you will be coupled to that dependency requires thought, and ultimately has consequences.

I’m hard on the Django folks here because I think the “tightly integrated/loosely coupled” buzz phrase is actually detrimental to understanding how the trade-off’s work.

And there are trade-offs and those trade-offs mean that there isn’t and will never be one perfect web-framework which somehow magically isn’t subject to the down-side of any of the constraints and design trade-offs that we all have to deal with every day.

Which brings me to the other major point I tried to make in my talk about django. Encapsulation, orthogality, or loose coupling is to some extent enforced by package boundaries. It’s not so much that you can’t be tightly coupled to the internals of an outside package, but that it feels wrong. And it feels wrong because when you start making libraries you start defining public interfaces, and making decisions about what’s internal and what’s external. And that makes monkeying with the internals feel icky. But if everything is all in one package, it’s a lot easier and less ‘icky” feeling to just grab some internal bit and do what you need to, since less thought has been given to what’s public and what’s private.

I think it’s a truism that there is no single barrier to IT project success more powerful than bad management. There are dozens, if not hundreds, of ways in which bad management directly and indirectly decreases productivity, and undermines even the the possibility of success.

Foucault, is the twentieth century master of uncovering hidden power relationships, and anybody who’s read his masterwork Discipline and Punish will learn a variety of tools to uncover hidden power relationships, and these tools can be immensely valuable in unpacking what’s really going on when a situation “just feels wrong.”

It’s hard to quantify, but knowing this stuff can definitely help you avoid political landmines, and create a safer work environment. But, I still haven’t figured out how to make Foucault palatable and understandable to the average IT manager. ;)

But Naked aggression is easy to spot

On the other hand, you don’t need a sophisticated set of tools for unpacking power relationships to see what’s wrong with some teams. For example, I once worked for a manager who held daily motivational meetings, in which he enumerated every small mistake that any of us could make that might make him look bad, and let each of us know in no uncertain terms that we would be “terminated” if we were seen to have made even the smallest mistake. If we didn’t answer the phone with the right phrase, or try to convince customers that they wanted more expensive products than they needed, or failed to live up to any of thirty different (and sometimes contradictory) arbitrary rules, we were told we would be fired instantly.

Doing the wrong thing may pay off now, but it almost always hurts you later.

I left after less than 6 months, by the time a year was up, only 2 of 20 of the original people on the team were left. But, the manager received a commendation for “running a tight ship.” Things were looking good. Then again, a year later he was fired, because his team just couldn’t keep up, and everyone on the team was so busy not screwing up that the had no time or energy left for the important things they were hired to do.

Dominance games, and pure aggressive pressure helped him meet his daily and weekly goals, but killed him in the long run.

Project managers need to learn to wield power with a light touch.

It’s inevitable that there are power-relationships involved in the context of an important project. If you’re doing things right people are passionate about the project, people want to do the right thing, and people don’t always agree about what the right thing is.

It can be tempting to jump into these disagreements and make decisions — and it can speed up progress in the short term. But be careful, just like the manager who made his short term goals, but lost in the long run, you may end up killing people’s passion, and cutting off discussion before critical information is revealed.

Sure, there are times when a strong good push can help people avoid prolonged and useless discussions, but it’s too easy to take advantage power relationships to avoid difficult but important discussions.

There’s lots more to be said on the subject of power relationships in software development, but I think one of the key things we have to understand is that computer nerds participate in power struggles too, they just do it naively and instinctively. And that leaves us easy prey for those who do it with knowledge, talent, and finesse. And those people are out there and in places where 2 years is a long time, they can get away with it for a surprisingly long time.

P.S. I wrote this post in 2007, almost published it in 2008, and am just now getting around to publishing it. So, please don’t think this is about anything happening to me personally right now, but it is something I keep seeing “around town” and something I think we need to understand better if we are going to stop falling prey to those who understand power politics better than we do and wield them more aggressively, and end up ruining many good projects.

Hosted Apps

One of the biggest changes is the new Hosted Apps system. We’re an Open Source hosting company, and we want to provide some of the best tools available for Open Source developers, so it only makes sense to use Open Source tools to do it. So, we now provide two dozen applications that you can install and use to help develop and manage your project, including trac, mediawiki, dotproject, a microbloging system, phpbb forums, an app for brainstorming ideas, and lots more.

From the perspective of an open source project maintainer, I think the best part of this is that I don’t have to manage them, do upgrades, backups, or worry about downtime — there are other people responsible for all that.

Have it your way

Part of the plan here is to make SourceForge more modular, and to let project managers use the tools that make sense for them and for their project. I think we’re the only open source hosting solution that provides svn, git, hg, bzr, and cvs source repositories. And with Trac, you’ve definitely got a far more full featured bug tracker than is available in most other open source project hosting.

One of the other advantages is that if you want new features in SourceForge, there’s now a clear and obvious way to do it. For example if you want some new ticket tracking feature you can add that feature to Trac, and once there’s a new release, it’ll be added to your sf.net project for free.

How the Consume side fits in

If you saw yesterday’s post, you’ll remember this diagram.

We’ve created the summary page based on the idea that projects have data in various places, in SourceForge developed apps, in hosted apps, on freshmeat, and out on the web in various places. In fact, all the the data on the sf.net project summary pages and download pages is fetched from the existing php apps via feeds and public API’s.

We still have lots of work to do, but all of this means that we’re changing the way SourceForge works to serve our developers better. We’re giving up on the assumption that we can provide the one right set of tools for all open source projects, and we’re also trying to leverage and improve existing open source solutions rather than reinvent the wheel.

To be fair, lots of this stuff wasn’t available when sourceforge got started, so back then we had to do some inventing, but the open solutions have passed us in lots of areas, and we’re taking advantage of that.

Developers need users too

And that brings us full circle, we want to grow project communities. In the end this means serving two sets of people well, and it means bringing those two communities together, serving developers means they get the tools they need to make better software, and serving downloaders and end users means the overall community grows. But the real growth happens when the line between developer and user begins to blur, with non-developers triaging bugs, writing documentation, doing translations, and sometimes even becoming developers themselves.

It’s all about users.

The big user experience wins come from some heuristics that we’ve developed to guess the best file to download. We’re analyzing your browser’s user agent for information about your operating system. And when the project has not told us about their preferred download, we’re analyzing file names and other data to get our best guess as to which file is the most relevant for a particular operating system.

This means we can generally give you a direct link to the right version of the file, right on the project page, so there’s no need to browse through a complex array of pages, and links just to finally get to the file you came to download in the first place.

Now with TG2 goodness.

Under the hood, there’s a new TurboGears 2 app powered by MongoDB. We’ve put this all together very quickly, and there have been a couple of rough spots here and there. Fortunately, none of the rough spots were in the TG2 or the TG2 stack.

But we did have a couple of rough patches. For example, we couldn’t use a mongo replica pair for the master database and have slaves on each node. So, we chose to try running the site all against a single replica pair rather than to do master-slave everywhere. This was compounded with a coding issue that, we were trying to pull tons more data out of MongoDB than we needed, and we ended up saturating a 2 gigabit network connection between the mongo master and the local slaves. Which if you think about it is kind of amazing, since MongoDB was still only using a few percent of the CPU on the box.

At this point though, all those issues are resolved and we are very confident that we’ll be able to add more user-facing improvements to SourceForge project pages and make the download side of things even better.

Since we had released a stable 2.0 version, followed by a 2.0.1 bugfix, we really were lacking a real pypi install mechanism. This is now a thing of the past! You can install a full blown TurboGears2 environment in one simple command:

easy_install tg.devtools

and this will fetch everything using normal lookups from the pypi. I hope this will help people out there.

At the same time I’d like to ask for help in that domain: If you are using TurboGears2 and find out that some dependency that needs compilation is not found in a compiled state (either on pypi or on our index), please help us:

1. By sending a compiled version of the egg to me (florent aide, you’ll find me on the mailing list easily)
2. By pointing us the missing dependency (send a mail in the turbogears or turbogears-trunk mailing list)
3. Or even better: try to become a maintainer for the specific dependency, that you know will always be missing for your architecture, and let us know about that great news. The whole open source community will become better, your karma (the real one, not the one you can admire on ohloh) will bump-up and you’ll be forever remembered on endor.

Now this leaves us with one more thing to do: finish the web site for turbogears that is currently in a poor state. If you would like to help us on the engine, please let us know in the mailing lists or via private message. If you want to help-out with content, please let us know also and we’ll be more than glad to give you an editor account and some directions to get some content online in this new website.

Cheers all,

Florent.