Compound Thinking

Serving Developers *and* Users at SourceForge

My last post might have left some folks thinking that we’ve been focused entirely on “end user” experience at SourceForge and have been ignoring the developer side of the equasion, if that’s you, and you’ve felt a bit left out, there’s very good news. In the 7 months since I’ve been here, there have been a significant number of developer focused changes at sf.net, and there are more coming before OSCON.

Hosted Apps

One of the biggest changes is the new Hosted Apps system. We’re an Open Source hosting company, and we want to provide some of the best tools available for Open Source developers, so it only makes sense to use Open Source tools to do it. So, we now provide two dozen applications that you can install and use to help develop and manage your project, including trac, mediawiki, dotproject, a microbloging system, phpbb forums, an app for brainstorming ideas, and lots more.

From the perspective of an open source project maintainer, I think the best part of this is that I don’t have to manage them, do upgrades, backups, or worry about downtime — there are other people responsible for all that.

Have it your way

Part of the plan here is to make SourceForge more modular, and to let project managers use the tools that make sense for them and for their project. I think we’re the only open source hosting solution that provides svn, git, hg, bzr, and cvs source repositories. And with Trac, you’ve definitely got a far more full featured bug tracker than is available in most other open source project hosting.

One of the other advantages is that if you want new features in SourceForge, there’s now a clear and obvious way to do it. For example if you want some new ticket tracking feature you can add that feature to Trac, and once there’s a new release, it’ll be added to your sf.net project for free.

How the Consume side fits in

If you saw yesterday’s post, you’ll remember this diagram.

We’ve created the summary page based on the idea that projects have data in various places, in SourceForge developed apps, in hosted apps, on freshmeat, and out on the web in various places. In fact, all the the data on the sf.net project summary pages and download pages is fetched from the existing php apps via feeds and public API’s.

We still have lots of work to do, but all of this means that we’re changing the way SourceForge works to serve our developers better. We’re giving up on the assumption that we can provide the one right set of tools for all open source projects, and we’re also trying to leverage and improve existing open source solutions rather than reinvent the wheel.

To be fair, lots of this stuff wasn’t available when sourceforge got started, so back then we had to do some inventing, but the open solutions have passed us in lots of areas, and we’re taking advantage of that.

Developers need users too

And that brings us full circle, we want to grow project communities. In the end this means serving two sets of people well, and it means bringing those two communities together, serving developers means they get the tools they need to make better software, and serving downloaders and end users means the overall community grows. But the real growth happens when the line between developer and user begins to blur, with non-developers triaging bugs, writing documentation, doing translations, and sometimes even becoming developers themselves.

TurboGears on Sourceforge

No, we’re not moving the TG2 hosting to SourceForge. Instead, Sourceforge is now using TG2 to display the front pages, project pages, and download pages for all projects. This comes with a new look for SourceForge, but more than that it’s the first step in a fundamental rethinking of what SourceForge does. We’ve been given the opportunity to focus on improving the experience of users of SourceForge hosted software. We’re not ignoring developers, and lots of good stuff is going on for developers, but this latest update is all about the users. We wanted to make downloading software from SourceForge faster and easier, because this will help projects attract and maintain users. And more than 9 out of 10 of page views on SourceForge are by end-users, which means that the vast majority of page views on the SourceForge site are now going through TG2.

It’s all about users.

The big user experience wins come from some heuristics that we’ve developed to guess the best file to download. We’re analyzing your browser’s user agent for information about your operating system. And when the project has not told us about their preferred download, we’re analyzing file names and other data to get our best guess as to which file is the most relevant for a particular operating system.

This means we can generally give you a direct link to the right version of the file, right on the project page, so there’s no need to browse through a complex array of pages, and links just to finally get to the file you came to download in the first place.

Now with TG2 goodness.

Under the hood, there’s a new TurboGears 2 app powered by MongoDB. We’ve put this all together very quickly, and there have been a couple of rough spots here and there. Fortunately, none of the rough spots were in the TG2 or the TG2 stack.

But we did have a couple of rough patches. For example, we couldn’t use a mongo replica pair for the master database and have slaves on each node. So, we chose to try running the site all against a single replica pair rather than to do master-slave everywhere. This was compounded with a coding issue that, we were trying to pull tons more data out of MongoDB than we needed, and we ended up saturating a 2 gigabit network connection between the mongo master and the local slaves. Which if you think about it is kind of amazing, since MongoDB was still only using a few percent of the CPU on the box.

At this point though, all those issues are resolved and we are very confident that we’ll be able to add more user-facing improvements to SourceForge project pages and make the download side of things even better.