TurboGears on Sourceforge

No, we’re not moving the TG2 hosting to SourceForge. Instead, Sourceforge is now using TG2 to display the front pages, project pages, and download pages for all projectsNew Sourceforge front page. This comes with a new look for SourceForge, but more than that it’s the first step in a fundamental rethinking of what SourceForge does. We’ve been given the opportunity to focus on improving the experience of users of SourceForge hosted software. We’re not ignoring developers, and lots of good stuff is going on for developers, but this latest update is all about the users. We wanted to make downloading software from SourceForge faster and easier, because this will help projects attract and maintain users. And more than 9 out of 10 of page views on SourceForge are by end-users, which means that the vast majority of page views on the SourceForge site are now going through TG2.

It’s all about users.

The big user experience wins come from some heuristics that we’ve developed to guess the best file to download. We’re analyzing your browser’s user agent for information about your operating system. And when the project has not told us about their preferred download, we’re analyzing file names and other data to get our best guess as to which file is the most relevant for a particular operating system.

This means we can generally give you a direct link to the right version of the file, right on the project page, so there’s no need to browse through a complex array of pages, and links just to finally get to the file you came to download in the first place.

Now with TG2 goodness.

sf.consume architectureUnder the hood, there’s a new TurboGears 2 app powered by MongoDB. We’ve put this all together very quickly, and there have been a couple of rough spots here and there. Fortunately, none of the rough spots were in the TG2 or the TG2 stack.

But we did have a couple of rough patches. For example, we couldn’t use a mongo replica pair for the master database and have slaves on each node. So, we chose to try running the site all against a single replica pair rather than to do master-slave everywhere. This was compounded with a coding issue that, we were trying to pull tons more data out of MongoDB than we needed, and we ended up saturating a 2 gigabit network connection between the mongo master and the local slaves. Which if you think about it is kind of amazing, since MongoDB was still only using a few percent of the CPU on the box.

At this point though, all those issues are resolved and we are very confident that we’ll be able to add more user-facing improvements to SourceForge project pages and make the download side of things even better.

12 Responses to “TurboGears on Sourceforge”

  1. 1Trevor

    What did you gain by using MongodB behind the scenes vs storing project details in a SQL database?

  2. Trevor:

    We store all of the project information we need in a single project document in mongoDB. Having a document oriented data store allows us to store all of that data in a very usable format, index it however we want, and retrieve it in a single database hit.

    This has proved very valuable, and MongoDB’s replication and peformance have both made it incredibly easy to scale sf.net up as traffic increases.

  3. Nice write-up, Mark. Don’t forget some of us are still going to work on the experience for developers! ;)

  4. 4Kevin Horn

    Thanks for writing this up, Mark! I’ve been looking forward to some details since the changeover.

  5. Luke.

    Yea I think it would be good to post something about the developer oriented improvements that have been made recently, as I don’t always feel like people see all of the hard work you guys have been putting in.

    Perhaps we can talk a bit more about that at OSCON.

  6. 6crkmstr

    oh my god, turbogears is alive an kick ass

  7. 7Daniel Lundin

    Very cool. Been evaluating mongodb for our app here as well, and seem to be running into the same issues and thoughts around replication.

    We initially wanted to have a single replica-pair and multiple slaves hinged off that pair, but can’t seem to get that to jive with automatic master failover.

    Do I understand correctly that you guys ended up running the entire thing on a single pair of servers then, in the end (unlike the illustration)?

  8. 8jack sikar

    So many say ‘ It’s so easy programming with python’, but I mean ‘It’s so very hard to istall TG2 on XP!’
    so I beg you for an installer, please let my heart laugh again.
    thank’s to all the members of TurboGears

  9. TurboGears2 is so cool! Nice choice!

  1. [...] on Sourceforge July 16th, 2009 admin Leave a comment Go to comments TurboGears on Sourceforge. Sourceforge recently relaunched, powered by TurboGears 2 and MongoDB. Mark Ramm has the [...]

  2. [...] Thinking about programming in new ways « TurboGears on Sourceforge [...]

Comments are currently closed.