Archive for the 'SE Michigan Tech' Category

TurboGears Joins the Pylons Project

After much debate, discussion, and contemplation, we’ve made an important decision, that will best ensure the future of TurboGears, and of the ideas on which it was based. TurboGears is merging into the Pylons Project.

A bit of background

We built TurboGears 2 on top of a the Pylons framework, and I have been working with Ben and the Pylons folks for a couple of years now.

Pylons is a lightweight framework that is neither full stack noropinionated. The Pylons team has recently joined up with the repoze.bfg folks to create a new low-level non-opinionated web application development library called Pyramid which is based on repoze.bfg and now part of the larger Pylons Project.

Pyramid status

You can use Pyramid to build web applications right now. It has more plug points, and is in many ways more flexible and extendable than the original Pylons and will provide an even better foundation for a full stack framework like TurboGears.

Pyramid has a strong, highly documented, well tested, approach, is already starting to show the fruits of merging in ideas from TurboGears, and Pylons. We expect it to be a great choice for those who want a flexible, non-opinionated and very fast framework for their projects.

The future of “Full Stack” frameworks

Fortunately, that’s not where the story ends. The Pylons Project leaders recognize that a “low level” framework is not enough.

Most web developers need to get things done quickly, and want a full stack setup and ready to go, so they can immediately start developing features rather than infrastructure.

That’s where we come in. TurboGears was the pioneer of the “full stack” set of integrated components approach among the modern Python web frameworks, and we have already developed many full stack tools.

Next Steps

So, our first step will be to add the TurboGears2 package to the legacy support in the Pylons Project. So, the Pylons 1.x and Turbogears2 packages will be maintained side by side as part of a single overall project. This change won’t impact the TG2 code, except that we will be officially acknowledging that both codebases are
tightly linked and now are part of a single project.

Maintaining the existing Pylons1+tg code will only be one part of the larger Pylons Project.

Big Harry Audacious Goal

Ultimately, we will also be working with the Pylons Project folks to create a new generation of rapid application development tools on top of Pyramid, using the TurboGears “Full Stack” philosophy. We will help to pick default templating engines, default session support, default data persistence mechanisms, integrate widget libraries and build high level tools like OAuth or OpenID support.

The main benifits of this merger will come when we reach across framework boundaries, work together with with Repoze, Pylons, and other web framework developers to build a set of high level tools that make building complex, modern web applications easier and faster. There’s a lot we can learn from each-other, and even more that we can do if we work through our differences and find new ways to collaborate.

There is no future but what we make

It’s been a great ride. And, it looks like just when we thought things were settling down, there’s another drop, and the roller coaster ride that is TurboGears isn’t done yet. I am forever grateful for the chance
to work with all the TurboGears developers, to face challenges, and to build something that’s been so valuable to so many. And I’m looking forward to what we can do together with the Pylons and Repoze folks.

A peek at a new

So, I’ve been working on in various ways for about a year now. It’s written in Python using modern open source tools, from RabbitMQ, and MongoDB, to Git and Mercurial. And we are committed to making this the most open forge possible. We’re committed, to open processes, open code, and perhaps most importantly open data.

The first thing we did was create some new pages for downloads. Recently we releases a new service designed just for open source project leaders who want to use as a directory and downloads service.

But, we’re also aware that one of the most important services we provide is project hosting. For the last several months a small group of us have been trying to bring’s tools into 2010. And now we’re releasing an early preview of those new developer/community tools:

We have a long way still to go, but every long journey begins with a single step, and today’s step is allowing you to try the new forge, to create new projects at:

Where you can go to get a new project, with our new tracker, wiki, git, svn, and other tools. Projects can have subprojects, and links to other tools hosted off site, along with the many features that brings (free web hosting, hosted apps, etc).

But, why do all this?

In 1999 SourceForge was cool.

It provided all the tools that an open source project needed to get going, from cvs hosting, to bug tracking, and e-mail list support.

They pioneered free free software project hosting, and helped to transform the software development culture from one which barely new about free software or open source, to one where nearly everybody I know uses open license software. Oh sure, some of them might not know it, but they have it on their phones, in their TVs, their wireless routers — not to mention all the websites they use everyday that run on open source.

But, time passed.

More alternatives came out, more projects (including my own) started self hosting, and the landscape of open source software development changed. took a long time coming out with support for new tools like svn, and then git.

Still, SourceForge has a special place in my heart. Partly it’s nostalgia, I suppose, but I still think:

  • the core mission is still right
  • and there is still a real need

We (Open Source developers) still need tools like git, mercurial, and svn hosting. We still need bug trackers and mailing lists. And in a meeting of other open source project leaders last fall, nearly every single one of them identified the time wasted integrating and administering these tools as one of their most important frustrations.

Not enough…

But, for many and other free project hosting services were just not good enough, they weren’t scriptable, the weren’t extensible, their data wasn’t portable, and so they felt like they had to take on that cost.

And I fundamentally believe that open source projects live an die by communication, and that can do something new by integrating the various kinds of “conversations” that happen around the project. We can integrate mailing lists and forums, we can integrate SCM and ticket trackers, etc.

New and improved

So, a couple of us have been quietly working on something new. The new forge is designed around a few core ideas:

  • that data should be portable (every project gets their own database, which they can take with them if they want),
  • that the open source community ought to be able to extend and enhance the tools they need,
  • that integrating and cross linking the various kinds of conversations that open source projects need to have ought to be easier.

So, what we’re announcing today is more of a commitment to getting there on all these things, and a commitment to the “release early, release often” project management strategy.

So, expect us to take your feedback and make things better. Expect us to release lots of small fixes, and expect a few places where things are broken/incomplete because we value feedback more than polish at this point.

People VS Process?

Lean Manufacturing people go around saying “it’s always a process problem.”

Meanwhile Gerry Weinberg, who wrote several books that I love, and gives lots of great advice, including the some of the best advice I’ve ever read about how to give advice, says “every problem is a people problem.”

So, which is it?

Are bad things that happen the result of bad processes, are they the result of things people do?

I’ve been party to a bit of discussion about this in the last month or two, and in the end it’s all pretty silly.

Processes are created by people, implemented by people, and are designed to accomplish the goals of people.

People run processes!

So, whenever something is broken, it’s people who will need to find the problem and fix it.

People can and do think of ways to improve processes everyday, but I’ll eat my shoe if you find a process that thinks of a way to improve people.

But there’s still a HUGE problem.

Modern companies seem to have a persistent failing — they look for people to blame when something goes wrong — and ignore the context in which those problems happened.

When something goes wrong, fire some people, and replace them with new people who make the same mistakes all over again.

Sometimes you “get lucky”.

The company might get lucky and find a person who’s able to raise awareness, reveal the larger contextual problems, and succeeded in spite of the fact that everything’s stacked against her.

More often than not though, the poor new guy doesn’t see the systematic pressures that caused everything to fall apart, at least not until it’s too late.

Sometimes replacing what’s broken isn’t enough.

Sometimes it’s the equivalent of a mechanic replacing your car’s engine several times in a row, because it keeps burning up — without ever checking to make sure oil is flowing normally, and the cooling system is working.

The easy way out.

It’s often easier to blame people because they don’t “control” them they way they do the context. This blame game is as old as the hills, but definitely not as pretty.

Help people fix processes

The solution is to ask people to look for the systematic pressures, give them the tools to find them, and to empower them to change the way work gets done.

In the end, people will improve the processes, if they believe they are allowed.

Sometimes a design isn’t working because you think you can’t change the one element that needs to be changed.

Ryan (via svn)

The same thing is true when you are designing the processes by which work gets done.

TurboGears on Sourceforge

No, we’re not moving the TG2 hosting to SourceForge. Instead, Sourceforge is now using TG2 to display the front pages, project pages, and download pages for all projectsNew Sourceforge front page. This comes with a new look for SourceForge, but more than that it’s the first step in a fundamental rethinking of what SourceForge does. We’ve been given the opportunity to focus on improving the experience of users of SourceForge hosted software. We’re not ignoring developers, and lots of good stuff is going on for developers, but this latest update is all about the users. We wanted to make downloading software from SourceForge faster and easier, because this will help projects attract and maintain users. And more than 9 out of 10 of page views on SourceForge are by end-users, which means that the vast majority of page views on the SourceForge site are now going through TG2.

It’s all about users.

The big user experience wins come from some heuristics that we’ve developed to guess the best file to download. We’re analyzing your browser’s user agent for information about your operating system. And when the project has not told us about their preferred download, we’re analyzing file names and other data to get our best guess as to which file is the most relevant for a particular operating system.

This means we can generally give you a direct link to the right version of the file, right on the project page, so there’s no need to browse through a complex array of pages, and links just to finally get to the file you came to download in the first place.

Now with TG2 goodness.

sf.consume architectureUnder the hood, there’s a new TurboGears 2 app powered by MongoDB. We’ve put this all together very quickly, and there have been a couple of rough spots here and there. Fortunately, none of the rough spots were in the TG2 or the TG2 stack.

But we did have a couple of rough patches. For example, we couldn’t use a mongo replica pair for the master database and have slaves on each node. So, we chose to try running the site all against a single replica pair rather than to do master-slave everywhere. This was compounded with a coding issue that, we were trying to pull tons more data out of MongoDB than we needed, and we ended up saturating a 2 gigabit network connection between the mongo master and the local slaves. Which if you think about it is kind of amazing, since MongoDB was still only using a few percent of the CPU on the box.

At this point though, all those issues are resolved and we are very confident that we’ll be able to add more user-facing improvements to SourceForge project pages and make the download side of things even better.

Ann Arbor ArbCamp Registration open again

I’ve been very, very busy trying to get TurboGears 2 beta 1 ready to go, as well as a few other interesting projects, and had neglected to blog about a ArbCamp before, then it was sold-out, and I didn’t blog about it because I didn’t want to raise people’s hopes only to have then dashed upon the rocks. But, we’ve secured a new venue, so ArbCamp registration is now Un-Sold-Out. It’s UnSold because it’s free, and it’s un-Sold-Out because we can now fit everybody in. We had over 160 people registered and on the wait-list, but could only let 100 people in. Now we have space for 200, so those on the wait-list and those who didn’t sign up in time have a second chance.

ArbCamp will be tomorrow night, in the upstairs of the downtown Cottage Inn’s, so this is kind of last minute, but I think it’ll be a very cool event. It’s an UnConference, and people will be self-organizing a variety of sessions, and the possibilities are endless.