Archive for the 'Programming' Category

Open Source SourceForge

It’s been a crazy, hectic, inspiring week for me. I had 3 talks at PyCon, travel, and the open sourcing of SourceForge.net’s new developer platform. The new tools, based on TurboGears, Python, MongoDB, RabbitMQ, and too many other python libraries to count, have been a labor of love for many of our developers.

Change at Sf.net

I very much feel like I’ve been running non-stop for the last couple of months. Last month we pushed out a new unified theme, while recovering from the attack earlier this year, making many small tweeks to the user interface. And now to top it all off this month we open sourced our new beta tools.

I have a feeling 2011 is going to be a long, tiring, and ultimately a great year. It’s certainly started out that way!

Open Source, Open Process, and an Open Framework

I think the decision to open source the forge is a huge step forward for sourceforge.net. We are moving towards:

  • more open processes,
  • more open source participation,
  • and a more open and flexible forge.

I can’t take credit for any of this. Since I started here two years ago, I’ve had the pleasure of working with a great team of people who care about Open Source, care about the site, and want to make things better.

We’ve released several libraries along the way, but I know we’ve all been looking forward to the release of the forge platform. It’s built on a powerful and flexible python stack, and while we know there are many places where we could have done better, we are proud to be a part of building the future of sourceforge.net.

And I think it’s paying off, I know that I’m personally very excited to see community participation already at this week’s PyCon sprint.

Now that the code is out in the open, and the community is looking at it more deeply, we are going to start focusing a bit more on opening up our development processes.

We track the work we do right on sourceforge, in the new ticket tracker:

sf.net/p/allura/tickets

And we are interested in hearing the feedback of the community, and seeing people get involved in the sausage making process.

We’re also working hard to make this new forge an open platform. Tools like the tracker, wiki, and forum are all plugins, and new tools can be easily created by anybody with the desire to make it happen.

Ultimately we think this is going to mean that the open source community gets better tools to manage their projects, and can focus their efforts where it counts — on doing the work that can change the world.

Feeling the Open Source Love

It’s kind of humbling to reallize how impossible this project would have been without the contributions of literally hundreds of open source developers that gave us everything from webob to mongodb, from html5lib to kombu, and from jquery to python.

I am so very grateful to live in a world where so many have given so much to make it possible for small teams like ours at sf.net to build stuff that works as quickly as we have. And given horror stories I’ve heard this week, I’m also grateful to be privileged to work for a company that understands that open source and business opportunity can go hand in hand.

Thanks

I particularly want to thank the TurboGears community, which provided us with lots of assistance along the way, and who put up with me being crazily busy and not as attentive as I wanted to be, or should have been. You guys rock, and we could not have done this without you!

TurboGears Joins the Pylons Project

After much debate, discussion, and contemplation, we’ve made an important decision, that will best ensure the future of TurboGears, and of the ideas on which it was based. TurboGears is merging into the Pylons Project.

A bit of background

We built TurboGears 2 on top of a the Pylons framework, and I have been working with Ben and the Pylons folks for a couple of years now.

Pylons is a lightweight framework that is neither full stack noropinionated. The Pylons team has recently joined up with the repoze.bfg folks to create a new low-level non-opinionated web application development library called Pyramid which is based on repoze.bfg and now part of the larger Pylons Project.

Pyramid status

You can use Pyramid to build web applications right now. It has more plug points, and is in many ways more flexible and extendable than the original Pylons and will provide an even better foundation for a full stack framework like TurboGears.

Pyramid has a strong, highly documented, well tested, approach, is already starting to show the fruits of merging in ideas from TurboGears, and Pylons. We expect it to be a great choice for those who want a flexible, non-opinionated and very fast framework for their projects.

The future of “Full Stack” frameworks

Fortunately, that’s not where the story ends. The Pylons Project leaders recognize that a “low level” framework is not enough.

Most web developers need to get things done quickly, and want a full stack setup and ready to go, so they can immediately start developing features rather than infrastructure.

That’s where we come in. TurboGears was the pioneer of the “full stack” set of integrated components approach among the modern Python web frameworks, and we have already developed many full stack tools.

Next Steps

So, our first step will be to add the TurboGears2 package to the legacy support in the Pylons Project. So, the Pylons 1.x and Turbogears2 packages will be maintained side by side as part of a single overall project. This change won’t impact the TG2 code, except that we will be officially acknowledging that both codebases are
tightly linked and now are part of a single project.

Maintaining the existing Pylons1+tg code will only be one part of the larger Pylons Project.

Big Harry Audacious Goal

Ultimately, we will also be working with the Pylons Project folks to create a new generation of rapid application development tools on top of Pyramid, using the TurboGears “Full Stack” philosophy. We will help to pick default templating engines, default session support, default data persistence mechanisms, integrate widget libraries and build high level tools like OAuth or OpenID support.

The main benifits of this merger will come when we reach across framework boundaries, work together with with Repoze, Pylons, and other web framework developers to build a set of high level tools that make building complex, modern web applications easier and faster. There’s a lot we can learn from each-other, and even more that we can do if we work through our differences and find new ways to collaborate.

There is no future but what we make

It’s been a great ride. And, it looks like just when we thought things were settling down, there’s another drop, and the roller coaster ride that is TurboGears isn’t done yet. I am forever grateful for the chance
to work with all the TurboGears developers, to face challenges, and to build something that’s been so valuable to so many. And I’m looking forward to what we can do together with the Pylons and Repoze folks.

Technical Debt isn’t always Debt

After yesterday’s posts about why you should not focus on reducing technical debt and why that’s not an excuse to ignore it either. Dave brought up a good point in a comment.

Technical debt can be assessed like real debt, to a large degree. How much are you paying monthly because of the debt? An annoying little thing, or wanting to upgrade to a new version just for newness’ sake has a low interest rate.

– Dave Brondsema

But it got me thinking, perhaps “debt” isn’t the right metaphor for this at all. Annoying issues that don’t cost anything, and don’t technically need to be “repaid” aren’t properly debt at all.

Perhaps technical warts would be a better term for those kind of things. It feels good to fix them. But rather than feel righteous for spending time fixing them all and “paying down our debt,” perhaps we ought to feel a little bit self indulgent for spending time and money on what amounts to cosmetic surgery for our code.

I’m not saying cosmetic surgery is never valuable, or that warts are never painful, and certainly not that you should ignore them. After all warts can be cancerous, and could perhaps kill you. But after investigating and discovering that they are benign, perhaps it’s not worth the cost to have them removed.

Ignoring “technical debt” is like playing with dynamite

Earlier today, I posted the somewhat controversial Focusing on Technical Debt is a dead end and apparently I missed some of the subtlety required to get my actual point across.

I still don’t believe that it makes sense to focus on technical debt.

Technical debt is a fact of life, and it’s a sign that you’re writing better code now than you did before. It’s a sign that the team is learning, and that things are getting better. So, you should learn to live with it, or even to celebrate it.

But, that does not mean you should ignore technical debt either.

Technical debt can have real costs, can create real project risk, and most importantly can limit the capacity of the system to produce value. If it’s doing that, you need to fix it. But even then you are better to FOCUS on the long view of increasing the capacity of the system. In other words:

  • Make sure your whole team is learning from the process
  • Make sure you fix the things that impede progress rather than just things that bug you
  • Make sure that you do enough root cause analysis to know why the obstructions got there in the first place

In other words elevate the priority of learning above the need to “fix” things now. That way you’ll avoid future mistakes, get more done, and at the same time continue to deliver value to customers.

In the end I don’t like the term “Technical Debt” as a catch-all because it’s a way of lumping all kinds of different code problems together into one basket which results in trivium being treated with the same seriousness as potentially project destroying technical risks.

So, you do need to pay attention to all technical debt, but some of it should be ignored, some of it should be responded to immediately, and some of it should be “fixed” when you next work on code that touches that thing. Not fixing the critical things is dangerous, and I thought everybody knew that. My point in my last post was that fixing the non-critical things with the same urgency is more insidious danger.

Focusing on it exclusively leads to a weird situation where you’re fixated on the past, and not moving forward anymore. And ultimately it short-circuits the critical thing, which is growing the capacity of the team to deliver value.

Focusing on removing “technical debt” is a dead end

Americans, myself included, have a very short term perspective. Results matter, and this quarter’s results matter most. Perhaps next quarter is important too, heck we might even think a year out.

But we rarely take the *long* view. Push harder and you can make this quarter’s numbers a bit better. Push harder and you’ll get this sale in before the end of the year.

Developers do this too. We push harder to get a product out the door on time, push harder to get a feature in to this release rather than the next.

And here’s one which I think is particularly pernicious, because it seems like long term thinking — push harder to reduce “technical debt” right now.

Technical debt is the distance between how you’d write it knowing what you know now, and how you did write it knowing what you knew then. If you’re constantly focused on fixing past mistakes, you’re not learning the things that will turn the code you’re writing today into “technical debt.”

My recomendation is to focus on other things, focus on solving the problems you’re facing today the best way you know how, focus on cleaning as you go, focus on growing the team’s ability to learn — in other words:

Don’t focus on maximizing the utilization of the system,
focus on growing the capacity of the system.

There are a lot of things in projects I’ve written that I wouldn’t do the same way again. But if I get caught up in fixing every bad API decision in TurboGears or whatever, I would be paralyzed and unable to move forward. Of course it’s not hypothetical, I’ve definitely been there, done that. But I’ve learned from that mistake, and hopefully I can maximize the capacity of the system by helping you to avoid it. ;)