Over 1000 defects?

Guy Kawasaki has a funny and true to life list of the top ten lies engineers tell. But, almost as an asside he says:

Generally speaking, if the largest number of documented bugs doesn’t ever exceed 1,000, it means that the company isn’t tracking bugs carefully.

Which I think is crazy. Well, perhaps if you are Apple, or Adobe, or Microsoft, there’s some truth to this. But, my experience is that if you let the bug list get that high, on any 10 person project, you have a serious problem. Rather than spending time documenting bugs, spend time fixing them.

Your client isn’t paying you to track bugs, they are paying you to write software that works!
If you do test driven development you will fix lots of bugs at the moment of writing them. My experience is that the lower the time between when the bug is written and when it is found, the less time and energy are required to fix it, partly because you know where to look, and partly because the problem is still fresh in your memory.
If you are a manager, prioritize fixing bugs above creating new functionality.

If your boss walks in the door and says, we need feature X for customer Y, and it is your number one priority starting now, take a deep breath. And say, “That sounds great, but we have some important issues we need to take care of first in order to make sure that customers X, Y, Z, and A are all happy wth the quality of the software we deliver.” If you are afraid to do this, then get a better contact list and start lookign for other work, because standing up and taking responsibility to get bugs fixed is your job. Having other options in case your boss tosses you out on your ass, might make it easier to find the courage to do your job.
It may take courage to say no to new features until the curent defect problems are resolved, but you are the only one who can do it.

The more bugs you have, the harder they are to fix.

Ultimately, if your team isn’t fixing bugs faster than you create them, you face the very real possibility that your software project will be flooded. Eventually, your project could end up taking on new bugs faster than you can triage the existing problems, and you’ll sink just as sure as the Titanic once it started taking on water faster than the bilge pumps could bail it out.
“Quality is free,” doesn’t mean that it’s cheep, or that you can get quality without sustained effort. It just means that creating quality work, from the ground up — every single time, will actually save you money when you no longer have to pay for all the inspections, returns, and other costs that come from poor quality.
The same is true in Software Development. Ron Jeffries seems always asks “Why do you have so many bugs that you need a bug tracker? What’s wrong with your process?”

Toyota, when faced with defects, asks “how is our process broken?” and “How do we fix it so we don’t get this problem again.”

You’d be surprised what you can do when you treat bugs as quality problems that shouldn’t exist. You’ll definitely write more tests, and at first you’ll spend a lot more time fixing bugs. But if you manage to get it right, everything else in your whole system will go faster and more smoothly. And eventually, you’ll think the idea of tracking 1000 defects is insane too.

4 Responses to “Over 1000 defects?”

  1. Yeah, couldn’t agree more. That’s what automated unit testing is for. You don’t stop working until you’re green-bar-ing.

  2. 2Hartmut Holzgraefe

    I’m pretty sure this 1000 figure is meant to include *all* bug reports, even for bugs already fixed, not just open bugs …

  3. 3Alan Yoder

    Based on informal evidence, I’d say that a mature (10 years old) highly stable product written by senior developers will have clocked roughly one bug for every 20 lines of code over its lifetime.

    A majority of those bugs are introduced in fixes to other bugs–and therefore by definition not caught by the automated unit tests that generate those lovely green bars. Many times the offender is “an easy one-line fix”–add that to the list of lies. Often enough the new bug is worse than the one getting fixed. Sometimes the original problem is that someone changed the code logic without updating all the relevant comments, leading someone else to introduce bugs based on incorrect assumptions. These facts and others make every bug fix a calculated risk.

    Tracking only 1000 bugs sounds like paradise to me.

    The real issue isn’t tracking bugs, it’s deciding which ones will cause enough customer pain to be worth the risk of trying to fix them.

  4. If 1000 bugs counts everything over the lifetime of the product, that’s one story. If there are 1000 open bugs, that’s another. I assumed Guy meant the later rather than the former. Usually my cleints aren’t interested in tracking bugs that are fixed.

    Alan, you say “A majority of those bugs are introduced in fixes to other bugs -– and therefore by definition not caught by the automated unit tests” I’m not sure what you mean by that. I’ve found my unit tests to be invaluable in finding regressions caused by one-line bug fixes.

    Also, you can use test driven development to fix bugs — write a test that fails because of the bug (in otherwords automate the reproduction of that bug) and then when you make that test pass — along with all your other tests — you can have reasonable confidence that the bug fix causes no other problems.

    If you have a good set of tests, your risk calculation gets a lot easier. When all of your unit and functional tests pass there’s a good chance the bugfix you just checked in didn’t break anything else.

    Sure, there is always some risk, but when you make the risk vanishingly small, bugfixes start feeling safe and easy — and that’s when life starts getting good. ;)

Comments are currently closed.