If you all don't mind long articles, I hope I can be forgiven for a long comment. This is an excerpt buried in Zed Shaw's interview with Geoffrey Grossenbach t

If you all don't mind long articles, I hope I can be forgiven for a long comment. This is an excerpt buried in Zed Shaw's interview with Geoffrey Grossenbach transcribed here: http://podcast.rubyonrails.org/programs/1/episodes/zed_shaw http://podcast.rubyonrails.org/programs/1/episodes/zed_shaw --- Geoffrey: One thing I’ve always been fascinated whenever I’ve talked to you about it is just your personal process of development. You’re definitely someone who tries to pick the best tools and customize them to make them work as well as you can. You also have something where you keep track of your bug rate and how many bugs you’re writing, and tests that fail and then you adjust your process. How does that work? Zed: That’s not recommended for everyone. You have to basically be really, really disciplined. I’m actually not really, really disciplined; I’m doing it on one project, on my U2 project, and I’m tryout basically kind of like a quality control process – physical quality control. All I do is I track a bunch of metrics that don’t necessarily say how many bugs there are exactly, but they’re indicators of the bugs. I track them over time, and then I use statistics to tell me if I’m starting to suck or if I’m improving. I’m doing mostly C coding on that project, so a lot of this is I’m running my program under Valgrind with heavy testing. Then I track what my test coverage is, and then basically it’s just a series of numbers stream across my screen as I code. It’s kind of like auto-test – when I compile the thing it codes it – and then about every maybe 300 sampled I take a break, go in and crunch the numbers, and I see if I did better than last month. A lot of times what I’ll do is I’ll try a new technique; I’ll try a technique for a while and then I’ll go crunch the numbers and see if I actually had a statistical improvement or not. That’s the biggest thing; I don’t waste my time on stuff that doesn’t actually improve the bug rate – the defect rate. For example, at first I wasn’t doing code coverage. I wanted to see if code coverage improved your testing – coverage of your test code, or your test code having coverage. I wanted to see if that improved quality. So I didn’t do any code coverage. I measured all of my defect rates and figured out what my average defect rate was. I did maybe about 700 or 800 samples. Then I started doing code coverage and beefing up my code coverage. I spent maybe about a month improving my code coverage. In C code it’s real hard to get really good coverage because so many lines do so much stuff. But I got it up to about 60 percent. Then I went and crunched the numbers again to see if increasing the code coverage in test improved my defect rate. What happened was it didn’t improve my defect rate; my defect rate was still about the same. What it did improve was when I made changes – like if I had to do re-factoring – it reduced the amount of time to get my defects back down. So you make a change, you do your re-factoring, your defects go up, your defects go up, and then you have to spend time fixing all that. With heavier, more test-coverage it made it go down quicker, but it didn’t really improve my defect rate much. There’s some complexity in that. When you have more coverage, you are seeing more of your defects, so that’s part of it, but I found that test-coverage doesn’t really justify an improvement in quality initially. It mostly just improves your time to fix later. But out of the ways, that’s some weird stats crunching. The process actually comes from the Capability Maturity Models – Skies, Watson Freeze, Personal Software Process. So all you’ve got to do is rather go get his book, go through what he recommends. The key is as your code. Keep metrics and then crunch numbers to see if that’s improving things for you, and that’s really all that is.