3 ms·
The Nagappan study is one of the few that at least tried to do a decent job, but there ought to be some sort of magic auto-responder every time someone cites it
by Silhouette 11y ago
The Nagappan study is one of the few that at least tried to do a decent job, but there ought to be some sort of magic auto-responder every time someone cites it that points out that none of the teams they studied was actually doing "pure TDD" as originally advocated by those who coined the term, so the baseline for all those comparative figures isn't entirely clear.
It's also pertinent that it always seems to be that same one report from nearly a decade ago, studying just a few development groups, that gets cited whenever anyone asks for hard data about the effectiveness of TDD. If TDD is as necessary as its more fervent advocates claim, then surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples.
- jacques_chester 11y ago> none of the teams they studied was actually doing "pure TDD" And still showed dramatic improvements in defect prevention. > surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples. The problem is funding. Nobody wants to stump up their project for it. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.
- Silhouette 11y agoAnd still showed dramatic improvements in defect prevention. Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out. Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.
- jacques_chester 11y ago> whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect. Here's what we know: under varying conditions, in multiple teams, with multiple levels of experience, in various problem domains, in multiple languages, in multiple companies in multiple locations, even a half-baked TDD had a powerful effect. Small sample sizes suck. But when you have a small sample and a very large effect, then the null hypothesis is under serious pressure. > Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. The point is not "did they pan out?" (modulo the usually utterly abysmal "success" rate in our industry). The point is "could they have panned out better?" > Unfortunately, none of those anecdotes in isolation tell us anything very useful I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences. I did so because you said "we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples." Then you shifted the goalposts. Pivotal is probably the most doctrinaire TDD company in the world. If you can find another company that has done TDD for as long, and as consistently, on as many projects, with as many different engineers, but showed a consistent failure rate, I'd love to hear about it. Your argument is "if it's so good, why hasn't everyone switched?" Because people are people. TDD is hard and an unfamiliar practice to almost everyone. It requires someone to teach, which makes it frustrating to the autodidacts who fill the industry. You might as well ask why we haven't all switched to Dvorak keyboards, why VHS beat Betacam, why we were stuck at 1080p screens for a decade, why we use enriched-uranium high pressure water reactors, why x86 beat every other ISA for decades and on and on and on. That a practice or technology is superior on the metrics we are "supposed" to care about is no guarantor of success. Contrariwise, an argument from popularity proves or disproves nothing about the actual empirical properties of a practice.