3 ms·

> none of the teams they studied was actually doing "pure TDD" And still showed dramatic improvements in defect prevention. > surely after all this time we sh

> none of the teams they studied was actually doing "pure TDD" And still showed dramatic improvements in defect prevention. > surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples. The problem is funding. Nobody wants to stump up their project for it. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.

Silhouette 11y ago
And still showed dramatic improvements in defect prevention. Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out. Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.
- jacques_chester 11y ago
  > whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect. Here's what we know: under varying conditions, in multiple teams, with multiple levels of experience, in various problem domains, in multiple languages, in multiple companies in multiple locations, even a half-baked TDD had a powerful effect. Small sample sizes suck. But when you have a small sample and a very large effect, then the null hypothesis is under serious pressure. > Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. The point is not "did they pan out?" (modulo the usually utterly abysmal "success" rate in our industry). The point is "could they have panned out better?" > Unfortunately, none of those anecdotes in isolation tell us anything very useful I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences. I did so because you said "we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples." Then you shifted the goalposts. Pivotal is probably the most doctrinaire TDD company in the world. If you can find another company that has done TDD for as long, and as consistently, on as many projects, with as many different engineers, but showed a consistent failure rate, I'd love to hear about it. Your argument is "if it's so good, why hasn't everyone switched?" Because people are people. TDD is hard and an unfamiliar practice to almost everyone. It requires someone to teach, which makes it frustrating to the autodidacts who fill the industry. You might as well ask why we haven't all switched to Dvorak keyboards, why VHS beat Betacam, why we were stuck at 1080p screens for a decade, why we use enriched-uranium high pressure water reactors, why x86 beat every other ISA for decades and on and on and on. That a practice or technology is superior on the metrics we are "supposed" to care about is no guarantor of success. Contrariwise, an argument from popularity proves or disproves nothing about the actual empirical properties of a practice.
  - Silhouette 11y ago
    While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect. You've invented this term "half-baked TDD", but that seems a little unfair. My point was that the groups in the Nagappan study were doing significantly more than just TDD. For example, they also had varying levels of dedicated design activities in addition to anything test-driven they were doing. It also wasn't clear to what extent some of them were doing similar kinds of unit testing in the alternative scenarios from the study. So you have neither a clear baseline nor a clear change. For example, one alternative possibility that still seems reasonably consistent with the evidence in that study is that unit testing is good at improving quality, that TDD promotes writing more tests, and that any design weaknesses that a TDD style might encourage in isolation were mitigated by the separate design activities those groups were doing beyond the basic fail-pass-refactor cycle required for TDD. The point is "could they have panned out better?" Erm... Yes. I just made exactly that point, in my very last post, the one you were replying to. I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences. Who was making that argument? I certainly wasn't. Your argument is "if it's so good, why hasn't everyone switched?" No, it isn't. Let's be clear about this. Prominent TDD advocates, Bob Martin among them, claim quite unambiguously that TDD is essential to writing good software, even using patronising and insulting language like "unprofessional" to describe anyone who doesn't do it. If that were actually true, if TDD is inherently superior to any other development process and anyone not following it is actively doing programming wrong, then the results achieved by organisations using TDD should be clearly and consistently superior to the results achieved by those not using TDD, other factors being reasonably equivalent. I claim that this is not the case, and I cite as evidence the simple fact that after nearly two decades TDD still represents a tiny part of the industry practice. This is an industry that over the same time period has brought numerous minor programming languages to prominence and mainstream acceptance; brought many ideas from fields like functional programming and distributed systems from niche applications or academic studies into the mainstream along the way; shifted a large proportion of mainstream development from the previous desktop/server model to web apps, cloud hosting, mobile development, and much more sophisticated embedded systems; made dramatic shifts in development processes such as the migration to a DevOps style of integration and deployment in many cases; and seen the rise of Open Source and more generally of collaborative development from a fun pastime for geeks to a strong influence behind much of the software we run every day. Given that, I'm sorry but I find it patently absurd to argue that the only reasons hardly anyone is doing TDD, even though it is so inherently superior in both quality of results and cost effectiveness, are that it is hard or unfamiliar. Many of those other changes I mentioned above have been adopted across large parts of the industry within much less time, even though the ideas were completely new and/or required completely different mindsets and understanding. If TDD really were essential to get good results and so clearly superior to other development processes as the evangelists frequently claim, then as I said before, by now we should see a vast pile of evidence, not just the same study with a sample size of four being cited nearly a decade later, and not just the occasional business -- particularly one that by your own admission is "probably the most doctrinaire TDD company in the world" -- that has used TDD and not failed. Many of us worked on software projects that have not failed. Not failing is table stakes for this debate.

Silhouette 11y ago

And still showed dramatic improvements in defect prevention. Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out. Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.

jacques_chester 11y ago
> whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect. Here's what we know: under varying conditions, in multiple teams, with multiple levels of experience, in various problem domains, in multiple languages, in multiple companies in multiple locations, even a half-baked TDD had a powerful effect. Small sample sizes suck. But when you have a small sample and a very large effect, then the null hypothesis is under serious pressure. > Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. The point is not "did they pan out?" (modulo the usually utterly abysmal "success" rate in our industry). The point is "could they have panned out better?" > Unfortunately, none of those anecdotes in isolation tell us anything very useful I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences. I did so because you said "we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples." Then you shifted the goalposts. Pivotal is probably the most doctrinaire TDD company in the world. If you can find another company that has done TDD for as long, and as consistently, on as many projects, with as many different engineers, but showed a consistent failure rate, I'd love to hear about it. Your argument is "if it's so good, why hasn't everyone switched?" Because people are people. TDD is hard and an unfamiliar practice to almost everyone. It requires someone to teach, which makes it frustrating to the autodidacts who fill the industry. You might as well ask why we haven't all switched to Dvorak keyboards, why VHS beat Betacam, why we were stuck at 1080p screens for a decade, why we use enriched-uranium high pressure water reactors, why x86 beat every other ISA for decades and on and on and on. That a practice or technology is superior on the metrics we are "supposed" to care about is no guarantor of success. Contrariwise, an argument from popularity proves or disproves nothing about the actual empirical properties of a practice.
- Silhouette 11y ago
  While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect. You've invented this term "half-baked TDD", but that seems a little unfair. My point was that the groups in the Nagappan study were doing significantly more than just TDD. For example, they also had varying levels of dedicated design activities in addition to anything test-driven they were doing. It also wasn't clear to what extent some of them were doing similar kinds of unit testing in the alternative scenarios from the study. So you have neither a clear baseline nor a clear change. For example, one alternative possibility that still seems reasonably consistent with the evidence in that study is that unit testing is good at improving quality, that TDD promotes writing more tests, and that any design weaknesses that a TDD style might encourage in isolation were mitigated by the separate design activities those groups were doing beyond the basic fail-pass-refactor cycle required for TDD. The point is "could they have panned out better?" Erm... Yes. I just made exactly that point, in my very last post, the one you were replying to. I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences. Who was making that argument? I certainly wasn't. Your argument is "if it's so good, why hasn't everyone switched?" No, it isn't. Let's be clear about this. Prominent TDD advocates, Bob Martin among them, claim quite unambiguously that TDD is essential to writing good software, even using patronising and insulting language like "unprofessional" to describe anyone who doesn't do it. If that were actually true, if TDD is inherently superior to any other development process and anyone not following it is actively doing programming wrong, then the results achieved by organisations using TDD should be clearly and consistently superior to the results achieved by those not using TDD, other factors being reasonably equivalent. I claim that this is not the case, and I cite as evidence the simple fact that after nearly two decades TDD still represents a tiny part of the industry practice. This is an industry that over the same time period has brought numerous minor programming languages to prominence and mainstream acceptance; brought many ideas from fields like functional programming and distributed systems from niche applications or academic studies into the mainstream along the way; shifted a large proportion of mainstream development from the previous desktop/server model to web apps, cloud hosting, mobile development, and much more sophisticated embedded systems; made dramatic shifts in development processes such as the migration to a DevOps style of integration and deployment in many cases; and seen the rise of Open Source and more generally of collaborative development from a fun pastime for geeks to a strong influence behind much of the software we run every day. Given that, I'm sorry but I find it patently absurd to argue that the only reasons hardly anyone is doing TDD, even though it is so inherently superior in both quality of results and cost effectiveness, are that it is hard or unfamiliar. Many of those other changes I mentioned above have been adopted across large parts of the industry within much less time, even though the ideas were completely new and/or required completely different mindsets and understanding. If TDD really were essential to get good results and so clearly superior to other development processes as the evangelists frequently claim, then as I said before, by now we should see a vast pile of evidence, not just the same study with a sample size of four being cited nearly a decade later, and not just the occasional business -- particularly one that by your own admission is "probably the most doctrinaire TDD company in the world" -- that has used TDD and not failed. Many of us worked on software projects that have not failed. Not failing is table stakes for this debate.