8 ms·

> Because the benefits aren't higher, in fact they are lower It depends on what you think the benefits are. > As usual my advice on this is: look at the peopl

> Because the benefits aren't higher, in fact they are lower It depends on what you think the benefits are. > As usual my advice on this is: look at the people who build things you find highly impressive, and study how they did it. Good idea. Researchers at Microsoft and IBM conducted studies of applying TDD to real, commercial projects of varying sizes, languages, team structure and complexity. They found that defects found in production were 40 to 90% lower. Development was slower by 15-35%, assuming that you don't count re-work after release towards the overall time spent on development. http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.... These studies were not conducted with experienced TDDers, nor with TDD advocates on staff. Some of the teams abandoned TDD afterwards.

simula67 11y ago
Each chapter in the book 'Making Software' by Greg Wilson and Andy Oram, takes one claim usually made by software developers and examines the body of research surrounding that claim. Chapter 12 is on TDD. The conclusion was that the evidence is not conclusive.
- jacques_chester 11y ago
  It's an excellent book. Software engineering research has the same problem as sports science. Small samples, lots of difficult-to-control variables. No way to perform real RCTs. Not enough money or interest to really dig. That doesn't make it useless, but it does mean exercising judgement.
- ArkyBeagle 11y ago
  Can we then conclude that there's not a way to balance the 15 to 35 % rise in (one) cost against the 40 to 90% drop in defects ( if those are the measurements )? That's really interesting if it is the case.
  - jacques_chester 11y ago
    I think it can be rationally traded off, so long as everyone is clear about how to measure and what the decision criteria are. I think that for products with long release latency (ie the kind of shrink-wrapped software Microsoft specialises in), the accounting very much works against TDD on these findings, because the incentives are to front-load features and catch up on bugs with service patches. If you miss your release window, your feature will be bumped by years. For products with continuous deployment, the economics shift and invert. Blasting out code willy-nilly will eventually cause engineering to steadily be diverted to firefighting. Eventually forward progress stalls.
    - ArkyBeagle 11y ago
      This just confirms my understanding that nobody really understands the cost of defects.
      - jacques_chester 11y ago
        In the way we work at Pivotal, feature stories "earn" velocity, but bugs do not. So if attention is not paid to code quality, velocity will eventually trend downwards. This is intentional, as it gives you a feedback loop on quality and helps to improve predictions of actual feature progress. From there you can invert to points/cost per engineer/week. Of interest, our average points per engineer per week on Cloud Foundry has remained largely steady over time and scale. I'm wary of that metric because it breaks a fundamental concept of measuring velocity (that points are only meaningful inside a single project).
Silhouette 11y ago
The Nagappan study is one of the few that at least tried to do a decent job, but there ought to be some sort of magic auto-responder every time someone cites it that points out that none of the teams they studied was actually doing "pure TDD" as originally advocated by those who coined the term, so the baseline for all those comparative figures isn't entirely clear. It's also pertinent that it always seems to be that same one report from nearly a decade ago, studying just a few development groups, that gets cited whenever anyone asks for hard data about the effectiveness of TDD. If TDD is as necessary as its more fervent advocates claim, then surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples.
- jacques_chester 11y ago
  > none of the teams they studied was actually doing "pure TDD" And still showed dramatic improvements in defect prevention. > surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples. The problem is funding. Nobody wants to stump up their project for it. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.
  - Silhouette 11y ago
    And still showed dramatic improvements in defect prevention. Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out. Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.

simula67 11y ago

Each chapter in the book 'Making Software' by Greg Wilson and Andy Oram, takes one claim usually made by software developers and examines the body of research surrounding that claim. Chapter 12 is on TDD. The conclusion was that the evidence is not conclusive.

jacques_chester 11y ago
It's an excellent book. Software engineering research has the same problem as sports science. Small samples, lots of difficult-to-control variables. No way to perform real RCTs. Not enough money or interest to really dig. That doesn't make it useless, but it does mean exercising judgement.
ArkyBeagle 11y ago
Can we then conclude that there's not a way to balance the 15 to 35 % rise in (one) cost against the 40 to 90% drop in defects ( if those are the measurements )? That's really interesting if it is the case.
- jacques_chester 11y ago
  I think it can be rationally traded off, so long as everyone is clear about how to measure and what the decision criteria are. I think that for products with long release latency (ie the kind of shrink-wrapped software Microsoft specialises in), the accounting very much works against TDD on these findings, because the incentives are to front-load features and catch up on bugs with service patches. If you miss your release window, your feature will be bumped by years. For products with continuous deployment, the economics shift and invert. Blasting out code willy-nilly will eventually cause engineering to steadily be diverted to firefighting. Eventually forward progress stalls.
  - ArkyBeagle 11y ago
    This just confirms my understanding that nobody really understands the cost of defects.
    - jacques_chester 11y ago
      In the way we work at Pivotal, feature stories "earn" velocity, but bugs do not. So if attention is not paid to code quality, velocity will eventually trend downwards. This is intentional, as it gives you a feedback loop on quality and helps to improve predictions of actual feature progress. From there you can invert to points/cost per engineer/week. Of interest, our average points per engineer per week on Cloud Foundry has remained largely steady over time and scale. I'm wary of that metric because it breaks a fundamental concept of measuring velocity (that points are only meaningful inside a single project).

Silhouette 11y ago

The Nagappan study is one of the few that at least tried to do a decent job, but there ought to be some sort of magic auto-responder every time someone cites it that points out that none of the teams they studied was actually doing "pure TDD" as originally advocated by those who coined the term, so the baseline for all those comparative figures isn't entirely clear. It's also pertinent that it always seems to be that same one report from nearly a decade ago, studying just a few development groups, that gets cited whenever anyone asks for hard data about the effectiveness of TDD. If TDD is as necessary as its more fervent advocates claim, then surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples.

jacques_chester 11y ago
> none of the teams they studied was actually doing "pure TDD" And still showed dramatic improvements in defect prevention. > surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples. The problem is funding. Nobody wants to stump up their project for it. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.
- Silhouette 11y ago
  And still showed dramatic improvements in defect prevention. Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts. Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out. Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.