4 ms·

Each chapter in the book 'Making Software' by Greg Wilson and Andy Oram, takes one claim usually made by software developers and examines the body of research s

Each chapter in the book 'Making Software' by Greg Wilson and Andy Oram, takes one claim usually made by software developers and examines the body of research surrounding that claim. Chapter 12 is on TDD. The conclusion was that the evidence is not conclusive.

jacques_chester 11y ago
It's an excellent book. Software engineering research has the same problem as sports science. Small samples, lots of difficult-to-control variables. No way to perform real RCTs. Not enough money or interest to really dig. That doesn't make it useless, but it does mean exercising judgement.
ArkyBeagle 11y ago
Can we then conclude that there's not a way to balance the 15 to 35 % rise in (one) cost against the 40 to 90% drop in defects ( if those are the measurements )? That's really interesting if it is the case.
- jacques_chester 11y ago
  I think it can be rationally traded off, so long as everyone is clear about how to measure and what the decision criteria are. I think that for products with long release latency (ie the kind of shrink-wrapped software Microsoft specialises in), the accounting very much works against TDD on these findings, because the incentives are to front-load features and catch up on bugs with service patches. If you miss your release window, your feature will be bumped by years. For products with continuous deployment, the economics shift and invert. Blasting out code willy-nilly will eventually cause engineering to steadily be diverted to firefighting. Eventually forward progress stalls.
  - ArkyBeagle 11y ago
    This just confirms my understanding that nobody really understands the cost of defects.
    - jacques_chester 11y ago
      In the way we work at Pivotal, feature stories "earn" velocity, but bugs do not. So if attention is not paid to code quality, velocity will eventually trend downwards. This is intentional, as it gives you a feedback loop on quality and helps to improve predictions of actual feature progress. From there you can invert to points/cost per engineer/week. Of interest, our average points per engineer per week on Cloud Foundry has remained largely steady over time and scale. I'm wary of that metric because it breaks a fundamental concept of measuring velocity (that points are only meaningful inside a single project).

jacques_chester 11y ago

It's an excellent book. Software engineering research has the same problem as sports science. Small samples, lots of difficult-to-control variables. No way to perform real RCTs. Not enough money or interest to really dig. That doesn't make it useless, but it does mean exercising judgement.

ArkyBeagle 11y ago

Can we then conclude that there's not a way to balance the 15 to 35 % rise in (one) cost against the 40 to 90% drop in defects ( if those are the measurements )? That's really interesting if it is the case.

jacques_chester 11y ago
I think it can be rationally traded off, so long as everyone is clear about how to measure and what the decision criteria are. I think that for products with long release latency (ie the kind of shrink-wrapped software Microsoft specialises in), the accounting very much works against TDD on these findings, because the incentives are to front-load features and catch up on bugs with service patches. If you miss your release window, your feature will be bumped by years. For products with continuous deployment, the economics shift and invert. Blasting out code willy-nilly will eventually cause engineering to steadily be diverted to firefighting. Eventually forward progress stalls.
- ArkyBeagle 11y ago
  This just confirms my understanding that nobody really understands the cost of defects.
  - jacques_chester 11y ago
    In the way we work at Pivotal, feature stories "earn" velocity, but bugs do not. So if attention is not paid to code quality, velocity will eventually trend downwards. This is intentional, as it gives you a feedback loop on quality and helps to improve predictions of actual feature progress. From there you can invert to points/cost per engineer/week. Of interest, our average points per engineer per week on Cloud Foundry has remained largely steady over time and scale. I'm wary of that metric because it breaks a fundamental concept of measuring velocity (that points are only meaningful inside a single project).