4 ms·

The article appears to assume what it is asserting about TDD (Test Driven Development): Of course, this is a fictional example making a lot of assumptions. You

The article appears to assume what it is asserting about TDD (Test Driven Development): Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc… Here are the assumptions made: Production bugs without TDD: 100 Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%) Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost) Average time to fix a production bug: ~15 hours The author does try to justify this by saying: The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD. BUT... Where is the specific data and how representative is the author's experience of all or most software development? Software development is enormously varied. Kent Beck initially hawked TDD based on a project to rewrite Chrysler's payroll system that was cancelled. https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens... Also see: https://stackoverflow.com/questions/10149798/how-tdd-is-related-to-extreme-programming https://stackoverflow.com/questions/10149798/how-tdd-is-rela... The Chrysler C3 project is not the most convincing argument for TDD or other of Beck's ideas. But even assuming the C3 project was a roaring success -- a highly debatable proposition, one should ask how applicable the results are to many other quite different types of software projects. Similarly, what is the author of this article's actual experience and how applicable is it to other types of software projects? I don't question that automated tests can be helpful and indeed critical in some projects but I do question rigid orthodoxies like TDD or requiring unit tests for every function or class after implementation (many people promote unit tests everywhere while claiming they are not doing TDD). Tests take additional time to design and implement and thus cost more. They also have maintenance costs. Architectural and other changes in the software as it is developed can and do result in revising or rewriting the tests as well. Many successful software projects were designed and implemented in the 1970's, 1980's, 1990's, and even today without TDD and without unit tests -- or code reviews although I am focusing here on TDD and unit tests.

ericelliott 10y ago
While the example is fictional, the numbers used for it come from a meta-study of TDD research, and for the relative cost of fixing a bug in production vs implementation time, "Integrating Software Assurance into the Software Development Life Cycle (SDLC)" from the IBM System Sciences Institute. Original sources and raw data are available if you click through the links. None of the examples or figures presented rely on the Chrysler C3 project, or anything published by Kent Beck. I'm unsure why you'd mention them except to build a straw man argument. Straw men aside, this is all independent research with lots of 3rd party citations. Yes, tests do take additional time and do incur maintenance costs as well, and that additional time is accounted for in the numbers presented. TDD still easily wins. Yes, you can produce successful projects without TDD. You can also run a marathon with only 1 leg -- but maybe it would be easier with 2. As for questioning my credentials, I don't blame you. It's certainly a lot easier to try to cast doubt on somebody's credentials than it is to come up with a valid argument or think critically about your own beliefs and bias. If you have any sincere doubts, a sampling of projects I've worked on are listed at the bottom of the article.
- NumberSix 10y ago
  The linked article TDD: the Art of Fearless Programming by Ron Jeffries and Greg Melnik, IEEE Spectrum May/June 2007 https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven Development: By Example (2002) Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many have done. The IBM article Integrating Software Assurance into the Software Development Life Cycle (SDLC) https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC https://www.researchgate.net/publication/255965523_Integrati... has the following section: PROCESS TO SECURE CODE In the event of a vulnerability finding, the software code may require redesign and implementation. This iterative cycle is costly in time and resources. To truly understand security threats to a system, security must be addressed beginning with the initiation phase of the development process. For an organization this means they must allow the IA controls and requirements to drive design and influence the software requirements. Therefore, any identified security threats found during the requirements and analysis phase will drive design requirements and implementation. Security defects discovered can then be addressed at a component level before implementation. The cost of discovery and mitigation can be absorbed within the review, analysis and quality check performed during the design, and implementation of our SDLC. The resultant product is one with security built in rather than security retrofitted. A study was performed by the IBM System Science Institute in order determine the relative cost in order to fix defects within the SDLC. Figure 2 displays their findings. followed by a figure 3 (not 2) which is a simple graphic. There is no identifiable reference to the underlying data which is presumably some sort of study for the US Department of Defense for security software issues presumably for military related software projects. This does not appear to be a general study, nor is it at all clear what was studied or how. I am not questioning credentials but extrapolation from in Kent Beck's case a payroll system to general software development or in the IBM case from some sort of DoD software security projects to general software development. DoD and aerospace projects often have unusually high costs for bugs in the production systems. A bug in a flight avionics system can cause a literal crash with loss of millions or even billions of dollars (Space Shuttle for example) and lives.

ericelliott 10y ago

While the example is fictional, the numbers used for it come from a meta-study of TDD research, and for the relative cost of fixing a bug in production vs implementation time, "Integrating Software Assurance into the Software Development Life Cycle (SDLC)" from the IBM System Sciences Institute. Original sources and raw data are available if you click through the links. None of the examples or figures presented rely on the Chrysler C3 project, or anything published by Kent Beck. I'm unsure why you'd mention them except to build a straw man argument. Straw men aside, this is all independent research with lots of 3rd party citations. Yes, tests do take additional time and do incur maintenance costs as well, and that additional time is accounted for in the numbers presented. TDD still easily wins. Yes, you can produce successful projects without TDD. You can also run a marathon with only 1 leg -- but maybe it would be easier with 2. As for questioning my credentials, I don't blame you. It's certainly a lot easier to try to cast doubt on somebody's credentials than it is to come up with a valid argument or think critically about your own beliefs and bias. If you have any sincere doubts, a sampling of projects I've worked on are listed at the bottom of the article.

NumberSix 10y ago
The linked article TDD: the Art of Fearless Programming by Ron Jeffries and Greg Melnik, IEEE Spectrum May/June 2007 https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven Development: By Example (2002) Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many have done. The IBM article Integrating Software Assurance into the Software Development Life Cycle (SDLC) https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC https://www.researchgate.net/publication/255965523_Integrati... has the following section: PROCESS TO SECURE CODE In the event of a vulnerability finding, the software code may require redesign and implementation. This iterative cycle is costly in time and resources. To truly understand security threats to a system, security must be addressed beginning with the initiation phase of the development process. For an organization this means they must allow the IA controls and requirements to drive design and influence the software requirements. Therefore, any identified security threats found during the requirements and analysis phase will drive design requirements and implementation. Security defects discovered can then be addressed at a component level before implementation. The cost of discovery and mitigation can be absorbed within the review, analysis and quality check performed during the design, and implementation of our SDLC. The resultant product is one with security built in rather than security retrofitted. A study was performed by the IBM System Science Institute in order determine the relative cost in order to fix defects within the SDLC. Figure 2 displays their findings. followed by a figure 3 (not 2) which is a simple graphic. There is no identifiable reference to the underlying data which is presumably some sort of study for the US Department of Defense for security software issues presumably for military related software projects. This does not appear to be a general study, nor is it at all clear what was studied or how. I am not questioning credentials but extrapolation from in Kent Beck's case a payroll system to general software development or in the IBM case from some sort of DoD software security projects to general software development. DoD and aerospace projects often have unusually high costs for bugs in the production systems. A bug in a flight avionics system can cause a literal crash with loss of millions or even billions of dollars (Space Shuttle for example) and lives.