3 ms·

I thought the numbers came from: Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on Computer SE-1(4):1226-1241. As cited in "The Economic Impacts

I thought the numbers came from: Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on Computer SE-1(4):1226-1241. As cited in "The Economic Impacts of Inadequate Infrastructure for Software Testing" by RTI. https://www.nist.gov/document/report02-3pdf https://www.nist.gov/document/report02-3pdf Pretty sure that both of those papers exist, and the NIST table cites up to 990x and refers to the 76 Boehm paper, not any IBM org.

dwheeler 5y ago
Yes. That first paper is Boehm, Barry W., "Software Engineering", IEEE Transactions on Computers, December 1976, pp. 1226-1241, vol. 25 DOI Bookmark: 10.1109/TC.1976.1674590, https://www.computer.org/csdl/journal/tc/1976/12/01674590/13rRUzp02n5 https://www.computer.org/csdl/journal/tc/1976/12/01674590/13... At the time Barry W. Boehm was at the TRW Systems and Energy Group. The Boehm paper says "Fig. 3 shows a summary of current experience at IBM[4], GTE[5], and TRW on the relative cost of correcting software errors as a function of the phase in which they are corrected." and indeed figure 3 (page 1228) shows exponential growth. It only shows averages or ranges for each data source, and that's a legitimate critique. That said, it does show them for multiple companies, and then presents a trend line that plausibly follows from the data provided. Boehm has a good reputation, I expect that this really was a reasonable observation from real data. It's legitimate to question whether or not that is still true. Computer "science" is notorious for having almost no science - experiments are almost non-existent. I would love to see this & many other experiments conducted to see what's true today.
- jackcviers3 5y ago
  I bet it was true, in the context of the time: Card punches, shared compute time, physical packaging and shipments; imagine the cost of fixing a bug in a program in the rope ROM of the Apollo guidance computers (had they actually bothered fixing one). I'd love to see the study redone in our postmodern OTA, client/server, no physical media world. The physical overhead cost of fixing a product bug would likely be orders of magnitude lower than in 76, but the units deployed in the field would be many many orders of magnitude higher; so cost per unit is down, but there are waaaaay more units in production. If the bug caused a material cost to the user, I could easily see it being very expensive. As for there being no experiments; computer science is more of a mathematical discipline than a scientific one. You might say the same thing about theoretical physics, but I still think it's likely that we can't break the speed of light. At any rate, the IBM study may not have existed, but others that took me less than ten minutes of googling to find definitely did, and do show results in the same order of magnitude as the questionable IBM study. The question I have is - what are we really arguing for here by refuting this data, allowing more bugs to reach production? Having more broken code in the wild than we already do? Less rigorous development methods? There's barely any rigor in SWE as it exists today, anyway, yet things mostly work through a generous application of trial and error.

dwheeler 5y ago

Yes. That first paper is Boehm, Barry W., "Software Engineering", IEEE Transactions on Computers, December 1976, pp. 1226-1241, vol. 25 DOI Bookmark: 10.1109/TC.1976.1674590, https://www.computer.org/csdl/journal/tc/1976/12/01674590/13rRUzp02n5 https://www.computer.org/csdl/journal/tc/1976/12/01674590/13... At the time Barry W. Boehm was at the TRW Systems and Energy Group. The Boehm paper says "Fig. 3 shows a summary of current experience at IBM[4], GTE[5], and TRW on the relative cost of correcting software errors as a function of the phase in which they are corrected." and indeed figure 3 (page 1228) shows exponential growth. It only shows averages or ranges for each data source, and that's a legitimate critique. That said, it does show them for multiple companies, and then presents a trend line that plausibly follows from the data provided. Boehm has a good reputation, I expect that this really was a reasonable observation from real data. It's legitimate to question whether or not that is still true. Computer "science" is notorious for having almost no science - experiments are almost non-existent. I would love to see this & many other experiments conducted to see what's true today.

jackcviers3 5y ago
I bet it was true, in the context of the time: Card punches, shared compute time, physical packaging and shipments; imagine the cost of fixing a bug in a program in the rope ROM of the Apollo guidance computers (had they actually bothered fixing one). I'd love to see the study redone in our postmodern OTA, client/server, no physical media world. The physical overhead cost of fixing a product bug would likely be orders of magnitude lower than in 76, but the units deployed in the field would be many many orders of magnitude higher; so cost per unit is down, but there are waaaaay more units in production. If the bug caused a material cost to the user, I could easily see it being very expensive. As for there being no experiments; computer science is more of a mathematical discipline than a scientific one. You might say the same thing about theoretical physics, but I still think it's likely that we can't break the speed of light. At any rate, the IBM study may not have existed, but others that took me less than ten minutes of googling to find definitely did, and do show results in the same order of magnitude as the questionable IBM study. The question I have is - what are we really arguing for here by refuting this data, allowing more bugs to reach production? Having more broken code in the wild than we already do? Less rigorous development methods? There's barely any rigor in SWE as it exists today, anyway, yet things mostly work through a generous application of trial and error.