4 ms·

> ...“bugs found in requirements are 100x cheaper than bugs found in implementations.” ... There’s one tiny problem with the IBM Systems Sciences Institute stud

> ...“bugs found in requirements are 100x cheaper than bugs found in implementations.” ... There’s one tiny problem with the IBM Systems Sciences Institute study: it doesn’t exist It's funny to think how much of an effect this chart had on software engineering as a whole. I remember learning it at university and until today I thought it had some basis in science.

eschneider 5y ago
It "feels right" and is therefore "truthy."
- krisoft 5y ago
  Do you guys doubt that it is right? What kind of information would you require to show you that it is true? I'm one of the lucky ones who tends to work on safety critical "complex cyber-physical systems". Is that maybe the difference? There is the case where a developer runs unit tests before a commit. Some of the tests are unhappy. Developer fixes it and commits it up. Maybe hours spent? Then there is the case where something goes funky in the real-world all-up integrated system at the test range. Even in the best case tens of people waste a day. If we are unlucky it is a hard one and a small army of the most senior developers with the best operators and hardware people are hunting the bug for weeks. I fear to even sum op the wasted cumulative work days. I lived through multiple of these at different companies on different projects. (Both the first kind and the second kind.) Is the question if this is true? Or maybe the question is if this is true for your area of the industry too? Obviously I won't be able to provide sources for the ones I worked on. I bet no company really would want to release the hard data on these things. So let's look at some well-publicised problem caught in production: The Boeing 737 MAX MCAS issue. 346 deaths, 1 year, 8 months and 5 days grounding and the estimated direct costs are US$20 billion. Obviously this is all super anecdotal. I just want to understand what part of the question/problem you have doubts about.
  - twobitshifter 5y ago
    I think that’s the point - the chart confirm your bias and everyone else’s so we accept the results at face value. Nobody really knows if those bar heights were ever measured but of the millions of people to see that chart few questioned it.
    - krisoft 5y ago
      I never had to quote those charts though. Instead the conversation went like this: “Hello boss, do you remember that fubar two months ago?” I even mention the concrete ticket number, and when the number alone makes my boss face twitch then I know i choose the right fubar. “We just merges a CI change. If that whole category of problems happens again we will know within the hour, instead of spending two weeks debuging.” Its he same wisdom, but communicated in a more direct way.
  - marcosdumay 5y ago
    The chart very likely has the correct direction, and the numbers are probably completely wrong. From another point of view, the literal phase division there is nonsensical, but the idea it communicates isn't. But that's not relevant. What's relevant is that this is not how you do research. You can't just assume that. Yeah, there are plenty of examples of high-cost production failures. There are plenty of examples of low-cost production failures too, and plenty of examples of projects that failed because people spent all their time fixing the same few issues. If you want to state it as a fact, you have to count all of those and see how they compare.
  - gilbetron 5y ago
    Imagine a requirements document for Hacker News - such documents traditionally have a tremendous amount of change control around it, with tons of review for every change. Remember, people like you believe it is 100x easier to catch a bug in requirements and 100x gives you lots of budget to spend to make sure you catch bugs in requirements. So, there's a bug in the requirements document. Specifically, the 'logout' link has a spelling error. It says 'lugout'. If someone were to catch this bug in the requirements doc, it would trigger a bunch of costly review, as we established earlier. However, in the implementation phase, once the entire site is up and running, it would take maybe 5 minutes to fix. Trivial example, but there it is.
    - krisoft 5y ago
      Is this an imaginary situation or a real world experience of yours? I don't deny that IT is a wide and varied place, everything is possible I guess. In my experience in situations where changing a typo in a document was hard, changing the production code was even harder. But happy to hear your story where this wasn't the case. > Remember, people like you believe it is 100x easier to catch a bug in requirements What? I never said that. It is easier to fix, if the problem is caught. That says nothing about how easy or hard it is to catch it. > Trivial example, but there it is. The original question was "Are Late-Stage Bugs More Expensive?" To which my answer was "Yes, but I can't show you my proof because I work on proprietary projects" (which I understand is an answer which leaves everyone unsatisfied.) Your counter argument is "But let's imagine it is not!" ... which is what the phrase "begging the question" originally used to refer to. Imagining that the answer to a question is not what my experience tells me is not a convincing argument. But maybe I misunderstood what you wrote. If I did I'm sorry about that.

eschneider 5y ago

It "feels right" and is therefore "truthy."

krisoft 5y ago
Do you guys doubt that it is right? What kind of information would you require to show you that it is true? I'm one of the lucky ones who tends to work on safety critical "complex cyber-physical systems". Is that maybe the difference? There is the case where a developer runs unit tests before a commit. Some of the tests are unhappy. Developer fixes it and commits it up. Maybe hours spent? Then there is the case where something goes funky in the real-world all-up integrated system at the test range. Even in the best case tens of people waste a day. If we are unlucky it is a hard one and a small army of the most senior developers with the best operators and hardware people are hunting the bug for weeks. I fear to even sum op the wasted cumulative work days. I lived through multiple of these at different companies on different projects. (Both the first kind and the second kind.) Is the question if this is true? Or maybe the question is if this is true for your area of the industry too? Obviously I won't be able to provide sources for the ones I worked on. I bet no company really would want to release the hard data on these things. So let's look at some well-publicised problem caught in production: The Boeing 737 MAX MCAS issue. 346 deaths, 1 year, 8 months and 5 days grounding and the estimated direct costs are US$20 billion. Obviously this is all super anecdotal. I just want to understand what part of the question/problem you have doubts about.
- twobitshifter 5y ago
  I think that’s the point - the chart confirm your bias and everyone else’s so we accept the results at face value. Nobody really knows if those bar heights were ever measured but of the millions of people to see that chart few questioned it.
  - krisoft 5y ago
    I never had to quote those charts though. Instead the conversation went like this: “Hello boss, do you remember that fubar two months ago?” I even mention the concrete ticket number, and when the number alone makes my boss face twitch then I know i choose the right fubar. “We just merges a CI change. If that whole category of problems happens again we will know within the hour, instead of spending two weeks debuging.” Its he same wisdom, but communicated in a more direct way.
- marcosdumay 5y ago
  The chart very likely has the correct direction, and the numbers are probably completely wrong. From another point of view, the literal phase division there is nonsensical, but the idea it communicates isn't. But that's not relevant. What's relevant is that this is not how you do research. You can't just assume that. Yeah, there are plenty of examples of high-cost production failures. There are plenty of examples of low-cost production failures too, and plenty of examples of projects that failed because people spent all their time fixing the same few issues. If you want to state it as a fact, you have to count all of those and see how they compare.
- gilbetron 5y ago
  Imagine a requirements document for Hacker News - such documents traditionally have a tremendous amount of change control around it, with tons of review for every change. Remember, people like you believe it is 100x easier to catch a bug in requirements and 100x gives you lots of budget to spend to make sure you catch bugs in requirements. So, there's a bug in the requirements document. Specifically, the 'logout' link has a spelling error. It says 'lugout'. If someone were to catch this bug in the requirements doc, it would trigger a bunch of costly review, as we established earlier. However, in the implementation phase, once the entire site is up and running, it would take maybe 5 minutes to fix. Trivial example, but there it is.
  - krisoft 5y ago
    Is this an imaginary situation or a real world experience of yours? I don't deny that IT is a wide and varied place, everything is possible I guess. In my experience in situations where changing a typo in a document was hard, changing the production code was even harder. But happy to hear your story where this wasn't the case. > Remember, people like you believe it is 100x easier to catch a bug in requirements What? I never said that. It is easier to fix, if the problem is caught. That says nothing about how easy or hard it is to catch it. > Trivial example, but there it is. The original question was "Are Late-Stage Bugs More Expensive?" To which my answer was "Yes, but I can't show you my proof because I work on proprietary projects" (which I understand is an answer which leaves everyone unsatisfied.) Your counter argument is "But let's imagine it is not!" ... which is what the phrase "begging the question" originally used to refer to. Imagining that the answer to a question is not what my experience tells me is not a convincing argument. But maybe I misunderstood what you wrote. If I did I'm sorry about that.