4 ms·

Sorry if this is naive, but my impression is that the statement "bugs are n times more expensive to fix in production" itself is ill-defined. How do you measure

Sorry if this is naive, but my impression is that the statement "bugs are n times more expensive to fix in production" itself is ill-defined. How do you measure it? I can think of two ways: 1. You measure the average cost of fixing bugs before they go to production. You also measure the average cost of fixing bugs after they go to production. I think this measurement is uninteresting, this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not. 2. Identify 2N bugs before going into production. Randomly select N bugs and don't go to production before they are fixed. Then go to production and fix the other N bugs after a while. I think this would be more interesting, but I suspect that nobody does this, as this measurement itself is possibly more costly than all of the 2N bugs together.

Someone 5y ago
“this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not.” I think you’re using the wrong definition of cost. A bug might take an engineer 3 seconds to fix, but still could cost millions if pushed to production. Certainly historically, that could mean producing thousands of CDs or floppy disks, or burning things to PROM. With products such as interplanetary probes, a bug could harm quite a few people in their career (some people do their PhD on an experiment for a Mars rover. If that crashes on landing, those designing it still have their design to show, but questions will linger whether it would have worked. Those doing a PhD on the results of that experiment truly are hosed)
- steerablesafe 5y ago
  And a product being delayed due to unimportant bugs could also cost millions, if I use the right definition. The point still stands, how do you actually measure it?
AstralStorm 5y ago
It really depends on how the system is deployed. For a web service, the cost might be almost identical, except reputation damage, which is impossible to properly quantify. For a game, where almost everyone plays it on launch, if it has too many bugs, the product might be sunk, or it might be a rolling update with minor extra cost of compatibility. For embedded firmware... Well... You might never really get it fixed, as not everyone will upgrade. For mission critical software, it can literally kill people or destroy hardware, or cause huge costs.

Someone 5y ago

“this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not.” I think you’re using the wrong definition of cost. A bug might take an engineer 3 seconds to fix, but still could cost millions if pushed to production. Certainly historically, that could mean producing thousands of CDs or floppy disks, or burning things to PROM. With products such as interplanetary probes, a bug could harm quite a few people in their career (some people do their PhD on an experiment for a Mars rover. If that crashes on landing, those designing it still have their design to show, but questions will linger whether it would have worked. Those doing a PhD on the results of that experiment truly are hosed)

steerablesafe 5y ago
And a product being delayed due to unimportant bugs could also cost millions, if I use the right definition. The point still stands, how do you actually measure it?

AstralStorm 5y ago

It really depends on how the system is deployed. For a web service, the cost might be almost identical, except reputation damage, which is impossible to properly quantify. For a game, where almost everyone plays it on launch, if it has too many bugs, the product might be sunk, or it might be a rolling update with minor extra cost of compatibility. For embedded firmware... Well... You might never really get it fixed, as not everyone will upgrade. For mission critical software, it can literally kill people or destroy hardware, or cause huge costs.