5 ms·

I guess that 100x is just an arbitrary number that means 'orders of magnitude'. If production means going to Pluto on a super expensive space mission, and the b

by jfmc 5y ago

I guess that 100x is just an arbitrary number that means 'orders of magnitude'. If production means going to Pluto on a super expensive space mission, and the bug ruins all the scientific experiments, and your future funding depends on the success of the mission, I'd say that the cost tends to infinity. If the product requires some kind of polishing that only interaction with real users can provide, then the cost of bugs at production be even negative (spend less time on the requirement side, etc.).

ballenf 5y ago
100x is probably conservative in an enterprise pure waterfall development environment, regardless of application.
- dumbfounder 5y ago
  This is why so many developers can’t be left to their own devices to decide when a product is launch ready. There will always be bugs. Always. You can ship with known bugs. It’s about being pragmatic and fixing it at the right time. Not all bugs have the same impact.
  - manquer 5y ago
    You could make the exact argument about product managers.. This is why product managers / business can't be left to their own devices to decide when a product is launch ready. Not all bugs have same impact, and they are not experts to understand which are high impact. This statement is just as true as yours. Doesn't mean you should ignore either one group to make a decision.
asah 5y ago
Ha! you're being too kind. :-) How about life saving medical applications (insulin pumps, pacemakers etc), infrastructure (power, transportation, internet) let alone weapons of mass destruction. Then, consider subtle security holes as serious bugs to be exploited en masse. People try to air gap the hardware, and even that's not always enough (see StuxNet).
- hutzlibu 5y ago
  Yeah, but this is the point. It really depends on the context. Trying to fix every bug in the pure informational website but missing your product launch? Not worth it, just fix it later. (or not at all, have you surfed the web with the console open? It is flooded with warnings and errors) But miss the critical bug in the webshop, that cause customers to legally buy for free? That is expensive.
  - ajeet_dhaliwal 5y ago
    As a software engineer who has been mostly focused on testing over the last few years and leading a team focused on this, it's clear to me now that balancing what's worth testing and what's not is a delicate art and very difficult to always get right. I do believe that a common failure is not investing in adequate regression focused tests. I've been on projects were these were not in place and an update lost a company millions of dollars very quickly prior to being noticed in production, I've been on projects were delaying any further to release would have lost critical learning data during a period of time that could not be guaranteed to come again quickly (for example market volatility or turmoil). The best engineers in my opinion love testing as much as building new features because they understand that production fire fighting is the least fun activity of all. Quick plug, I believe solid test reporting makes the ROI on test development clearer. I'm founder at Tesults - a test results reporting app (https://www.tesults.com https://www.tesults.com) and I'd love your feedback on it, send me an email if you have a moment.
scotty79 5y ago
Even if 'it's orders of magnitude' it still must be orders of magnitude cheaper than for some other industry. I bet recalling millions of cars due to some small mechanical bug is very expensive.
speeder 5y ago
Or see the case of the installer on Myth 2, that you could also say the cost was infinite, since the but on the installer made the company litearlly go bankrupt and get purchased by Microsoft.
- empthought 5y ago
  1. It was the uninstaller, not the installer. 2. That is not at all what happened. Bungie never went bankrupt.
onionisafruit 5y ago
You need to remember the context when this study supposedly took place. "Production" used to have a literal meaning -- producing physical media. 100x was a reasonable number to believe in that case even without extreme situations like a mission to Pluto. I first learned this in QA training. It's been decades so I don't remember if 100x was mentioned. They listed all the people required to fix a bug once it had gone to production. Off the top of my head that included: - Customer support handling calls from customers -- possibly thousands of times. This isn't necessarily part of the cost of fixing the bug, but it's part of discovering a bug in production. - Some combination of managers and product managers prioritizing the bug and scheduling programmers to fix it. - Programmers finding and fixing the bug. - QA validating the fix. - Writing an installer for the fix. - QA validating the installer. - Producing the fix. - Shipping the fix to effected customers -- which may be all customers. We were being trained to do QA, so we compared that to the cost involved when QA discovers a bug. That was essentially just the cost of us writing a bug report and programmers fixing the bug.
- BeefWellington 5y ago
  Adding onto this, if the fix requires some kind of notice or error message detail that didn't previously exist it can also require work from your internationalization team to translate that error into N languages your software supports.
  - onionisafruit 5y ago
    I did forget that. I'm pretty sure that was actually listed in the training because the company I was at made software that was translated into a lot of languages.
- onionisafruit 5y ago
  Replying to myself because I got too excited and hit enter. Compare the above to a modern SaaS company. Customer support handling the reports is a fraction of the cost because it doesn't require spending time on the phone with each customer. Most don't have QA, so those costs are gone. No writing and validating installers. No cost of making and shipping physical media. One other thing is that programmers were a lot cheaper then. So the time a programmer spent actually fixing the bug was even less significant.

by jfmc 5y ago

ballenf 5y ago

100x is probably conservative in an enterprise pure waterfall development environment, regardless of application.

dumbfounder 5y ago
This is why so many developers can’t be left to their own devices to decide when a product is launch ready. There will always be bugs. Always. You can ship with known bugs. It’s about being pragmatic and fixing it at the right time. Not all bugs have the same impact.
- manquer 5y ago
  You could make the exact argument about product managers.. This is why product managers / business can't be left to their own devices to decide when a product is launch ready. Not all bugs have same impact, and they are not experts to understand which are high impact. This statement is just as true as yours. Doesn't mean you should ignore either one group to make a decision.

asah 5y ago

Ha! you're being too kind. :-) How about life saving medical applications (insulin pumps, pacemakers etc), infrastructure (power, transportation, internet) let alone weapons of mass destruction. Then, consider subtle security holes as serious bugs to be exploited en masse. People try to air gap the hardware, and even that's not always enough (see StuxNet).

hutzlibu 5y ago
Yeah, but this is the point. It really depends on the context. Trying to fix every bug in the pure informational website but missing your product launch? Not worth it, just fix it later. (or not at all, have you surfed the web with the console open? It is flooded with warnings and errors) But miss the critical bug in the webshop, that cause customers to legally buy for free? That is expensive.
- ajeet_dhaliwal 5y ago
  As a software engineer who has been mostly focused on testing over the last few years and leading a team focused on this, it's clear to me now that balancing what's worth testing and what's not is a delicate art and very difficult to always get right. I do believe that a common failure is not investing in adequate regression focused tests. I've been on projects were these were not in place and an update lost a company millions of dollars very quickly prior to being noticed in production, I've been on projects were delaying any further to release would have lost critical learning data during a period of time that could not be guaranteed to come again quickly (for example market volatility or turmoil). The best engineers in my opinion love testing as much as building new features because they understand that production fire fighting is the least fun activity of all. Quick plug, I believe solid test reporting makes the ROI on test development clearer. I'm founder at Tesults - a test results reporting app (https://www.tesults.com https://www.tesults.com) and I'd love your feedback on it, send me an email if you have a moment.

scotty79 5y ago

Even if 'it's orders of magnitude' it still must be orders of magnitude cheaper than for some other industry. I bet recalling millions of cars due to some small mechanical bug is very expensive.

speeder 5y ago

Or see the case of the installer on Myth 2, that you could also say the cost was infinite, since the but on the installer made the company litearlly go bankrupt and get purchased by Microsoft.

empthought 5y ago
1. It was the uninstaller, not the installer. 2. That is not at all what happened. Bungie never went bankrupt.

onionisafruit 5y ago

You need to remember the context when this study supposedly took place. "Production" used to have a literal meaning -- producing physical media. 100x was a reasonable number to believe in that case even without extreme situations like a mission to Pluto. I first learned this in QA training. It's been decades so I don't remember if 100x was mentioned. They listed all the people required to fix a bug once it had gone to production. Off the top of my head that included: - Customer support handling calls from customers -- possibly thousands of times. This isn't necessarily part of the cost of fixing the bug, but it's part of discovering a bug in production. - Some combination of managers and product managers prioritizing the bug and scheduling programmers to fix it. - Programmers finding and fixing the bug. - QA validating the fix. - Writing an installer for the fix. - QA validating the installer. - Producing the fix. - Shipping the fix to effected customers -- which may be all customers. We were being trained to do QA, so we compared that to the cost involved when QA discovers a bug. That was essentially just the cost of us writing a bug report and programmers fixing the bug.

BeefWellington 5y ago
Adding onto this, if the fix requires some kind of notice or error message detail that didn't previously exist it can also require work from your internationalization team to translate that error into N languages your software supports.
- onionisafruit 5y ago
  I did forget that. I'm pretty sure that was actually listed in the training because the company I was at made software that was translated into a lot of languages.
onionisafruit 5y ago
Replying to myself because I got too excited and hit enter. Compare the above to a modern SaaS company. Customer support handling the reports is a fraction of the cost because it doesn't require spending time on the phone with each customer. Most don't have QA, so those costs are gone. No writing and validating installers. No cost of making and shipping physical media. One other thing is that programmers were a lot cheaper then. So the time a programmer spent actually fixing the bug was even less significant.