9 ms·

The “bugs are 100x more expensive to fix in production” study might not exist

159 points by sohkamyung 5y ago

raxxorrax 5y ago
I am not sure if you can answer the question. It depends on the bug of course. If data sets of a production system are affected, it can indeed mean a lot of work For some other bugs there might not be a difference. There cannot be a general answer to this question, there are too many factors involved. Modern development makes some things easier than it was in the past. Then again there will also be new problems.
- krisoft 5y ago
  It is also very field dependent. I worked on a web project where we could deploy with a line of command. Also worked on a project where "testing the new version for real" meant that we had to charter an airplane, hire a pilot, drive to the airport, strap things to the aircraft, calibrate the system and then get airborne. Understandably this changes the cost of errors, and thus people's attitude to quality.
- db48x 5y ago
  Of course we could answer the question if we really wanted to. It’s not actually very hard to count the time spent on a bug and then multiply that by the salaries of the people involved.
  - incrudible 5y ago
    This is an oversimplification of the problem. Nobody is interested in the cost of "a bug", but their set of bugs. The cost is not just salary, it's also "missed sales". It may be the one bug, otherwise harmless, that shifts the opinion of a major decision maker. It may be the bug that causes some amount of users to finally look elsewhere, already being frustrated from previous bugs. It may be the bug that triggers a disgruntled user to write a rant on your product that ends up as a top google search result.
    - db48x 5y ago
      Direct costs are all anyone means when they say that “bugs are 100× more expensive to fix in production”. There’s nothing in that statement about indirect or reputational costs, just the cost to fix. The factors you mention all exist, but they are irrelevant to the cost of fixing a bug. They are very relevant to the cost of _failing_ to fix a bug. What is relevant is your definition of the various stages of development. The numbers that are the source of the “100×” figure are from the development of an anti–ballistic missile system from the sixties or seventies. They would have been programming in assembler (thus the “27 lines of code patched per change” is not unusual). Finding and fixing a bug in the software after you’ve built all the missiles and shipped them to every ballistic missile base in the US could indeed be very expensive! But if your release process is just copying the new code to a server (or even a fleet of servers) and restarting some services, then it’s not expensive at all. Most of us have a very different idea of what “releasing” the software means than the folks on this project had. https://gist.github.com/Morendil/f9c2e9f3f450d3a76de8aeee7cf2bd00 https://gist.github.com/Morendil/f9c2e9f3f450d3a76de8aeee7cf...
myfavoritedog 5y ago
“Water is wet” study might not even exist The fact that the sooner you fix a bug, the far-less impact it has, is logically obvious from anyone who has beginning-to-end product development, release, and maintenance experience. Whether or not a study was done to prove an obvious principle is unimportant.
jfmc 5y ago
I guess that 100x is just an arbitrary number that means 'orders of magnitude'. If production means going to Pluto on a super expensive space mission, and the bug ruins all the scientific experiments, and your future funding depends on the success of the mission, I'd say that the cost tends to infinity. If the product requires some kind of polishing that only interaction with real users can provide, then the cost of bugs at production be even negative (spend less time on the requirement side, etc.).
- ballenf 5y ago
  100x is probably conservative in an enterprise pure waterfall development environment, regardless of application.
  - dumbfounder 5y ago
    This is why so many developers can’t be left to their own devices to decide when a product is launch ready. There will always be bugs. Always. You can ship with known bugs. It’s about being pragmatic and fixing it at the right time. Not all bugs have the same impact.
    - manquer 5y ago
      You could make the exact argument about product managers.. This is why product managers / business can't be left to their own devices to decide when a product is launch ready. Not all bugs have same impact, and they are not experts to understand which are high impact. This statement is just as true as yours. Doesn't mean you should ignore either one group to make a decision.
- asah 5y ago
  Ha! you're being too kind. :-) How about life saving medical applications (insulin pumps, pacemakers etc), infrastructure (power, transportation, internet) let alone weapons of mass destruction. Then, consider subtle security holes as serious bugs to be exploited en masse. People try to air gap the hardware, and even that's not always enough (see StuxNet).
runawaybottle 5y ago
Do we even need a study for this? Production means you have limited time to make changes on there. It’s like when they went to that planet in Interstellar where each second represents a day. Time is money, you age fast on production. It will obviously be more expensive.
- tgv 5y ago
  At the risk of derailing the discussion even further, I'd say: yes, we do. There's quite a difference between 1.25x and 100x more expensive, especially if you're building the next AWS. However, such a number would need to be conditioned. I can imagine that fixing an error in the i9 microcode will be a lot costlier to fix in "production" than your average NaN JS bug.
- AmericanChopper 5y ago
  That’s an intuitive explanation for why the claim might be true. An intuitive explanation for why the claim might not be true is that production has your entire user base testing it in real time. Recreating, diagnosing, and collecting feedback on a fix would be much cheaper. Of course that doesn’t necessarily account for the expense to your customers. Which is why it’s important to have both a specific explanation for what a claim is claiming, and some empirical verification to back it up. This claim seems to have neither.
- ordu 5y ago
  Yes, we need to study this. "Obviously" is not good enough. People want to find the best use for their time, to get maximum profits. An optimization can be done for some extent by relying on "obvious", but if really you want to maximize productivity, you need to measure things. You need to go from qualitative reasoning to quantitative one. We need not just measure comparative costs of bugs in productions with bugs in development, we ways to measure amount of not found yet bugs, we need to predict speed of bug-hunting in advance, we need a lot of tools to really quantify a development process. To make a decision we need a way to predict future events, it allow to compare consequences of different decisions and to choose one wisely. > Time is money, you age fast on production. Bugs is also money. So we need a way to compare them quantitatively.
habibur 5y ago
Rather the last few bugs are 100x more expensive to find and fix. That will be this much costly even if you try to find those in pre-production.
lmilcin 5y ago
As much as I like to produce correct code in the first place, I think it is very relative and depends on the project. There are projects where nobody cares much about bugs in production and it would cost a lot to make sure no bugs enter production. These projects typically tend to rely on feedback from users. Then there are projects where production errors are just not acceptable (think Solid State Booster controller or ABS break controller). These projects tend to fail catastrophically in case of a production error. Most projects are on the spectrum but that spectrum is very, very wide. Because I like to produce correct code and I don't like to be pressured by the "business" to hurry up I tend to choose projects where errors tend to be costly if they get into production, but other people may have different preferences. Now, it is important to understand what is the cost of the bug. We kinda understand cost of production bugs (the damage to company + all actions necessary to remove the bug and any lasting effects). The cost of removing the bug before it gets into production is less well understood. If you invest in better developers or put automation or work to imbue your developers with "drive for excellence" or delay it to give them a little bit more time -- it is difficult to quantify these costs because there is no baseline. You don't know how much the project would cost if you did not do all these things and you don't know how much more crappy the end result would be.
steerablesafe 5y ago
Sorry if this is naive, but my impression is that the statement "bugs are n times more expensive to fix in production" itself is ill-defined. How do you measure it? I can think of two ways: 1. You measure the average cost of fixing bugs before they go to production. You also measure the average cost of fixing bugs after they go to production. I think this measurement is uninteresting, this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not. 2. Identify 2N bugs before going into production. Randomly select N bugs and don't go to production before they are fixed. Then go to production and fix the other N bugs after a while. I think this would be more interesting, but I suspect that nobody does this, as this measurement itself is possibly more costly than all of the 2N bugs together.
- Someone 5y ago
  “this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not.” I think you’re using the wrong definition of cost. A bug might take an engineer 3 seconds to fix, but still could cost millions if pushed to production. Certainly historically, that could mean producing thousands of CDs or floppy disks, or burning things to PROM. With products such as interplanetary probes, a bug could harm quite a few people in their career (some people do their PhD on an experiment for a Mars rover. If that crashes on landing, those designing it still have their design to show, but questions will linger whether it would have worked. Those doing a PhD on the results of that experiment truly are hosed)
  - steerablesafe 5y ago
    And a product being delayed due to unimportant bugs could also cost millions, if I use the right definition. The point still stands, how do you actually measure it?
- AstralStorm 5y ago
  It really depends on how the system is deployed. For a web service, the cost might be almost identical, except reputation damage, which is impossible to properly quantify. For a game, where almost everyone plays it on launch, if it has too many bugs, the product might be sunk, or it might be a rolling update with minor extra cost of compatibility. For embedded firmware... Well... You might never really get it fixed, as not everyone will upgrade. For mission critical software, it can literally kill people or destroy hardware, or cause huge costs.
whoisthemachine 5y ago
This article is about the poor state of research in the software field. The 'bugs are 100x more expensive to fix in production' bit is just a glaring example of that. The article isn't in any major way refuting 'bugs are 100x more expensive to fix in production', it's just stating that the study may not exist, so there's no need to defend your beliefs as many are doing in the comments.
- nerdponx 5y ago
  Heck, the subtitle even says it: "It's probably still true, though, says formal methods expert" What's more interesting anyway is the circumstances around the nonexistence of the study. It's not a case of falsified research! It's a case of someone citing apocrypha as fact, and then other people citing that citation, etc. And nobody questioned it because it's so obviously true!
  - whoisthemachine 5y ago
    Truly seems to get to the root of some of software's troubles...
  - Arnt 5y ago
    I came across it in a book that we were supposed to read at university thirty years ago, but the book was outrageously expensive. It cost more than all of my other textbooks put together, and the university book shop didn't have it and told me I'd have to wait months. It's probably different now that I can click around a little on some web sites and three days later a book arrives from New Zealand. But that was thirty years ago. Can't remember how I read that book, but I did and I know that most others in the course didn't. I find it very easy to believe that people might assume things about what it said rather than actually read the text, and hand it down to others as fact. And so what's illustrative anecdote transforms into assumedly well-researched fact.
    - Arnt 5y ago
      Replying to myself... I wonder what book that was. It was a large slim hardback published sometime in the 1970s, around A4, so much earlier than the 1987 book. I don't have any course materials any more. As I recall, they argued that making a change in the requirements specification was ~10 times cheaper than finding and fixing a bug while writing the implementation, which in turn was ~10 times cheaper than fixing it after delivery, ie. on other people's production computers at other site. There was some argument around it, but the text didn't imply exactness, as far as I can remember.
rusk 5y ago
Does this even need a study? Isn’t it simply obvious? Perhaps less so with online applications and continuous updates but in the classical model of software development that most of the industry still depends on the costs and hassles of fixing a bug will obviously increase the further a piece of code goes along the development pipeline the more people, communications and effort it requires to right it. A stitch in time saves nine goes the saying. This reality is at odds with the cult of “move fast and break things” that seems to be somewhat popular these days. In any sector where quality is a factor this will always lead to increased expense and stress. Of course if you’re just running a website and you’ve got a nice network effect going on, or you’ve got your customers locked into contracts based on matters other than the users’ satisfaction all of a sudden matters of quality become a “cost” and you all of a sudden have to come up with this hokum to try and discourage your developers from doing a good job contra their intuition. I have a close acquaintance who works in medlab and frustratingly we’ve seen this mentality creeping into the equipment even they use where the system can be down for days at a time due to some software update and then they won’t even properly resource field engineers because they then refuse to reap the costs that the 100x model predicts. Certainly they’re probably saving pennies on the dime but who ends up bearing costs but users and healthcare budgets. There are those that will quibble this is but an anecdote but you can see this kind of sloppy crack-handedness creeping in everywhere. Oh ship it today we can fix it tomorrow when we ship a whole slew of new bugs too. Sorry. A bit of a sideways rant.
- chrisseaton 5y ago
  > Does this even need a study? Isn’t it simply obvious? Actually checking what seems obvious is good science.
  - rusk 5y ago
    What goes up must come down.
    - II2II 5y ago
      Then you learn about escape velocity (to choose an example where no "tricks" are involved).
mytailorisrich 5y ago
I think it would be wrong to focus on the number. The important piece of wisdom here is that bugs get more expensive, even exponentially more expensive, to fix the later they are found. If you find a bug during unit testing while developing a feature you just fix it. If the bug is found in integration tests it may pull more resources and people to process. Even more so if found during integration tests. If found after release to customer it might get very, very expensive. The takeaway is to find and fix bugs as early as possible. If you want to put an actual number on how much more expensive, you indeed need to carry a study but that will probably vary with the field or type of products and may in the end not provide additional actionable insights. Especially, I think this is risk mitigation so one would want to consider the worst case and its likelihood and not only an average number. That's probably difficult to capture in a generic study and product teams will know best what applies to their specific product.
- ckastner 5y ago
  > I think it would be wrong to focus on the number. The important piece of wisdom here is that bugs get more expensive, even exponentially more expensive, to fix the later they are found. This is always how I interpreted it. It thought this was the obvious interpretation, since it's all but certain that no universally applicable number exists.
- TheOtherHobbes 5y ago
  They don't though. Bugs have three costs. Immediate corporate costs, corporate restitution costs, and cost-to-fix. Immediate corporate costs are loss of customers, loss of sales, and loss of goodwill leading to future loss of customers/sales. Restitution costs are costs to repair reputational damage and make good on customer losses. This element can be zero or it can be billions, depending on the exact nature of the loss. Cost-to-fix can be trivial, or it can be months of work. Cost-to-fix is unrelated to corporate costs. There is absolutely no relationship of any kind between the ease with which a bug can be fixed in code/production and the amount of corporate loss it caused. There is a very rough relationship between age and cost-to-fix, but it's very approximate. If code is documented and maintained and there are no forgotten areas written by people who have left, age effects should be minor. If the code is an impenetrable ball of mouldy spaghetti written by people who are long gone without leaving docs, you're screwed.
saint_abroad 5y ago
The "Y2K bug" cost an estimated US$300 billion in remedial work, having been in production for ~40 years. Corollary: The most expensive bugs to fix are the oldest bugs.
- protomyth 5y ago
  I do wonder what 2038 is going to bring since it goes well beyond the IT department. I get the feeling the bill is going to be massive. I'm just glad we finally were able to replace the Windows 95 box running the boilers.
- gmfawcett 5y ago
  Hm, Y2K issues were the result of conscious design trade-offs, not bugs. (Nobody ever looked back and said: oops, we accidentally left the century off of our dates.)
dumbfounder 5y ago
Obviously this depends on context, as others have stated. One other factor should be the effort of test deployment. If it is very easy to test and deploy then it is not 100x. And may even be less than 1x in certain cases when you can’t test something but your users can. Whether or not you care about the bug surfacing is a different story. Context, people.
femto 5y ago
I'd say it is an engineering truism, gleaned from experience with designing and building physical things, that has been adopted by the software community. The way I originally heard it was that problems are 1 x harder to fix during design, 10x harder during prototype and 100x during production. It's about orders of magnitude rather than the numbers. Another way of putting it is: problems get harder to fix the later you detect them.
loopz 5y ago
The key is what is the right question: The right question is not wether it's more expensive to fix bugs in production or earlier. The key question is: Who bears the brunt of the cost of fixing bugs at one stage and another? This is a parallell to Conway's law: Which cost-center benefits for NOT working with agility (not rituals) and using Dev(Sec)Ops (collaboration) to improve processes and techniques? It's that simple. Anything else will not get oxygen to thrive, simply because of no allocated priority and funds to bridge the disconnect. What to call this law? Maybe "The Law of Cost Centers", or something similar. Anyone trying to fix this flaw will be fighting windmills head-on every step of the way. The big international corporations will have budgeted this due to scale and a must-have need, to be much more available and reliable than smaller shops.
- manmal 5y ago
  Indeed. Many custom software vendors justify their high maintenance fee (in part) with "ongoing bugfixes and improvements". What would be their incentive to write higher quality software, once they've signed the deal?
  - loopz 5y ago
    This is the classical squeeze of the employees, between management and vendors, neither of which have incentives for improvements of software due to costs. So it is in many cases up to each customer to discover all bugs, and often even specify(!) technical solutions to a vendor that maybe even forgotten the domain, mix and misplaces their branches (yay, new bugs!) and require handholding and escalations to get through to. In management's eyes, this is of course all problems with employee laziness: That employees aren't fighting enough of their windmills. Documentations of testresults and bug statistics go to /dev/null. If you're really unlucky, you are expected to lobby for others' targets, but not your own. Good times.
- satyrnein 5y ago
  It seems like there's something interesting here but I'm having trouble getting from the questions to the actual answer. Could you please state directly, what is the (proposed) "Law of Cost Centers"? Sorry if I'm being dense!
ZeroGravitas 5y ago
It seems the real conclusion is that while the phrase captures some vague truth-y reality that people can identify with, it's not a scientific enough hypothesis to even be testable. If a study did exist and had the 100x figure based on real data, then it would still have been paraphrased enough to get back to the same status and no one would be aware of what the original conditions and assumptions were. So in many ways it doesn't matter if it exists and so it's non-existamce is not the problem that needs fixed, but how we document our knowledge generally.
rileymat2 5y ago
I have heard this expression for a long time, at the time it was very logically true(ish, the number is inexact), no study needed. Now with most update and delivery mechanisms it should be studied.
rubyist5eva 5y ago
Bugs in production often become features. Fancy that, zero cost to fix!
oliwarner 5y ago
Seems pretty irrelevant to the underlying truths. A spate of even small bugs can cost you customers. Big bugs can cost you colossal fines from data protection regulatory agencies, especially if you operate under a "we'll fix it later" attitude. If course there's a point of diminishing losses to aim for. Sweeping statements either way on software testing aren't helpful.
jsnell 5y ago
Just a paraphrasing of the original article, which was discussed 2 days ago (165 comments): https://news.ycombinator.com/item?id=27892615 https://news.ycombinator.com/item?id=27892615
chillingeffect 5y ago
this is the study: "If the cost of fixing a requirements error discovered during the requirements phase is defined to be 1 unit, the cost to fix that error if found during the design phase increases to 3 - 8 units; at the manufacturing/build phase, the cost to fix the error is 7 - 16 units; at the integration and test phase, the cost to fix the error becomes 21 - 78 units; and at the operations phase, the cost to fix the requirements error ranged from 29 units to more than 1500 units" [0] [0] https://ntrs.nasa.gov/citations/20100036670 https://ntrs.nasa.gov/citations/20100036670
atomicbeanie 5y ago
And I would offer does not match the experience of those working on web apps with large amounts of at-rest data. Internet connected services are subjected to constantly changing data inputs which break software unexpectedly. The only place to see these data outliers is production. The companies that make this automated, auditable, safe and repeatable win.
jackcviers3 5y ago
I thought the numbers came from: Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on Computer SE-1(4):1226-1241. As cited in "The Economic Impacts of Inadequate Infrastructure for Software Testing" by RTI. https://www.nist.gov/document/report02-3pdf https://www.nist.gov/document/report02-3pdf Pretty sure that both of those papers exist, and the NIST table cites up to 990x and refers to the 76 Boehm paper, not any IBM org.
- dwheeler 5y ago
  Yes. That first paper is Boehm, Barry W., "Software Engineering", IEEE Transactions on Computers, December 1976, pp. 1226-1241, vol. 25 DOI Bookmark: 10.1109/TC.1976.1674590, https://www.computer.org/csdl/journal/tc/1976/12/01674590/13rRUzp02n5 https://www.computer.org/csdl/journal/tc/1976/12/01674590/13... At the time Barry W. Boehm was at the TRW Systems and Energy Group. The Boehm paper says "Fig. 3 shows a summary of current experience at IBM[4], GTE[5], and TRW on the relative cost of correcting software errors as a function of the phase in which they are corrected." and indeed figure 3 (page 1228) shows exponential growth. It only shows averages or ranges for each data source, and that's a legitimate critique. That said, it does show them for multiple companies, and then presents a trend line that plausibly follows from the data provided. Boehm has a good reputation, I expect that this really was a reasonable observation from real data. It's legitimate to question whether or not that is still true. Computer "science" is notorious for having almost no science - experiments are almost non-existent. I would love to see this & many other experiments conducted to see what's true today.
  - jackcviers3 5y ago
    I bet it was true, in the context of the time: Card punches, shared compute time, physical packaging and shipments; imagine the cost of fixing a bug in a program in the rope ROM of the Apollo guidance computers (had they actually bothered fixing one). I'd love to see the study redone in our postmodern OTA, client/server, no physical media world. The physical overhead cost of fixing a product bug would likely be orders of magnitude lower than in 76, but the units deployed in the field would be many many orders of magnitude higher; so cost per unit is down, but there are waaaaay more units in production. If the bug caused a material cost to the user, I could easily see it being very expensive. As for there being no experiments; computer science is more of a mathematical discipline than a scientific one. You might say the same thing about theoretical physics, but I still think it's likely that we can't break the speed of light. At any rate, the IBM study may not have existed, but others that took me less than ten minutes of googling to find definitely did, and do show results in the same order of magnitude as the questionable IBM study. The question I have is - what are we really arguing for here by refuting this data, allowing more bugs to reach production? Having more broken code in the wild than we already do? Less rigorous development methods? There's barely any rigor in SWE as it exists today, anyway, yet things mostly work through a generous application of trial and error.
anm89 5y ago
I mean, the fact that the study came up with exactly 100x is pretty suspicious
patja 5y ago
I'm sure this study that may not have been actually performed has been loosely replicated by many managers using their own teams' actual data. I've certainly worked in organizations very familiar with figures they calculated for the cost to fix defects early vs late.
captainmuon 5y ago
It doesn't really matter to me if the study is factual, because it is not about whether this study actually took place. It is not even about having a general law that bugs are always more expensive to fix in production. It's just an idea that you can use in an argument. When you invoke this idea, what you are actually saying is: "I want to prioritize this bug now, because my experience tells me it will save cost and/or effort." You use this rhetorical device to put weight behind your argument. It's like invoking "80/20 principle" or "technical debt". Maybe I've become cynical but the longer I've been working, the more I like the postmodern theory that almost everything is a language game. I start to recognize bullshit, and to recognize when it's the right time to use BS myself. (Not saying that truth doesn't matter. Just that sometimes you have a correct but not convincing point, and you can support it with a not-100%-scientifically correct, but more convincing point.)
- piva00 5y ago
  > Maybe I've become cynical but the longer I've been working, the more I like the postmodern theory that almost everything is a language game. I start to recognize bullshit, and to recognize when it's the right time to use BS myself. (Not saying that truth doesn't matter. Just that sometimes you have a correct but not convincing point, and you can support it with a not-100%-scientifically correct, but more convincing point.) After 17 years of career I can relate to this feeling, at least the start of it. I'm not at the point where I can safely evaluate when it's the right time to bullshit myself, as a way to avoid putting effort on convincing someone about intuitive feelings I have, from all my accumulated experience, on why something is good or bad to do. I'm still at the stage where I pick my battles instead of trying to use language to empower an argument I feel strong about, I'm still too "naive and honest" and would feel it's wrong to do it. But fucking hell, how many discussions I have simply avoided or jumped out because it would take too much energy and time to explain to someone that's been too stubborn or sure about some solution or process. I have much less patience for these, I got more in the mode of "show, don't tell" and if I feel I don't have energy for a quick demo I know it's not a battle worth of my time. I think I'm still too much of a modernist and seeking truth...
- Cthulhu_ 5y ago
  And while 100x might not be true, you KNOW it's a lot more damaging, costly, and disruptive to have to fix a bug that makes it way to production - arguments in favor of testing, canary releases, code reviews, robust programming languages and architectures, etc. They won't stop all issues, but the less that make it to production the better. That one doesn't take language games or political games, I think. It shouldn't anyway.
thesz 5y ago
Let me present an axiom of software engineering: "The effort (cost) to fix a software defect is proportional to the time between defect's introduction and discovery". Please note, that the axiom does not say "100x more", it says that cost increases (usually; it may even decrease, why not? consider a bug in ransomware that prevents it from encrypting files properly). The concrete cost may be 1.5x, 10x or 1Mx, it all depends on the circumstances. And here's a pretty picture from the research paper on the subject: https://www.researchgate.net/figure/Cost-of-Fixing-a-Defect-10_fig4_228658128 https://www.researchgate.net/figure/Cost-of-Fixing-a-Defect-...
- gmfawcett 5y ago
  It might be a rule of thumb, but it's not an axiom (otherwise, there could be no counterexamples). Some software defects are only discovered years later, but can be fixed in minutes.
- Morendil 5y ago
  "Research paper", ha. I came across this a few years ago, I call it the "Frankenpaper". See this gist for why I call it that: https://gist.github.com/Morendil/85336bf97211f9f31102ce2ee4e44829 https://gist.github.com/Morendil/85336bf97211f9f31102ce2ee4e...
toolslive 5y ago
If you would try to unearth the "10x developer" study you will be in for a similar surprise.
- Cthulhu_ 5y ago
  Probably, but there's a grain of truth in it. A "1x developer" will do what they're told, write code, do the 9-5 churn, go home, sleep, repeat. A "10x developer" will / should apply a bit more critical thinking and see if there's a ready-made solution, a product, or a workaround to avoid writing code in the first place. They use their experience and, I presume, personality, to provide 10x more value to their employer or project than the head-down-and-write-code developer. Remember, code is just a tool to achieve a goal, it is not THE goal itself. At least, it shouldn't be, I've seen enough projects where they turned code and architecture into the goal itself (think: microservices well before launch, Scala to stroke own egoes and weed out mediocre developers, writing your own frameworks or ready-to-run cloud platforms, etc)
  - randomdata 5y ago
    > Remember, code is just a tool to achieve a goal, it is not THE goal itself. Which is why I find it strange that we're hired as "software developers", not "problem solvers." But since we are, it pigeonholes the solutions, and with that code does become the goal. In most shops "I'm going to need a lathe and a mill" would never fly with management, even if that is what would actually provide the best solution. Once hired as a software developer, the business expects that you will find a software development solution.
ldiracdelta 5y ago
I know that bugs in silicon are 100x more expensive to fix. Sometimes if you find a show-stopper bug on your chip after you have "taped-out" (sent the design to the fab' on magnetic tape storage), then you just go bankrupt or the team is fired because you can't pay the ~$10 million for a new silicon manufacturing mask set.
GuB-42 5y ago
I never expected a study. The neat 100x value, the lack of context, it sounds more a saying from grandma, which is wise, but not to take at face value. Sometimes (rarely) it is actually cheaper to fix in production, sometimes, it is way more than 100x. I had a coworker who worked with IC cards and fixing a bug in production meant scrapping a whole batch of cards (can be millions).
xedrac 5y ago
Dynamically typed languages are so prevalent, where the approach is "blow up at runtime". I think in some contexts, this is fine. But in other contexts, it's more like a 1000x problem. Nothing bothers me more than runtime errors that would not have been possible had a good statically typed language been used.
cestith 5y ago
The times to resolve the bug in code may not be significantly different at all. In the 1967 to 1981 period cited, the expense isn't putting it in the main branch of your repo and letting your pipeline put it everywhere. It was putting it on tape or disk, sending it to a customer, sometimes flying your own technician to the customer's site, installing it, maybe performing a backup beforehand, letting your customers know this is all necessary, publishing errata in a physical manual, and convincing these customers you still take quality very seriously.
xornox 5y ago
Let’s say that we have an error in the requirement definitions. The error may make whole product almost useless. Is it cheap to detect error from initial requirement specifications or when the product is in sales and nobody buys it because of the requirement flaw? If we make wrong kind of products in a way or another, it is always very expensive.
mumblemumble 5y ago
Even if we take for the sake of argument that they are, that's not enough information to form the foundation of a policy. You've also got to look at how much it costs to find bugs at each stage of the lifecycle, and also the direct costs of any damage the defect might cause. I also just can't believe that any of these numbers are universal constants. For example, at my current job, a bug being discovered is likely to cause a moderate amount of damage, but actually shipping a fix for it might take months. At my previous job, I think my record time to have a fix for a defect deployed to production was about 30 minutes, but I also once had a bug do more than my annual salary's worth of tangible damage in just a couple minutes. At the job before that, production defects were typically relatively low-impact and also fairly inexpensive to fix. It should come as no surprise that each of these companies had developed a dramatically different approach to quality assurance. Where I'm going with that is, it's not just that the research is bad, it's also that our ways of trying to use that research are nonsensical. Even if that study existed and were accurate, it would still be useless for most of us. Because it would only describe how things worked under one set of conditions that is almost certainly not the set of conditions under which you are operating. And yet, people often quote it without offering any real context whatsoever, as if it were some fixed constant that applies to everyone everywhere. It's not exacltly that the figure's veracity doesn't matter. Garbage in, garbage out. But if the thing you're putting stuff into and out of is something like a trash compactor, then the more immediately relevant principle is, anything in, garbage out.
wiz21c 5y ago
They may be 100x more expensive but they sure are 100x more stressful to debug :-) (especially on friday afternoon)
magwa101 5y ago
The bugs that have no effect may offset the cost. I'd like to see that study.
magwa101 5y ago
The bugs that have no effect may offset the cost. I'd like to see that study. Hardware is once, software is forever.
njharman 5y ago
The variables are way, way too carried to make any broad claims. I've fixed production bugs in literal minutes. Because the service was web API, that reported error as soon as that endpoint was hit. Sentry gave all the info I needed to know exactly what was wrong. Our procedures and tools for hotfixing made rolling out super fast. Same bug in smartform app would have been days, and dependent on user upgrades. Same bug in industrial or medical, etc proly months and much more dev effort due to testing verificatiin. Point being bug cost has more to due with platform(web, app, desktop, embedded), ops environment(game vs lives at risk), and how much effort has been prespent in clean codebase, procedures, monitoring and tools. Than dev/prod environment.
deleted 5y ago
[deleted]
ameixaseca 5y ago
Bugs are infinitely more expensive to fix in production. The best example is 737 Max - how do you quantify monetarily the lost lives? The reputation damage? The lawsuits? The future contracts that were lost? But you don't have to go to that extreme. For instance, I just heard today of a lost contract from a coworker: the client was fed up with a call management system that was dropping calls like crazy and nobody was able to fix since you could not even replicate the system (the last time they tried it, the test system pulled stuff from the production db and it basically brought everything down). How do you calculate the cost of this? Cost of debugging, lost revenue, lost referrals, brand damage, and the toll on developers and everyone doing crisis management.. I'd say 100x is a very optimistic estimate. It can very well cost much more than that. Trivial bugs are trivial bugs, and not all the bugs are the same. A misaligned table is nothing compared to a monolith/blob of 50Mb (either a binary compiled from C or minified JavaScript) that crashes with "segfault" or "undefined" but runs fine in debug mode. Bottom line: bugs in production, especially on complex systems, can be very hard to identify and fix - and sometimes can cost you your business.

159 points by sohkamyung 5y ago

I am not sure if you can answer the question. It depends on the bug of course. If data sets of a production system are affected, it can indeed mean a lot of work For some other bugs there might not be a difference. There cannot be a general answer to this question, there are too many factors involved. Modern development makes some things easier than it was in the past. Then again there will also be new problems.

krisoft 5y ago
It is also very field dependent. I worked on a web project where we could deploy with a line of command. Also worked on a project where "testing the new version for real" meant that we had to charter an airplane, hire a pilot, drive to the airport, strap things to the aircraft, calibrate the system and then get airborne. Understandably this changes the cost of errors, and thus people's attitude to quality.
db48x 5y ago
Of course we could answer the question if we really wanted to. It’s not actually very hard to count the time spent on a bug and then multiply that by the salaries of the people involved.
- incrudible 5y ago
  This is an oversimplification of the problem. Nobody is interested in the cost of "a bug", but their set of bugs. The cost is not just salary, it's also "missed sales". It may be the one bug, otherwise harmless, that shifts the opinion of a major decision maker. It may be the bug that causes some amount of users to finally look elsewhere, already being frustrated from previous bugs. It may be the bug that triggers a disgruntled user to write a rant on your product that ends up as a top google search result.
  - db48x 5y ago
    Direct costs are all anyone means when they say that “bugs are 100× more expensive to fix in production”. There’s nothing in that statement about indirect or reputational costs, just the cost to fix. The factors you mention all exist, but they are irrelevant to the cost of fixing a bug. They are very relevant to the cost of _failing_ to fix a bug. What is relevant is your definition of the various stages of development. The numbers that are the source of the “100×” figure are from the development of an anti–ballistic missile system from the sixties or seventies. They would have been programming in assembler (thus the “27 lines of code patched per change” is not unusual). Finding and fixing a bug in the software after you’ve built all the missiles and shipped them to every ballistic missile base in the US could indeed be very expensive! But if your release process is just copying the new code to a server (or even a fleet of servers) and restarting some services, then it’s not expensive at all. Most of us have a very different idea of what “releasing” the software means than the folks on this project had. https://gist.github.com/Morendil/f9c2e9f3f450d3a76de8aeee7cf2bd00 https://gist.github.com/Morendil/f9c2e9f3f450d3a76de8aeee7cf...

myfavoritedog 5y ago

“Water is wet” study might not even exist The fact that the sooner you fix a bug, the far-less impact it has, is logically obvious from anyone who has beginning-to-end product development, release, and maintenance experience. Whether or not a study was done to prove an obvious principle is unimportant.

jfmc 5y ago

I guess that 100x is just an arbitrary number that means 'orders of magnitude'. If production means going to Pluto on a super expensive space mission, and the bug ruins all the scientific experiments, and your future funding depends on the success of the mission, I'd say that the cost tends to infinity. If the product requires some kind of polishing that only interaction with real users can provide, then the cost of bugs at production be even negative (spend less time on the requirement side, etc.).

ballenf 5y ago
100x is probably conservative in an enterprise pure waterfall development environment, regardless of application.
- dumbfounder 5y ago
  This is why so many developers can’t be left to their own devices to decide when a product is launch ready. There will always be bugs. Always. You can ship with known bugs. It’s about being pragmatic and fixing it at the right time. Not all bugs have the same impact.
  - manquer 5y ago
    You could make the exact argument about product managers.. This is why product managers / business can't be left to their own devices to decide when a product is launch ready. Not all bugs have same impact, and they are not experts to understand which are high impact. This statement is just as true as yours. Doesn't mean you should ignore either one group to make a decision.
asah 5y ago
Ha! you're being too kind. :-) How about life saving medical applications (insulin pumps, pacemakers etc), infrastructure (power, transportation, internet) let alone weapons of mass destruction. Then, consider subtle security holes as serious bugs to be exploited en masse. People try to air gap the hardware, and even that's not always enough (see StuxNet).

runawaybottle 5y ago

Do we even need a study for this? Production means you have limited time to make changes on there. It’s like when they went to that planet in Interstellar where each second represents a day. Time is money, you age fast on production. It will obviously be more expensive.

tgv 5y ago
At the risk of derailing the discussion even further, I'd say: yes, we do. There's quite a difference between 1.25x and 100x more expensive, especially if you're building the next AWS. However, such a number would need to be conditioned. I can imagine that fixing an error in the i9 microcode will be a lot costlier to fix in "production" than your average NaN JS bug.
AmericanChopper 5y ago
That’s an intuitive explanation for why the claim might be true. An intuitive explanation for why the claim might not be true is that production has your entire user base testing it in real time. Recreating, diagnosing, and collecting feedback on a fix would be much cheaper. Of course that doesn’t necessarily account for the expense to your customers. Which is why it’s important to have both a specific explanation for what a claim is claiming, and some empirical verification to back it up. This claim seems to have neither.
ordu 5y ago
Yes, we need to study this. "Obviously" is not good enough. People want to find the best use for their time, to get maximum profits. An optimization can be done for some extent by relying on "obvious", but if really you want to maximize productivity, you need to measure things. You need to go from qualitative reasoning to quantitative one. We need not just measure comparative costs of bugs in productions with bugs in development, we ways to measure amount of not found yet bugs, we need to predict speed of bug-hunting in advance, we need a lot of tools to really quantify a development process. To make a decision we need a way to predict future events, it allow to compare consequences of different decisions and to choose one wisely. > Time is money, you age fast on production. Bugs is also money. So we need a way to compare them quantitatively.

habibur 5y ago

Rather the last few bugs are 100x more expensive to find and fix. That will be this much costly even if you try to find those in pre-production.

lmilcin 5y ago

As much as I like to produce correct code in the first place, I think it is very relative and depends on the project. There are projects where nobody cares much about bugs in production and it would cost a lot to make sure no bugs enter production. These projects typically tend to rely on feedback from users. Then there are projects where production errors are just not acceptable (think Solid State Booster controller or ABS break controller). These projects tend to fail catastrophically in case of a production error. Most projects are on the spectrum but that spectrum is very, very wide. Because I like to produce correct code and I don't like to be pressured by the "business" to hurry up I tend to choose projects where errors tend to be costly if they get into production, but other people may have different preferences. Now, it is important to understand what is the cost of the bug. We kinda understand cost of production bugs (the damage to company + all actions necessary to remove the bug and any lasting effects). The cost of removing the bug before it gets into production is less well understood. If you invest in better developers or put automation or work to imbue your developers with "drive for excellence" or delay it to give them a little bit more time -- it is difficult to quantify these costs because there is no baseline. You don't know how much the project would cost if you did not do all these things and you don't know how much more crappy the end result would be.

steerablesafe 5y ago

Sorry if this is naive, but my impression is that the statement "bugs are n times more expensive to fix in production" itself is ill-defined. How do you measure it? I can think of two ways: 1. You measure the average cost of fixing bugs before they go to production. You also measure the average cost of fixing bugs after they go to production. I think this measurement is uninteresting, this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not. 2. Identify 2N bugs before going into production. Randomly select N bugs and don't go to production before they are fixed. Then go to production and fix the other N bugs after a while. I think this would be more interesting, but I suspect that nobody does this, as this measurement itself is possibly more costly than all of the 2N bugs together.

Someone 5y ago
“this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not.” I think you’re using the wrong definition of cost. A bug might take an engineer 3 seconds to fix, but still could cost millions if pushed to production. Certainly historically, that could mean producing thousands of CDs or floppy disks, or burning things to PROM. With products such as interplanetary probes, a bug could harm quite a few people in their career (some people do their PhD on an experiment for a Mars rover. If that crashes on landing, those designing it still have their design to show, but questions will linger whether it would have worked. Those doing a PhD on the results of that experiment truly are hosed)
- steerablesafe 5y ago
  And a product being delayed due to unimportant bugs could also cost millions, if I use the right definition. The point still stands, how do you actually measure it?
AstralStorm 5y ago
It really depends on how the system is deployed. For a web service, the cost might be almost identical, except reputation damage, which is impossible to properly quantify. For a game, where almost everyone plays it on launch, if it has too many bugs, the product might be sunk, or it might be a rolling update with minor extra cost of compatibility. For embedded firmware... Well... You might never really get it fixed, as not everyone will upgrade. For mission critical software, it can literally kill people or destroy hardware, or cause huge costs.

whoisthemachine 5y ago

This article is about the poor state of research in the software field. The 'bugs are 100x more expensive to fix in production' bit is just a glaring example of that. The article isn't in any major way refuting 'bugs are 100x more expensive to fix in production', it's just stating that the study may not exist, so there's no need to defend your beliefs as many are doing in the comments.

nerdponx 5y ago
Heck, the subtitle even says it: "It's probably still true, though, says formal methods expert" What's more interesting anyway is the circumstances around the nonexistence of the study. It's not a case of falsified research! It's a case of someone citing apocrypha as fact, and then other people citing that citation, etc. And nobody questioned it because it's so obviously true!
- whoisthemachine 5y ago
  Truly seems to get to the root of some of software's troubles...
- Arnt 5y ago
  I came across it in a book that we were supposed to read at university thirty years ago, but the book was outrageously expensive. It cost more than all of my other textbooks put together, and the university book shop didn't have it and told me I'd have to wait months. It's probably different now that I can click around a little on some web sites and three days later a book arrives from New Zealand. But that was thirty years ago. Can't remember how I read that book, but I did and I know that most others in the course didn't. I find it very easy to believe that people might assume things about what it said rather than actually read the text, and hand it down to others as fact. And so what's illustrative anecdote transforms into assumedly well-researched fact.
  - Arnt 5y ago
    Replying to myself... I wonder what book that was. It was a large slim hardback published sometime in the 1970s, around A4, so much earlier than the 1987 book. I don't have any course materials any more. As I recall, they argued that making a change in the requirements specification was ~10 times cheaper than finding and fixing a bug while writing the implementation, which in turn was ~10 times cheaper than fixing it after delivery, ie. on other people's production computers at other site. There was some argument around it, but the text didn't imply exactness, as far as I can remember.

rusk 5y ago

Does this even need a study? Isn’t it simply obvious? Perhaps less so with online applications and continuous updates but in the classical model of software development that most of the industry still depends on the costs and hassles of fixing a bug will obviously increase the further a piece of code goes along the development pipeline the more people, communications and effort it requires to right it. A stitch in time saves nine goes the saying. This reality is at odds with the cult of “move fast and break things” that seems to be somewhat popular these days. In any sector where quality is a factor this will always lead to increased expense and stress. Of course if you’re just running a website and you’ve got a nice network effect going on, or you’ve got your customers locked into contracts based on matters other than the users’ satisfaction all of a sudden matters of quality become a “cost” and you all of a sudden have to come up with this hokum to try and discourage your developers from doing a good job contra their intuition. I have a close acquaintance who works in medlab and frustratingly we’ve seen this mentality creeping into the equipment even they use where the system can be down for days at a time due to some software update and then they won’t even properly resource field engineers because they then refuse to reap the costs that the 100x model predicts. Certainly they’re probably saving pennies on the dime but who ends up bearing costs but users and healthcare budgets. There are those that will quibble this is but an anecdote but you can see this kind of sloppy crack-handedness creeping in everywhere. Oh ship it today we can fix it tomorrow when we ship a whole slew of new bugs too. Sorry. A bit of a sideways rant.

chrisseaton 5y ago
> Does this even need a study? Isn’t it simply obvious? Actually checking what seems obvious is good science.
- rusk 5y ago
  What goes up must come down.
  - II2II 5y ago
    Then you learn about escape velocity (to choose an example where no "tricks" are involved).

mytailorisrich 5y ago

I think it would be wrong to focus on the number. The important piece of wisdom here is that bugs get more expensive, even exponentially more expensive, to fix the later they are found. If you find a bug during unit testing while developing a feature you just fix it. If the bug is found in integration tests it may pull more resources and people to process. Even more so if found during integration tests. If found after release to customer it might get very, very expensive. The takeaway is to find and fix bugs as early as possible. If you want to put an actual number on how much more expensive, you indeed need to carry a study but that will probably vary with the field or type of products and may in the end not provide additional actionable insights. Especially, I think this is risk mitigation so one would want to consider the worst case and its likelihood and not only an average number. That's probably difficult to capture in a generic study and product teams will know best what applies to their specific product.

ckastner 5y ago
> I think it would be wrong to focus on the number. The important piece of wisdom here is that bugs get more expensive, even exponentially more expensive, to fix the later they are found. This is always how I interpreted it. It thought this was the obvious interpretation, since it's all but certain that no universally applicable number exists.
TheOtherHobbes 5y ago
They don't though. Bugs have three costs. Immediate corporate costs, corporate restitution costs, and cost-to-fix. Immediate corporate costs are loss of customers, loss of sales, and loss of goodwill leading to future loss of customers/sales. Restitution costs are costs to repair reputational damage and make good on customer losses. This element can be zero or it can be billions, depending on the exact nature of the loss. Cost-to-fix can be trivial, or it can be months of work. Cost-to-fix is unrelated to corporate costs. There is absolutely no relationship of any kind between the ease with which a bug can be fixed in code/production and the amount of corporate loss it caused. There is a very rough relationship between age and cost-to-fix, but it's very approximate. If code is documented and maintained and there are no forgotten areas written by people who have left, age effects should be minor. If the code is an impenetrable ball of mouldy spaghetti written by people who are long gone without leaving docs, you're screwed.

saint_abroad 5y ago

The "Y2K bug" cost an estimated US$300 billion in remedial work, having been in production for ~40 years. Corollary: The most expensive bugs to fix are the oldest bugs.

protomyth 5y ago
I do wonder what 2038 is going to bring since it goes well beyond the IT department. I get the feeling the bill is going to be massive. I'm just glad we finally were able to replace the Windows 95 box running the boilers.
gmfawcett 5y ago
Hm, Y2K issues were the result of conscious design trade-offs, not bugs. (Nobody ever looked back and said: oops, we accidentally left the century off of our dates.)

dumbfounder 5y ago

Obviously this depends on context, as others have stated. One other factor should be the effort of test deployment. If it is very easy to test and deploy then it is not 100x. And may even be less than 1x in certain cases when you can’t test something but your users can. Whether or not you care about the bug surfacing is a different story. Context, people.

femto 5y ago

I'd say it is an engineering truism, gleaned from experience with designing and building physical things, that has been adopted by the software community. The way I originally heard it was that problems are 1 x harder to fix during design, 10x harder during prototype and 100x during production. It's about orders of magnitude rather than the numbers. Another way of putting it is: problems get harder to fix the later you detect them.

loopz 5y ago

The key is what is the right question: The right question is not wether it's more expensive to fix bugs in production or earlier. The key question is: Who bears the brunt of the cost of fixing bugs at one stage and another? This is a parallell to Conway's law: Which cost-center benefits for NOT working with agility (not rituals) and using Dev(Sec)Ops (collaboration) to improve processes and techniques? It's that simple. Anything else will not get oxygen to thrive, simply because of no allocated priority and funds to bridge the disconnect. What to call this law? Maybe "The Law of Cost Centers", or something similar. Anyone trying to fix this flaw will be fighting windmills head-on every step of the way. The big international corporations will have budgeted this due to scale and a must-have need, to be much more available and reliable than smaller shops.

manmal 5y ago
Indeed. Many custom software vendors justify their high maintenance fee (in part) with "ongoing bugfixes and improvements". What would be their incentive to write higher quality software, once they've signed the deal?
- loopz 5y ago
  This is the classical squeeze of the employees, between management and vendors, neither of which have incentives for improvements of software due to costs. So it is in many cases up to each customer to discover all bugs, and often even specify(!) technical solutions to a vendor that maybe even forgotten the domain, mix and misplaces their branches (yay, new bugs!) and require handholding and escalations to get through to. In management's eyes, this is of course all problems with employee laziness: That employees aren't fighting enough of their windmills. Documentations of testresults and bug statistics go to /dev/null. If you're really unlucky, you are expected to lobby for others' targets, but not your own. Good times.
satyrnein 5y ago
It seems like there's something interesting here but I'm having trouble getting from the questions to the actual answer. Could you please state directly, what is the (proposed) "Law of Cost Centers"? Sorry if I'm being dense!

ZeroGravitas 5y ago

It seems the real conclusion is that while the phrase captures some vague truth-y reality that people can identify with, it's not a scientific enough hypothesis to even be testable. If a study did exist and had the 100x figure based on real data, then it would still have been paraphrased enough to get back to the same status and no one would be aware of what the original conditions and assumptions were. So in many ways it doesn't matter if it exists and so it's non-existamce is not the problem that needs fixed, but how we document our knowledge generally.

rileymat2 5y ago

I have heard this expression for a long time, at the time it was very logically true(ish, the number is inexact), no study needed. Now with most update and delivery mechanisms it should be studied.

rubyist5eva 5y ago

Bugs in production often become features. Fancy that, zero cost to fix!

oliwarner 5y ago

Seems pretty irrelevant to the underlying truths. A spate of even small bugs can cost you customers. Big bugs can cost you colossal fines from data protection regulatory agencies, especially if you operate under a "we'll fix it later" attitude. If course there's a point of diminishing losses to aim for. Sweeping statements either way on software testing aren't helpful.

jsnell 5y ago

Just a paraphrasing of the original article, which was discussed 2 days ago (165 comments): https://news.ycombinator.com/item?id=27892615 https://news.ycombinator.com/item?id=27892615

chillingeffect 5y ago

this is the study: "If the cost of fixing a requirements error discovered during the requirements phase is defined to be 1 unit, the cost to fix that error if found during the design phase increases to 3 - 8 units; at the manufacturing/build phase, the cost to fix the error is 7 - 16 units; at the integration and test phase, the cost to fix the error becomes 21 - 78 units; and at the operations phase, the cost to fix the requirements error ranged from 29 units to more than 1500 units" [0] [0] https://ntrs.nasa.gov/citations/20100036670 https://ntrs.nasa.gov/citations/20100036670

atomicbeanie 5y ago

And I would offer does not match the experience of those working on web apps with large amounts of at-rest data. Internet connected services are subjected to constantly changing data inputs which break software unexpectedly. The only place to see these data outliers is production. The companies that make this automated, auditable, safe and repeatable win.

jackcviers3 5y ago

I thought the numbers came from: Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on Computer SE-1(4):1226-1241. As cited in "The Economic Impacts of Inadequate Infrastructure for Software Testing" by RTI. https://www.nist.gov/document/report02-3pdf https://www.nist.gov/document/report02-3pdf Pretty sure that both of those papers exist, and the NIST table cites up to 990x and refers to the 76 Boehm paper, not any IBM org.

dwheeler 5y ago
Yes. That first paper is Boehm, Barry W., "Software Engineering", IEEE Transactions on Computers, December 1976, pp. 1226-1241, vol. 25 DOI Bookmark: 10.1109/TC.1976.1674590, https://www.computer.org/csdl/journal/tc/1976/12/01674590/13rRUzp02n5 https://www.computer.org/csdl/journal/tc/1976/12/01674590/13... At the time Barry W. Boehm was at the TRW Systems and Energy Group. The Boehm paper says "Fig. 3 shows a summary of current experience at IBM[4], GTE[5], and TRW on the relative cost of correcting software errors as a function of the phase in which they are corrected." and indeed figure 3 (page 1228) shows exponential growth. It only shows averages or ranges for each data source, and that's a legitimate critique. That said, it does show them for multiple companies, and then presents a trend line that plausibly follows from the data provided. Boehm has a good reputation, I expect that this really was a reasonable observation from real data. It's legitimate to question whether or not that is still true. Computer "science" is notorious for having almost no science - experiments are almost non-existent. I would love to see this & many other experiments conducted to see what's true today.
- jackcviers3 5y ago
  I bet it was true, in the context of the time: Card punches, shared compute time, physical packaging and shipments; imagine the cost of fixing a bug in a program in the rope ROM of the Apollo guidance computers (had they actually bothered fixing one). I'd love to see the study redone in our postmodern OTA, client/server, no physical media world. The physical overhead cost of fixing a product bug would likely be orders of magnitude lower than in 76, but the units deployed in the field would be many many orders of magnitude higher; so cost per unit is down, but there are waaaaay more units in production. If the bug caused a material cost to the user, I could easily see it being very expensive. As for there being no experiments; computer science is more of a mathematical discipline than a scientific one. You might say the same thing about theoretical physics, but I still think it's likely that we can't break the speed of light. At any rate, the IBM study may not have existed, but others that took me less than ten minutes of googling to find definitely did, and do show results in the same order of magnitude as the questionable IBM study. The question I have is - what are we really arguing for here by refuting this data, allowing more bugs to reach production? Having more broken code in the wild than we already do? Less rigorous development methods? There's barely any rigor in SWE as it exists today, anyway, yet things mostly work through a generous application of trial and error.

anm89 5y ago

I mean, the fact that the study came up with exactly 100x is pretty suspicious

patja 5y ago

I'm sure this study that may not have been actually performed has been loosely replicated by many managers using their own teams' actual data. I've certainly worked in organizations very familiar with figures they calculated for the cost to fix defects early vs late.

captainmuon 5y ago

It doesn't really matter to me if the study is factual, because it is not about whether this study actually took place. It is not even about having a general law that bugs are always more expensive to fix in production. It's just an idea that you can use in an argument. When you invoke this idea, what you are actually saying is: "I want to prioritize this bug now, because my experience tells me it will save cost and/or effort." You use this rhetorical device to put weight behind your argument. It's like invoking "80/20 principle" or "technical debt". Maybe I've become cynical but the longer I've been working, the more I like the postmodern theory that almost everything is a language game. I start to recognize bullshit, and to recognize when it's the right time to use BS myself. (Not saying that truth doesn't matter. Just that sometimes you have a correct but not convincing point, and you can support it with a not-100%-scientifically correct, but more convincing point.)

piva00 5y ago
> Maybe I've become cynical but the longer I've been working, the more I like the postmodern theory that almost everything is a language game. I start to recognize bullshit, and to recognize when it's the right time to use BS myself. (Not saying that truth doesn't matter. Just that sometimes you have a correct but not convincing point, and you can support it with a not-100%-scientifically correct, but more convincing point.) After 17 years of career I can relate to this feeling, at least the start of it. I'm not at the point where I can safely evaluate when it's the right time to bullshit myself, as a way to avoid putting effort on convincing someone about intuitive feelings I have, from all my accumulated experience, on why something is good or bad to do. I'm still at the stage where I pick my battles instead of trying to use language to empower an argument I feel strong about, I'm still too "naive and honest" and would feel it's wrong to do it. But fucking hell, how many discussions I have simply avoided or jumped out because it would take too much energy and time to explain to someone that's been too stubborn or sure about some solution or process. I have much less patience for these, I got more in the mode of "show, don't tell" and if I feel I don't have energy for a quick demo I know it's not a battle worth of my time. I think I'm still too much of a modernist and seeking truth...
Cthulhu_ 5y ago
And while 100x might not be true, you KNOW it's a lot more damaging, costly, and disruptive to have to fix a bug that makes it way to production - arguments in favor of testing, canary releases, code reviews, robust programming languages and architectures, etc. They won't stop all issues, but the less that make it to production the better. That one doesn't take language games or political games, I think. It shouldn't anyway.

thesz 5y ago

Let me present an axiom of software engineering: "The effort (cost) to fix a software defect is proportional to the time between defect's introduction and discovery". Please note, that the axiom does not say "100x more", it says that cost increases (usually; it may even decrease, why not? consider a bug in ransomware that prevents it from encrypting files properly). The concrete cost may be 1.5x, 10x or 1Mx, it all depends on the circumstances. And here's a pretty picture from the research paper on the subject: https://www.researchgate.net/figure/Cost-of-Fixing-a-Defect-10_fig4_228658128 https://www.researchgate.net/figure/Cost-of-Fixing-a-Defect-...

gmfawcett 5y ago
It might be a rule of thumb, but it's not an axiom (otherwise, there could be no counterexamples). Some software defects are only discovered years later, but can be fixed in minutes.
Morendil 5y ago
"Research paper", ha. I came across this a few years ago, I call it the "Frankenpaper". See this gist for why I call it that: https://gist.github.com/Morendil/85336bf97211f9f31102ce2ee4e44829 https://gist.github.com/Morendil/85336bf97211f9f31102ce2ee4e...

toolslive 5y ago

If you would try to unearth the "10x developer" study you will be in for a similar surprise.

Cthulhu_ 5y ago
Probably, but there's a grain of truth in it. A "1x developer" will do what they're told, write code, do the 9-5 churn, go home, sleep, repeat. A "10x developer" will / should apply a bit more critical thinking and see if there's a ready-made solution, a product, or a workaround to avoid writing code in the first place. They use their experience and, I presume, personality, to provide 10x more value to their employer or project than the head-down-and-write-code developer. Remember, code is just a tool to achieve a goal, it is not THE goal itself. At least, it shouldn't be, I've seen enough projects where they turned code and architecture into the goal itself (think: microservices well before launch, Scala to stroke own egoes and weed out mediocre developers, writing your own frameworks or ready-to-run cloud platforms, etc)
- randomdata 5y ago
  > Remember, code is just a tool to achieve a goal, it is not THE goal itself. Which is why I find it strange that we're hired as "software developers", not "problem solvers." But since we are, it pigeonholes the solutions, and with that code does become the goal. In most shops "I'm going to need a lathe and a mill" would never fly with management, even if that is what would actually provide the best solution. Once hired as a software developer, the business expects that you will find a software development solution.

ldiracdelta 5y ago

I know that bugs in silicon are 100x more expensive to fix. Sometimes if you find a show-stopper bug on your chip after you have "taped-out" (sent the design to the fab' on magnetic tape storage), then you just go bankrupt or the team is fired because you can't pay the ~$10 million for a new silicon manufacturing mask set.

GuB-42 5y ago

I never expected a study. The neat 100x value, the lack of context, it sounds more a saying from grandma, which is wise, but not to take at face value. Sometimes (rarely) it is actually cheaper to fix in production, sometimes, it is way more than 100x. I had a coworker who worked with IC cards and fixing a bug in production meant scrapping a whole batch of cards (can be millions).

xedrac 5y ago

Dynamically typed languages are so prevalent, where the approach is "blow up at runtime". I think in some contexts, this is fine. But in other contexts, it's more like a 1000x problem. Nothing bothers me more than runtime errors that would not have been possible had a good statically typed language been used.

cestith 5y ago

The times to resolve the bug in code may not be significantly different at all. In the 1967 to 1981 period cited, the expense isn't putting it in the main branch of your repo and letting your pipeline put it everywhere. It was putting it on tape or disk, sending it to a customer, sometimes flying your own technician to the customer's site, installing it, maybe performing a backup beforehand, letting your customers know this is all necessary, publishing errata in a physical manual, and convincing these customers you still take quality very seriously.

xornox 5y ago

Let’s say that we have an error in the requirement definitions. The error may make whole product almost useless. Is it cheap to detect error from initial requirement specifications or when the product is in sales and nobody buys it because of the requirement flaw? If we make wrong kind of products in a way or another, it is always very expensive.

mumblemumble 5y ago

Even if we take for the sake of argument that they are, that's not enough information to form the foundation of a policy. You've also got to look at how much it costs to find bugs at each stage of the lifecycle, and also the direct costs of any damage the defect might cause. I also just can't believe that any of these numbers are universal constants. For example, at my current job, a bug being discovered is likely to cause a moderate amount of damage, but actually shipping a fix for it might take months. At my previous job, I think my record time to have a fix for a defect deployed to production was about 30 minutes, but I also once had a bug do more than my annual salary's worth of tangible damage in just a couple minutes. At the job before that, production defects were typically relatively low-impact and also fairly inexpensive to fix. It should come as no surprise that each of these companies had developed a dramatically different approach to quality assurance. Where I'm going with that is, it's not just that the research is bad, it's also that our ways of trying to use that research are nonsensical. Even if that study existed and were accurate, it would still be useless for most of us. Because it would only describe how things worked under one set of conditions that is almost certainly not the set of conditions under which you are operating. And yet, people often quote it without offering any real context whatsoever, as if it were some fixed constant that applies to everyone everywhere. It's not exacltly that the figure's veracity doesn't matter. Garbage in, garbage out. But if the thing you're putting stuff into and out of is something like a trash compactor, then the more immediately relevant principle is, anything in, garbage out.

wiz21c 5y ago

They may be 100x more expensive but they sure are 100x more stressful to debug :-) (especially on friday afternoon)

magwa101 5y ago

The bugs that have no effect may offset the cost. I'd like to see that study.

magwa101 5y ago

The bugs that have no effect may offset the cost. I'd like to see that study. Hardware is once, software is forever.

njharman 5y ago

The variables are way, way too carried to make any broad claims. I've fixed production bugs in literal minutes. Because the service was web API, that reported error as soon as that endpoint was hit. Sentry gave all the info I needed to know exactly what was wrong. Our procedures and tools for hotfixing made rolling out super fast. Same bug in smartform app would have been days, and dependent on user upgrades. Same bug in industrial or medical, etc proly months and much more dev effort due to testing verificatiin. Point being bug cost has more to due with platform(web, app, desktop, embedded), ops environment(game vs lives at risk), and how much effort has been prespent in clean codebase, procedures, monitoring and tools. Than dev/prod environment.

deleted 5y ago

[deleted]

ameixaseca 5y ago

Bugs are infinitely more expensive to fix in production. The best example is 737 Max - how do you quantify monetarily the lost lives? The reputation damage? The lawsuits? The future contracts that were lost? But you don't have to go to that extreme. For instance, I just heard today of a lost contract from a coworker: the client was fed up with a call management system that was dropping calls like crazy and nobody was able to fix since you could not even replicate the system (the last time they tried it, the test system pulled stuff from the production db and it basically brought everything down). How do you calculate the cost of this? Cost of debugging, lost revenue, lost referrals, brand damage, and the toll on developers and everyone doing crisis management.. I'd say 100x is a very optimistic estimate. It can very well cost much more than that. Trivial bugs are trivial bugs, and not all the bugs are the same. A misaligned table is nothing compared to a monolith/blob of 50Mb (either a binary compiled from C or minified JavaScript) that crashes with "segfault" or "undefined" but runs fine in debug mode. Bottom line: bugs in production, especially on complex systems, can be very hard to identify and fix - and sometimes can cost you your business.