4 ms·

The strange case of the inverted chart

65 points by gjm11 15y ago

Maxious 15y ago
Somehow I think this post would have been more valuable if it hadn't played stupid games about imagination. Sorry to spoil it but the punchline is that the study that everyone claims proves "[software bugs] detected during system testing cost 10 times more to fix than in the construction phase and 10-25 times more to fix when detected post-release" actually didn't prove that and software engineering academics have been parroting it because they're citing a powerpoint presentation that doesn't mention the caveats of the study. Would it have been so hard to lead with that fact?
- hammerdr 15y ago
  It isn't entirely false: If one assumes that bugs are introduced at construction (the writing of the software) then you can use that as a fixed point to determine the cost of bugs based on when they are found. Which, using the original study, would mean that in testing the bugs would be more costly and that in production the bugs would be even more costly. What is interesting is that it is a function of time and not phase. So, for example, if you were continuously deploying code and a QA found a bug the same day it would effectively be the same cost (assuming everything else is equal) as a code review done by a developer. It would also be significantly less than a bug found several months down the pipeline once your code was finally integrated into a QA environment in a non CD example.
  - barefoot 15y ago
    Agreed: When I work on a block of code I am quickest at making changes to it within the hour (as I still have it's design and structure loaded into my brain). I'm slower to make the same changes after a few days or a week, and even more so after six months. My experience is that I'm not alone. Well structured code, unit tests, and documentation help - but will never eliminate the work of loading the system into your brain again.
  - Morendil 15y ago
    Counterexample: the Debian bug of 2008 lingered in the code base for years before being detected, but the fix was still a one-liner (as was the original bug).
- jerf 15y ago
  In the context of lesswrong.com, that's a fine post. You are certainly correct that in another forum, and particularly in this one, the much shorter version would also be valuable.
  - MaxGabriel 15y ago
    That is the worst aspect of LessWrong. Unnecessarily long with huge analogies.
- SkyMarshal 15y ago
  >Would it have been so hard to lead with that fact? Good observation. I find that the most engaging writing is when someone writes the standard intro/problem statement->supporting data->conclusion, and then completely inverts it. Assert the final conclusion in the very first sentence, then work backwards through the supporting data to the problem statement. Quick readers will usually already be able to extrapolate the problem from the conclusion, and then can either move on with just that executive summary in mind, or can read through the supporting data and actively evaluate it. Both quicker and more engaging overall. Instead it seems we're taught to ramble on and on in an attempt to build up to some grand conclusion, but in this day and age of information overload, few people have the time, patience, and/or attention span to ingest all that, at least in the form of a blog post (books, academic articles are a different case).
  - gwern 15y ago
    Yes, but the point is not to simply state 'this common citation in software engineering is bullshit, which indicates they're all idiots and full of bullshit', but to get the reader to actually think about and agree to the logical argument before revealing the specifics of it being software engineering - before readers can engage in self-justification and criticism and confirmation bias 'oh we're special it's not as simple as you think'. It's like if you took a speech by a politician, removed the obvious identifiers, and showed it to his enemies who agree with it, and then reveal who it was by - you've demonstrated something interesting about human biases. And his enemies may even learn something.
JanezStupar 15y ago
The good news is that most people reading and using the data will be too illiterate to know the difference.
SoftwareMaven 15y ago
""defects detected in the operations phase (once software is in the field) cost more to fix the earlier they were introduced""" I've never heard it phrased that way "in industry" (as opposed to academia, where I have no experience). Instead, what I've heard is: ""defects detected in the operations phase (once software is in the field) cost more to fix than those detected and fixed earlier""" While perhaps not backed up with published results (again, I don't know), just by 1) taking the actual development cost of fixing a bug as constant (which is extremely generous for comparing a bug in dev versus production) and 2) then counting the number of other people involved in identifying and resolving the bug shows it to be true.
eps 15y ago
By the way, regarding inverted charts per se. There is an anecdote of one physicist showing an X-Y chart to another. Latter elegantly explains the phenomena, but then the first guy says "oh, you are looking at it upside-down", to which the other guy says - "ah, now it makes even more sense."
- gjm11 15y ago
  That's a failure mode that's had quite a bit of attention at Less Wrong; see e.g. http://lesswrong.com/lw/ia/focus_your_uncertainty/ http://lesswrong.com/lw/ia/focus_your_uncertainty/ and http://lesswrong.com/lw/ii/conservation_of_expected_evidence/ http://lesswrong.com/lw/ii/conservation_of_expected_evidence... .
ThomPete 15y ago
Feynman have a great little story about checking the facts behind the facts. Anyone got a link to that?
Drbble 15y ago
Has anyone ever made the claim that LW is lampooning? I have only heard that testing early is important because it is expensive to fix mistakes after development, and more expensive after release. I guess the straw man argument is that design bugs are worse than implementation bugs in a project, but that's also true. In both cases, the issue is that time cures all wounds, where cure is used in the sense of made permanent, not fixed, or fixed is used on the sense of locked, not restored to working order.

65 points by gjm11 15y ago

Somehow I think this post would have been more valuable if it hadn't played stupid games about imagination. Sorry to spoil it but the punchline is that the study that everyone claims proves "[software bugs] detected during system testing cost 10 times more to fix than in the construction phase and 10-25 times more to fix when detected post-release" actually didn't prove that and software engineering academics have been parroting it because they're citing a powerpoint presentation that doesn't mention the caveats of the study. Would it have been so hard to lead with that fact?

hammerdr 15y ago
It isn't entirely false: If one assumes that bugs are introduced at construction (the writing of the software) then you can use that as a fixed point to determine the cost of bugs based on when they are found. Which, using the original study, would mean that in testing the bugs would be more costly and that in production the bugs would be even more costly. What is interesting is that it is a function of time and not phase. So, for example, if you were continuously deploying code and a QA found a bug the same day it would effectively be the same cost (assuming everything else is equal) as a code review done by a developer. It would also be significantly less than a bug found several months down the pipeline once your code was finally integrated into a QA environment in a non CD example.
- barefoot 15y ago
  Agreed: When I work on a block of code I am quickest at making changes to it within the hour (as I still have it's design and structure loaded into my brain). I'm slower to make the same changes after a few days or a week, and even more so after six months. My experience is that I'm not alone. Well structured code, unit tests, and documentation help - but will never eliminate the work of loading the system into your brain again.
- Morendil 15y ago
  Counterexample: the Debian bug of 2008 lingered in the code base for years before being detected, but the fix was still a one-liner (as was the original bug).
jerf 15y ago
In the context of lesswrong.com, that's a fine post. You are certainly correct that in another forum, and particularly in this one, the much shorter version would also be valuable.
- MaxGabriel 15y ago
  That is the worst aspect of LessWrong. Unnecessarily long with huge analogies.
SkyMarshal 15y ago
>Would it have been so hard to lead with that fact? Good observation. I find that the most engaging writing is when someone writes the standard intro/problem statement->supporting data->conclusion, and then completely inverts it. Assert the final conclusion in the very first sentence, then work backwards through the supporting data to the problem statement. Quick readers will usually already be able to extrapolate the problem from the conclusion, and then can either move on with just that executive summary in mind, or can read through the supporting data and actively evaluate it. Both quicker and more engaging overall. Instead it seems we're taught to ramble on and on in an attempt to build up to some grand conclusion, but in this day and age of information overload, few people have the time, patience, and/or attention span to ingest all that, at least in the form of a blog post (books, academic articles are a different case).
- gwern 15y ago
  Yes, but the point is not to simply state 'this common citation in software engineering is bullshit, which indicates they're all idiots and full of bullshit', but to get the reader to actually think about and agree to the logical argument before revealing the specifics of it being software engineering - before readers can engage in self-justification and criticism and confirmation bias 'oh we're special it's not as simple as you think'. It's like if you took a speech by a politician, removed the obvious identifiers, and showed it to his enemies who agree with it, and then reveal who it was by - you've demonstrated something interesting about human biases. And his enemies may even learn something.

JanezStupar 15y ago

The good news is that most people reading and using the data will be too illiterate to know the difference.

SoftwareMaven 15y ago

""defects detected in the operations phase (once software is in the field) cost more to fix the earlier they were introduced""" I've never heard it phrased that way "in industry" (as opposed to academia, where I have no experience). Instead, what I've heard is: ""defects detected in the operations phase (once software is in the field) cost more to fix than those detected and fixed earlier""" While perhaps not backed up with published results (again, I don't know), just by 1) taking the actual development cost of fixing a bug as constant (which is extremely generous for comparing a bug in dev versus production) and 2) then counting the number of other people involved in identifying and resolving the bug shows it to be true.

eps 15y ago

By the way, regarding inverted charts per se. There is an anecdote of one physicist showing an X-Y chart to another. Latter elegantly explains the phenomena, but then the first guy says "oh, you are looking at it upside-down", to which the other guy says - "ah, now it makes even more sense."

gjm11 15y ago
That's a failure mode that's had quite a bit of attention at Less Wrong; see e.g. http://lesswrong.com/lw/ia/focus_your_uncertainty/ http://lesswrong.com/lw/ia/focus_your_uncertainty/ and http://lesswrong.com/lw/ii/conservation_of_expected_evidence/ http://lesswrong.com/lw/ii/conservation_of_expected_evidence... .

ThomPete 15y ago

Feynman have a great little story about checking the facts behind the facts. Anyone got a link to that?

Drbble 15y ago

Has anyone ever made the claim that LW is lampooning? I have only heard that testing early is important because it is expensive to fix mistakes after development, and more expensive after release. I guess the straw man argument is that design bugs are worse than implementation bugs in a project, but that's also true. In both cases, the issue is that time cures all wounds, where cure is used in the sense of made permanent, not fixed, or fixed is used on the sense of locked, not restored to working order.