6 ms·

Minesweeper automates root cause analysis as a first-line defense against bugs

98 points by muglug 6y ago

stingraycharles 6y ago
While this is a great strategy for figuring out the cause of a bug, I’d argue that “root cause analysis” in engineering is typically a much more qualitative analysis, and more about high impact failures than mere bug reports. A more accurate title may be “automatic data collection and analysis for bug reports”; I’m also confident that Microsoft has been doing this exact same thing for at least a few decades.
- tehjoker 6y ago
  I did like the idea that they recorded low memory conditions. That seems incredibly useful for debugging issues that occur in the wild. A natural next step would be checking GPU memory as well if they haven't already. Is it possible to measure overall system memory pressure in JS or is that sandboxed?
  - puttycat 6y ago
    I’m pretty confident that they use the same OS data for fingerprinting as well.
    - hanniabu 6y ago
      What special identifiers are there in the OS? And does this also apply to Linux/Ubuntu?
- cat199 6y ago
  > and more about high impact failures than mere bug reports. It's often more about people problems than software problems..
schemescape 6y ago
This seems like a great system for isolating steps to reproduce a bug, but I’m not sure I would consider this “root cause analysis”.
1_2__4 6y ago
It’s been a disappointing trend that SRE execs have adopted magical ML thinking, believing they’re just a few short years away from ditching all those annoying engineers and replacing them with a model. I am skeptical.
qbasic_forever 6y ago
So any concrete stats on how this has helped shorten bug investigation time, improve quality of releases, etc? It looks like an interesting data-driven approach to bug finding but there's curiously no qualitative analysis of how it's actually working in practice. I'd be a little concerned that systems like this can fall into the background as a flurry of noise and process that doesn't actually improve the quality of the product.
vladd 6y ago
Seems the article is confusing the "trigger" of an event with its "root cause". I like to give the example of the Concorde airplane crash [0] to exemplify the difference: the incident was triggered by debris on the runway (which caused the tires to explode, igniting the fuel tank above). But the root cause was the placement of fuel in proximity of inflatable tire materials. [0] https://en.wikipedia.org/wiki/Air_France_Flight_4590 https://en.wikipedia.org/wiki/Air_France_Flight_4590
- k1t 6y ago
  Is there really a difference though? To me a "trigger" is the initial event that begins a sequence. Isn't that also a "root cause"? Since everything is connected to everything else, it seems like the point that you decide is the "root" is fairly arbitrary. It seems you could easily conclude that the root cause was that the runway wasn't cleaned/inspected often enough. Or that the departures were scheduled too close together, preventing such an inspection, etc.. If anything I would say the root cause was the piece of engine cowl falling from the preceding flight - since that seems to be the first thing that "went wrong" in the process.
  - azinman2 6y ago
    If I said 'screw you', and that cause you to flip out and kill everyone around you, you couldn't say the root cause was me saying 'screw you'. The root cause might be childhood trauma, extreme emotional imbalance, irrational thoughts, etc. The statement 'screw you' was the trigger. Similarly here, a trigger might be uploading a photo to FB, but the root cause of an issue might be a bug in encoding JPGs.
  - spockz 6y ago
    To me the placement of the tanks or lack of protection would be the root cause. Because this failure could have been triggered by any other debris as well. So the root cause is the thing that If you fix it, it fixes the problem in a fundamental level.
  - dathinab 6y ago
    Debris on the runway is something to be expected in rare cases, given how many flights there are a day it's just a matter of time until it happens. As such the problem is an air plan which is designed in a way too prone to cause (too) fatal accidents in certain "rare but guaranteed to happen at some point" situation. But in the end if you say both are trigger which together lead to the catastrophe or one is a trigger and another is the root cause is indeed irrelevant. The problem is if you do something I will call trigger analysis but refer to root cause analysis treating it as if it gives you the root cause it can very easily to situations where you fix one of the problems but not all, and potentially not even the biggest problem. I.e. you make it slightly less likely that there is debris on the runway but you don't fix the problem of the airplane being too prone to certain kinds of catastrophic failure.
nomy99 6y ago
typo: Engineering not Enginering
- pronoiac 6y ago
  I emailed the mods.
Tarsul 6y ago
It appears to have nothing to do with the game.
- KMag 6y ago
  Dammit. You just made me lose The Game[0] [0] https://en.wikipedia.org/wiki/The_Game_(mind_game) https://en.wikipedia.org/wiki/The_Game_(mind_game)
haihaibye 6y ago
When naming your software, it's probably not a good idea to re-use the same name as one of the most popular games of all time.
RamRavi 6y ago
Thanks for sharing this informative content, Great work. To crack Scrum master interview: https://leanpitch.com/blogs/scrum-master-interview-questions https://leanpitch.com/blogs/scrum-master-interview-questions

98 points by muglug 6y ago

While this is a great strategy for figuring out the cause of a bug, I’d argue that “root cause analysis” in engineering is typically a much more qualitative analysis, and more about high impact failures than mere bug reports. A more accurate title may be “automatic data collection and analysis for bug reports”; I’m also confident that Microsoft has been doing this exact same thing for at least a few decades.

tehjoker 6y ago
I did like the idea that they recorded low memory conditions. That seems incredibly useful for debugging issues that occur in the wild. A natural next step would be checking GPU memory as well if they haven't already. Is it possible to measure overall system memory pressure in JS or is that sandboxed?
- puttycat 6y ago
  I’m pretty confident that they use the same OS data for fingerprinting as well.
  - hanniabu 6y ago
    What special identifiers are there in the OS? And does this also apply to Linux/Ubuntu?
cat199 6y ago
> and more about high impact failures than mere bug reports. It's often more about people problems than software problems..

schemescape 6y ago

This seems like a great system for isolating steps to reproduce a bug, but I’m not sure I would consider this “root cause analysis”.

1_2__4 6y ago

It’s been a disappointing trend that SRE execs have adopted magical ML thinking, believing they’re just a few short years away from ditching all those annoying engineers and replacing them with a model. I am skeptical.

qbasic_forever 6y ago

So any concrete stats on how this has helped shorten bug investigation time, improve quality of releases, etc? It looks like an interesting data-driven approach to bug finding but there's curiously no qualitative analysis of how it's actually working in practice. I'd be a little concerned that systems like this can fall into the background as a flurry of noise and process that doesn't actually improve the quality of the product.

vladd 6y ago

Seems the article is confusing the "trigger" of an event with its "root cause". I like to give the example of the Concorde airplane crash [0] to exemplify the difference: the incident was triggered by debris on the runway (which caused the tires to explode, igniting the fuel tank above). But the root cause was the placement of fuel in proximity of inflatable tire materials. [0] https://en.wikipedia.org/wiki/Air_France_Flight_4590 https://en.wikipedia.org/wiki/Air_France_Flight_4590

k1t 6y ago
Is there really a difference though? To me a "trigger" is the initial event that begins a sequence. Isn't that also a "root cause"? Since everything is connected to everything else, it seems like the point that you decide is the "root" is fairly arbitrary. It seems you could easily conclude that the root cause was that the runway wasn't cleaned/inspected often enough. Or that the departures were scheduled too close together, preventing such an inspection, etc.. If anything I would say the root cause was the piece of engine cowl falling from the preceding flight - since that seems to be the first thing that "went wrong" in the process.
- azinman2 6y ago
  If I said 'screw you', and that cause you to flip out and kill everyone around you, you couldn't say the root cause was me saying 'screw you'. The root cause might be childhood trauma, extreme emotional imbalance, irrational thoughts, etc. The statement 'screw you' was the trigger. Similarly here, a trigger might be uploading a photo to FB, but the root cause of an issue might be a bug in encoding JPGs.
- spockz 6y ago
  To me the placement of the tanks or lack of protection would be the root cause. Because this failure could have been triggered by any other debris as well. So the root cause is the thing that If you fix it, it fixes the problem in a fundamental level.
- dathinab 6y ago
  Debris on the runway is something to be expected in rare cases, given how many flights there are a day it's just a matter of time until it happens. As such the problem is an air plan which is designed in a way too prone to cause (too) fatal accidents in certain "rare but guaranteed to happen at some point" situation. But in the end if you say both are trigger which together lead to the catastrophe or one is a trigger and another is the root cause is indeed irrelevant. The problem is if you do something I will call trigger analysis but refer to root cause analysis treating it as if it gives you the root cause it can very easily to situations where you fix one of the problems but not all, and potentially not even the biggest problem. I.e. you make it slightly less likely that there is debris on the runway but you don't fix the problem of the airplane being too prone to certain kinds of catastrophic failure.

nomy99 6y ago

typo: Engineering not Enginering

pronoiac 6y ago
I emailed the mods.

Tarsul 6y ago

It appears to have nothing to do with the game.

KMag 6y ago
Dammit. You just made me lose The Game[0] [0] https://en.wikipedia.org/wiki/The_Game_(mind_game) https://en.wikipedia.org/wiki/The_Game_(mind_game)

haihaibye 6y ago

When naming your software, it's probably not a good idea to re-use the same name as one of the most popular games of all time.

RamRavi 6y ago

Thanks for sharing this informative content, Great work. To crack Scrum master interview: https://leanpitch.com/blogs/scrum-master-interview-questions https://leanpitch.com/blogs/scrum-master-interview-questions