3 ms·

Cloudflare doesn't seem to have called it a "Root Cause Analysis" and, in fact, the term "root cause" doesn't appear to occur in Prince's report. I bring this u

Cloudflare doesn't seem to have called it a "Root Cause Analysis" and, in fact, the term "root cause" doesn't appear to occur in Prince's report. I bring this up because there's a school of thought that says "root cause analysis" is counterproductive: complex systems are always balanced on the precipice of multicausal failure.

Analemma_ 1y ago
When I was at AWS, when we did postmortems on incidents we called it "root cause analysis", but it was understood by everyone that most incidents are multicausal and the actual analyses always ended up being fishbone diagrams. Probably there are some teams which don't do this and really do treat RCA as trying to find a sole root cause, but I think a lot of "getting mad at RCA" is bikeshedding the terminology, and nothing to do with the actual practice.
- tptacek 1y ago
  Right, I'm not a semantic zealot on this point, but the post we're commenting on really does suggest that the Cloudflare incident had a root cause in basic database management failures, which is the substantive issue the root-cause-haters have with the term.
  - otterley 1y ago
    The layered-swiss-cheese model of understanding incidents tends to map to the real world better than the alternatives.
- cyberax 1y ago
  > to find a sole root cause "Six billion years ago the dust around the young Sun coalesced into planets"
  - luhn 1y ago
    "Workaround: If we wait long enough, the earth will eventually be consumed by the sun." https://xkcd.com/1822/ https://xkcd.com/1822/
- otterley 1y ago
  These days we tend to spend more time thinking about the "5 whys" (which often turn into more than 5) than the root cause itself. It's much more productive and useful.
PunchyHamster 1y ago
then rename it to "root causes analysis"
parados 1y ago
True, and I agree, but from their report they do seem to be doing Root Cause Analysis (RCA) even if they don't call it that. RCA is a really bad way of investigating a failure. Simply put; if you show me your RCA I know exactly where you couldn't be bothered to look any further. I think most software engineers using RCA confuse the "cause" ("Why did this happen") with the solution ("We have changed this line of code and it's fixed"). These are quite different problem domains. Using RCA to determine "Why did this happen" is only useful for explaining the last stages of an accident. It focuses on cause->effect relationships and tells a relatively simple story but one that is easy to communicate - Hi there managers and media! But RCA only encourages simple countermeasures which will probably be ineffective and will be easily outrun by the complexity of real systems However one thing RCA is really good at is allocating blame. If your organisation is using RCA then, what ever you pretend, your organisation has a culture of blame. With a blame culture (rather than a reporting culture) your organisation is much more likely to fail again. You will lack operational resilience.

Analemma_ 1y ago

When I was at AWS, when we did postmortems on incidents we called it "root cause analysis", but it was understood by everyone that most incidents are multicausal and the actual analyses always ended up being fishbone diagrams. Probably there are some teams which don't do this and really do treat RCA as trying to find a sole root cause, but I think a lot of "getting mad at RCA" is bikeshedding the terminology, and nothing to do with the actual practice.

tptacek 1y ago
Right, I'm not a semantic zealot on this point, but the post we're commenting on really does suggest that the Cloudflare incident had a root cause in basic database management failures, which is the substantive issue the root-cause-haters have with the term.
- otterley 1y ago
  The layered-swiss-cheese model of understanding incidents tends to map to the real world better than the alternatives.
cyberax 1y ago
> to find a sole root cause "Six billion years ago the dust around the young Sun coalesced into planets"
- luhn 1y ago
  "Workaround: If we wait long enough, the earth will eventually be consumed by the sun." https://xkcd.com/1822/ https://xkcd.com/1822/
otterley 1y ago
These days we tend to spend more time thinking about the "5 whys" (which often turn into more than 5) than the root cause itself. It's much more productive and useful.

PunchyHamster 1y ago

then rename it to "root causes analysis"

parados 1y ago

True, and I agree, but from their report they do seem to be doing Root Cause Analysis (RCA) even if they don't call it that. RCA is a really bad way of investigating a failure. Simply put; if you show me your RCA I know exactly where you couldn't be bothered to look any further. I think most software engineers using RCA confuse the "cause" ("Why did this happen") with the solution ("We have changed this line of code and it's fixed"). These are quite different problem domains. Using RCA to determine "Why did this happen" is only useful for explaining the last stages of an accident. It focuses on cause->effect relationships and tells a relatively simple story but one that is easy to communicate - Hi there managers and media! But RCA only encourages simple countermeasures which will probably be ineffective and will be easily outrun by the complexity of real systems However one thing RCA is really good at is allocating blame. If your organisation is using RCA then, what ever you pretend, your organisation has a culture of blame. With a blame culture (rather than a reporting culture) your organisation is much more likely to fail again. You will lack operational resilience.