5 ms·

Root cause analysis? You're doing it wrong

137 points by davedx 1y ago

Recent and related: https://news.ycombinator.com/item?id=45547079 https://news.ycombinator.com/item?id=45547079

kqr 1y ago
Author here. Please note this is an early draft/stream-of-consciousness. Feel free to read and share anyway but my actual published articles hold a higher standard!
- maybelsyrup 1y ago
  I caught your related comments and eventual link to this post in another HN thread earlier this week and really liked them / it. I'm glad you posted it by itself!
tptacek 1y ago
Some of the same thoughts in Richard Cook, which was a brain-altering read for me: https://how.complexsystems.fail/ https://how.complexsystems.fail/
- wrs 1y ago
  For a deeper dive there’s a somewhat old but excellent book on most of these points called Normal Accidents. [0] [0] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
- Waterluvian 1y ago
  Hah #7 really hits home. Every RCA I’ve been a part of always ends up pointing to systemic failures in the org at the top level, because walking the tree always leads there. You can’t blame any one person or system for a failure in isolation. It’s usually some form of, “this is ultimately a consequence of miscalibrating risk associated to business/financial decisions.” I forget where I heard this but, “you manage risk, but risk cannot be managed.” Ie. there is no terminal state where it’s been “solved.” It’s much like “culture of safety.”
  - anonymars 1y ago
    I was pondering this a bit recently while going through The Wire Though unsatisfying it feels like a lot boils down to "shit rolls downhill" or "fish rot from the head down"
    - jaggederest 1y ago
      Ultimately this is what management is at least nominally supposed to do: decide values, set standards, and ensure that people down the chain are strategically aligned and have an environment where everyone is working together. That includes Andy Grove's "getting people to resolve their own differences" and other meta tasks. But fundamentally the person in charge is responsible for everything, even though they can't know everything. Any RCA that doesn't provide useful feedback to management up and down the chain is missing pieces, but there's lots of discussion elsewhere in this thread about that by people better at elaborating than I am.
  - crazygringo 1y ago
    Of course, and that's very much the point. If you get a bunch of analyses that point to underinvestment in x in order to achieve y, and you can measure that this is losing money, then the top level recalibrates. It's not about blame, it's about course-correcting including at the top level. They can't do that course-correcting without these analyses.
    - hinkley 1y ago
      As one of my early mentorish people used to say: sometimes you have to let the baby fall down. Before that I hated it when people confessed to me that they knew a problem was going to happen and they did nothing to stop it. But the problem with doing things right the first time is that nobody appreciates how hard it is. You spend all your political capital trying to stop the company from making a mistake it desperately wants to make. You get no credit when it doesn't happen because they already figured it wouldn't. And you don't get any buy-in for keeping it from repeating in the future. So now you have this plate you have to spin all by yourself, and nobody who can hand your more control gives a shit other than the fact that you seem like an asshole so we aren't going to promote you.
- bombcar 1y ago
  I think #7 strikes (but barely misses) the point - root cause analysis is not root blame analysis- but we often combine them in our mind.
- growse 1y ago
  This paper so affected me that I scrapped a talk I was writing three days ahead of an (internal) conference and wrote a talk about this paper instead!
bluGill 1y ago
Root cause works better if you can come back next time the same thing happens and find a different root cause to fix. keep repeating until the problem doesn't happen enough to care anymore. If the result/accident is too bad though you need to find all the different faults and mitigate as manyias possible the first time.
- kqr 1y ago
  > Root cause works better if you can come back next time the same thing happens and find a different root cause to fix. keep repeating until the problem doesn't happen enough to care anymore. This sounds like continuously firefighting to paper over symptoms rather than address the problems at a deeper level.
  - bluGill 1y ago
    Sometimes, but sometimes you are allowed to discover and fix the real proplems which the never return.
gtirloni 1y ago
I recommend attending the next STAMP Workshop offered by MIT if you have a chance: https://psas.scripts.mit.edu/home/stamp-workshops https://psas.scripts.mit.edu/home/stamp-workshops
jph 1y ago
Thanks for the article and shoutout - CAST is great and I use it extensively with tech teams. Causal Analysis based on Systems Theory - my notes - https://github.com/joelparkerhenderson/causal-analysis-based-on-system-theory https://github.com/joelparkerhenderson/causal-analysis-based... The full handbook by Nancy G. Leveson at MIT is free here: http://sunnyday.mit.edu/CAST-Handbook.pdf http://sunnyday.mit.edu/CAST-Handbook.pdf
gmuslera 1y ago
Not all problems (and systems) are alike. And probably simple approaches like Occam's Razor will work good enough with most. But the remaining 10% will need deeper digging into more data and correlations.
opwieurposiu 1y ago
I feel like half the time issues are caused by adding some stupid feature that nobody really wants, but makes it in anyways because the incentive is to add features, not make good software. People rarely react well if you tell them "Hey this feature ticket you made is poorly conceived and will cause problems, can we just not do it?" It is easier just to implement whatever it is and deal with the fallout later.
- hinkley 1y ago
  It's hard to prove the cost to a feature or bug fix or library upgrade we desperately need has been doubled by all of the features we didn't need. My 'favorite' is when we implement stupid, self-limiting, corner-painting features for a customer who leaves us anyway. Or who we never manage to make money from.
CobrastanJorji 1y ago
Many years ago, I worked at Amazon, and it was at the time quite fond of the "five whys" approach to root cause analysis: say what happened, ask why that happened, ask why that in turn happened, and keep going until you get to some very fundamental problem. I was asked to write up such a document for an incident where our team had written a new feature which, upon launch, did absolutely nothing. Our team had accidentally mistyped a flag name on the last day before we handed it to a test team, the test team examined the (nonfunctional) tool for a few weeks and blessed it, and then upon turning it on, it failed to do anything. My five whys document was most about "what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do." I recall my manager handing the doc back to me and saying that I needed to completely redo it because it was unacceptable for us to blame another team for our team's bug, which is how I learned that you can make a five why process blame any team you find convenient by choosing the question. I quit not too long after that.
- sanman8119 1y ago
  A very relatable experience, lot of pressure to stop the Whys at the dev team and not question larger leadership or organizational moves
- hshdhdhehd 1y ago
  Interesting one. My first thought is why is rolling out a new system to prod that is not used yet an incident? I dont think "being in prod" is sufficient. There should be tiers of service and a brand new service should not be on a tier where it having teething issues is an incident. > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do would be interested to see the doc, but imagine you'd branch off the causes, one branch of the tree is: UAT didnt pick up the bug. why didn't UAT pick up the bug? .... (you'd need that teams help). I think that team would have something that is a contributing cause. You shouldn't rely on UAT to pick up a bug in a released product. However just because it is not a root cause doesnt mean it shouldn't be addressed. Today's contributing cause can be tomorrow's root cause! So yeah yiu dont blame another team but you also dont shield another team from one of their systems needing attention! The wording matters alot though. The way you worded the question seems a little loaded. But you may be paraphrasing? 5 whys are usually more like "Why did they papaya team not detect the bug before deployment?" Whereas > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do Is more emotive. Sounds like a cross examiners question which isn't the vibe you'd want to go for. 5 whys should be 5 drys. Nothing spicy!
captainkrtek 1y ago
At a large cloud provider I held a role for a bit in the “safety” organization that was tasked with developing better understanding of our incidents, working on tooling to protect systems, and so on. A few problems I faced: - culturally a lack of deeper understanding or care around “safety” topics. Forces that be inherently are motivated by launching features and increasing sales, so more often than not you could write an awesome incident retro doc and just get people who are laser focused on the bare minimum of action items. - security folks co-opting the safety things, because removing access to things can be misconstrued to mean making things safer. While somewhat true, it also makes doing jobs more difficult if not replaced with adequate tooling. What this meant was taking away access and replacing everything with “break glass” mechanisms. If your team is breaking glass every day and ticketing security, you’re probably failing to address both security and safety.. - related to the last point, but a lack of introspection as to the means of making changes which led to the incident. For example: user uses ssh to run a command and ran the wrong command -> we should eliminate ssh. Rather than asking why was ssh the best / only way the user could affect change to the system? Could we build an api for this with tooling and safeguards before cutting off ssh?
- hinkley 1y ago
  I've applied for a couple jobs like this and was somewhat relieved they didn't call me back. When you move thinking about reliability or safety outside of the teams that generate the problems, you replace self reflection with scolding, and you have to either cajole people to make changes for you or jump into code you're not spending enough time with to truly understand. And then if you make a mistake this is evidence that you shouldn't be touching it at all. See we told you this would end badly.
  - captainkrtek 1y ago
    Yeah I think that’s accurate. The org had good intentions and owned some reasonable programs, but more and more became basically a security organization focused on cutting off hands to prevent people from touching keyboards, rather than addressing real systemic risks and patterns of operator behaviour leading to incidents.
hshdhdhehd 1y ago
Please keep working on that piece I think it will be very useful for incident reviewers. Someone said the quiet part loud! : """ Common circumstances missing from accident reports are: Pressures to cut costs or work quicker, Competing requests for colleagues, Unnecessarily complicated systems, Broken tools, Biological needs (e.g. sleep or hunger), Cumbersome enforced processes, Fear of being consequences of doing something out of the ordinary, and Shame of feeling in over one’s head. """
- VirusNewbie 1y ago
  >Pressures to cut costs or work quicker, >Unnecessarily complicated systems, >Broken tools, >cumbersome enforced processes, I have seen all of these specifically called out in Post Mortems at Google, so that's a plus in my book.
- hinkley 1y ago
  Saying "I was wrong" can be a very powerful debate tool, especially if you can follow it up with concrete actions to fix the problem. There were tools I wrote at my last job because basically an incident could IMO be tracked back to, "I was on step 5 and George interrupted me to ask a question of some other high priority effort, and when I got back I forgot to finish step 5". The more familiar you are with a runbook, the more your brain will confuse memories from an hour or a day ago with memories of the same routine at a different time. It's literally "did I turn off the stove". You remember turning off the stove. Many, many times. But did you turn off both burners you turned on for lunch? You're certain about one. But what about the other? That's a blur. But we're software developers. The more mundane a task, the more likely that we can replace it with a program to do the same thing. And as you get more familiar with the task you keep adding more to the automatic parts until one day it's just a couple buttons that can be reliably pushed during peak traffic on your services. Just make sure there isn't an active outage going on when you push the buttons.
  - hshdhdhehd 1y ago
    Tip. Every runbook should have this first step: 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I am blessed with a bad memory so I do this anyway! But not everyone has that "advantage".
exmadscientist 1y ago
I agree with a lot of the statements at the top of the article, but some of them are just nonsense. This one, in particular: > If we analyse accidents more deeply, we can get by analysing fewer accidents and still learn more. Yeah, that's not how it works. The failure modes of your system might be concentrated in one particularly brittle area, but you really need as much breadth as you can get: the bullets are always fired at the entire plane. > An accident happens when a system in a hazardous state encounters unfavourable environmental conditions. We cannot control environmental conditions, so we need to prevent hazards. I mean, I'm an R&D guy, so my experience is biased, but... sometimes the system is just broke and no amount of saying "the system is in a hazardous state" can paper over the fact that you shipped (or, best-case, stress-tested) trash. You absolutely have to run these cases through the failure analysis pipeline, there's no out there, but the analysis flow looks a bit different for things that should-have worked versus things that could-never-have worked. And, yes, it will roll up on management, but... still.
- kqr 1y ago
  > you really need as much breadth as you can get Sure, more is always better. Practically, though, we are trading depth for breadth. In my experience, many problems that look dissimilar after a shallow analysis turn out to be caused by the same thing when analysed in depth. In that case, it is more economical to analyse fewer incidents in greater depth and actually find their common factors, rather than make a shallow pass over many incidents and continue to paper over symptoms of the undiscovered deeper problem.
  - exmadscientist 1y ago
    I guess that's where our experience differs. I am 100% on board with you for chasing as many incidents to true root cause as possible and I agree that doing so can be extremely helpful in ways that might not be easy to foresee. But my experience is also that you cannot ignore anything. Even the little stuff. The number of difficult system-level bugs I have resolved by remembering "you know, two weeks ago, it briefly did this weird thing that really shouldn't have been possible, but this might be related to that if only..." is crazy. It's been a superpower for me through the years. However, I mostly work on hardware. Hardware's complexity envelope is straight-up different to software's. So that might explain some of our different perspective. Hardware absolutely never randomly misbehaves (which is to say that all its bad behavior has some kind of cause one might reasonably be able to ascertain, and that cause isn't from a level unreachable for a hardware engineer), but software carries enough state, and state from other levels of the stack, that I would not make the same statement. Thus the fault-chasing priorities aren't quite the same.
AstroJetson 1y ago
I did a very long RCA on a problem. My management at the time was really BIG into looking at ALL THE CAUSES. They wanted HUGE fishbone diagrams to show that we had looked at everything. This was in the days of having huge drum plotters, so the diagrams could be 36" and many feet long. So I did what they wanted and the root cause was: On December 11 1963 Mr and Mrs Stanley Smith had sexual intercourse. I got asked what that had to do with anything and I said, "If you look up a few lines you'll see that the issue was a human error caused by Bob Smith, if he hadn't been born we wouldn't have had this problem and I just went back to the actual conception date." I got asked how I was able to pin it to that date and said "I asked Bob what his father's birthday was and extrapolated that info" I was never asked to do a RCA again.
- hinkley 1y ago
  I model RCAs after my understanding of NTSB and incident response after my understanding of NASA command centers. They're both flawed but often replace something that works 3x worse than my caricature of both. The findings should always result in a material change that is worth at least the effort of having done it. Not just a checkbox that proves we did something. The investment in the mitigation should honor the consequences of the failure, and the uniqueness of the failure. Or rather, the lack of uniqueness. As a failure repeats in kind (eg, a bunch of 737 Maxes crashing), trust that the system is put in jeopardy. By the time a problem has happened three times, the response should begin to resemble penance. So how do we get the problem not to hit production again, or how do we at least keep it happening due to the exact same error? And for some failure modes, we need to project the consequences going forward. Let's say you find your app is occasionally crashing over a weekend because of memory leaks, plus the lack of Continuous Deployment forcibly restarting the services. We can predict this problem will happen reliably on Memorial Day, and Labor Day. So we need to do something relatively serious now. But it'll also get much worse on Thanksgiving weekend, and just stupid around Christmas, when we have code freezes. So we do something to get us through Memorial Day but we also need a second story near the top of the backlog that needs to be done by Labor Day, Thanksgiving at the latest. But we don't necessarily have to do that story next sprint.
protocolture 1y ago
Good RCA: Produce some useful documentation to prevent issue from recurring. Fantastic RCA: Remove requirement that caused the action that resulted in the problem occurring. Bad RCA: Lets get 12 non technical people on a call to ask the on call engineer who is tired from 6 hours managing the fault, a bunch of technical questions they don't understand the answers to anyway. (Worst possible fault practice is to bring in a bunch of stakeholders and force the engineer to be on a call with them while they try and investigate the fault) Worst RCA: A half paragraph describing the problem in the most general terms to meet a contractual RCA requirement.
0xbadcafebee 1y ago
A lot of this is based on heavy assumptions about systems and risk/safety analysis. The biggest assumption this post is making is that humans should be involved at all. Systems do not have to facilitate operators in building accurate mental models. In fact, safe systems disregard mental models, because a mental model is a human thing, and humans are fallible. Remove the human and you have a safer system. Safety is not a dynamic control problem, it's a quality-state management problem. You need to maintain a state of quality assurance/quality control to ensure safety. When it falters, so does safety. Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Also fwiw, there's often not a root cause, but instead multiple causes (or coincidental states). For getting better at tracking down causes of failures (and preventing them), I recommend learning the Toyota Production System, then reading Out Of The Crisis. That'll kickstart your brain enough to be more effective than 99.99% of people.
- kqr 1y ago
  > mental model It should be more clear in the article that this term is used more broadly. The "mental model" is the function that converts feedback into control actions. In that sense, even simple automated controllers have "mental models". > Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Sometimes it isn't but often it is. Yes, that makes the system more complex, along with other properties such as organicism, non-linearity, parallelism, interactivity, long-term recurrences, etc. These properties are inconvenient but we cannot just wish them away. We have to design for the system we have. > Toyota, Deming, SPC I have read more books in this area than most people, and I don't really see how you came away without an appreciation for the importance of humans in the loop, and the dangers of overautomation. Could you illustrate more clearly what you mean?
o4c 1y ago
[dead]

137 points by davedx 1y ago

Recent and related: https://news.ycombinator.com/item?id=45547079 https://news.ycombinator.com/item?id=45547079

kqr 1y ago

Author here. Please note this is an early draft/stream-of-consciousness. Feel free to read and share anyway but my actual published articles hold a higher standard!

maybelsyrup 1y ago
I caught your related comments and eventual link to this post in another HN thread earlier this week and really liked them / it. I'm glad you posted it by itself!

tptacek 1y ago

Some of the same thoughts in Richard Cook, which was a brain-altering read for me: https://how.complexsystems.fail/ https://how.complexsystems.fail/

wrs 1y ago
For a deeper dive there’s a somewhat old but excellent book on most of these points called Normal Accidents. [0] [0] https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
Waterluvian 1y ago
Hah #7 really hits home. Every RCA I’ve been a part of always ends up pointing to systemic failures in the org at the top level, because walking the tree always leads there. You can’t blame any one person or system for a failure in isolation. It’s usually some form of, “this is ultimately a consequence of miscalibrating risk associated to business/financial decisions.” I forget where I heard this but, “you manage risk, but risk cannot be managed.” Ie. there is no terminal state where it’s been “solved.” It’s much like “culture of safety.”
- anonymars 1y ago
  I was pondering this a bit recently while going through The Wire Though unsatisfying it feels like a lot boils down to "shit rolls downhill" or "fish rot from the head down"
  - jaggederest 1y ago
    Ultimately this is what management is at least nominally supposed to do: decide values, set standards, and ensure that people down the chain are strategically aligned and have an environment where everyone is working together. That includes Andy Grove's "getting people to resolve their own differences" and other meta tasks. But fundamentally the person in charge is responsible for everything, even though they can't know everything. Any RCA that doesn't provide useful feedback to management up and down the chain is missing pieces, but there's lots of discussion elsewhere in this thread about that by people better at elaborating than I am.
- crazygringo 1y ago
  Of course, and that's very much the point. If you get a bunch of analyses that point to underinvestment in x in order to achieve y, and you can measure that this is losing money, then the top level recalibrates. It's not about blame, it's about course-correcting including at the top level. They can't do that course-correcting without these analyses.
  - hinkley 1y ago
    As one of my early mentorish people used to say: sometimes you have to let the baby fall down. Before that I hated it when people confessed to me that they knew a problem was going to happen and they did nothing to stop it. But the problem with doing things right the first time is that nobody appreciates how hard it is. You spend all your political capital trying to stop the company from making a mistake it desperately wants to make. You get no credit when it doesn't happen because they already figured it wouldn't. And you don't get any buy-in for keeping it from repeating in the future. So now you have this plate you have to spin all by yourself, and nobody who can hand your more control gives a shit other than the fact that you seem like an asshole so we aren't going to promote you.
bombcar 1y ago
I think #7 strikes (but barely misses) the point - root cause analysis is not root blame analysis- but we often combine them in our mind.
growse 1y ago
This paper so affected me that I scrapped a talk I was writing three days ahead of an (internal) conference and wrote a talk about this paper instead!

bluGill 1y ago

Root cause works better if you can come back next time the same thing happens and find a different root cause to fix. keep repeating until the problem doesn't happen enough to care anymore. If the result/accident is too bad though you need to find all the different faults and mitigate as manyias possible the first time.

kqr 1y ago
> Root cause works better if you can come back next time the same thing happens and find a different root cause to fix. keep repeating until the problem doesn't happen enough to care anymore. This sounds like continuously firefighting to paper over symptoms rather than address the problems at a deeper level.
- bluGill 1y ago
  Sometimes, but sometimes you are allowed to discover and fix the real proplems which the never return.

gtirloni 1y ago

I recommend attending the next STAMP Workshop offered by MIT if you have a chance: https://psas.scripts.mit.edu/home/stamp-workshops https://psas.scripts.mit.edu/home/stamp-workshops

jph 1y ago

Thanks for the article and shoutout - CAST is great and I use it extensively with tech teams. Causal Analysis based on Systems Theory - my notes - https://github.com/joelparkerhenderson/causal-analysis-based-on-system-theory https://github.com/joelparkerhenderson/causal-analysis-based... The full handbook by Nancy G. Leveson at MIT is free here: http://sunnyday.mit.edu/CAST-Handbook.pdf http://sunnyday.mit.edu/CAST-Handbook.pdf

gmuslera 1y ago

Not all problems (and systems) are alike. And probably simple approaches like Occam's Razor will work good enough with most. But the remaining 10% will need deeper digging into more data and correlations.

opwieurposiu 1y ago

I feel like half the time issues are caused by adding some stupid feature that nobody really wants, but makes it in anyways because the incentive is to add features, not make good software. People rarely react well if you tell them "Hey this feature ticket you made is poorly conceived and will cause problems, can we just not do it?" It is easier just to implement whatever it is and deal with the fallout later.

hinkley 1y ago
It's hard to prove the cost to a feature or bug fix or library upgrade we desperately need has been doubled by all of the features we didn't need. My 'favorite' is when we implement stupid, self-limiting, corner-painting features for a customer who leaves us anyway. Or who we never manage to make money from.

CobrastanJorji 1y ago

Many years ago, I worked at Amazon, and it was at the time quite fond of the "five whys" approach to root cause analysis: say what happened, ask why that happened, ask why that in turn happened, and keep going until you get to some very fundamental problem. I was asked to write up such a document for an incident where our team had written a new feature which, upon launch, did absolutely nothing. Our team had accidentally mistyped a flag name on the last day before we handed it to a test team, the test team examined the (nonfunctional) tool for a few weeks and blessed it, and then upon turning it on, it failed to do anything. My five whys document was most about "what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do." I recall my manager handing the doc back to me and saying that I needed to completely redo it because it was unacceptable for us to blame another team for our team's bug, which is how I learned that you can make a five why process blame any team you find convenient by choosing the question. I quit not too long after that.

sanman8119 1y ago
A very relatable experience, lot of pressure to stop the Whys at the dev team and not question larger leadership or organizational moves
hshdhdhehd 1y ago
Interesting one. My first thought is why is rolling out a new system to prod that is not used yet an incident? I dont think "being in prod" is sufficient. There should be tiers of service and a brand new service should not be on a tier where it having teething issues is an incident. > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do would be interested to see the doc, but imagine you'd branch off the causes, one branch of the tree is: UAT didnt pick up the bug. why didn't UAT pick up the bug? .... (you'd need that teams help). I think that team would have something that is a contributing cause. You shouldn't rely on UAT to pick up a bug in a released product. However just because it is not a root cause doesnt mean it shouldn't be addressed. Today's contributing cause can be tomorrow's root cause! So yeah yiu dont blame another team but you also dont shield another team from one of their systems needing attention! The wording matters alot though. The way you worded the question seems a little loaded. But you may be paraphrasing? 5 whys are usually more like "Why did they papaya team not detect the bug before deployment?" Whereas > what part of our process led to a multiweek test effort that would greenlight a tool that does nothing that it is required to do Is more emotive. Sounds like a cross examiners question which isn't the vibe you'd want to go for. 5 whys should be 5 drys. Nothing spicy!

captainkrtek 1y ago

At a large cloud provider I held a role for a bit in the “safety” organization that was tasked with developing better understanding of our incidents, working on tooling to protect systems, and so on. A few problems I faced: - culturally a lack of deeper understanding or care around “safety” topics. Forces that be inherently are motivated by launching features and increasing sales, so more often than not you could write an awesome incident retro doc and just get people who are laser focused on the bare minimum of action items. - security folks co-opting the safety things, because removing access to things can be misconstrued to mean making things safer. While somewhat true, it also makes doing jobs more difficult if not replaced with adequate tooling. What this meant was taking away access and replacing everything with “break glass” mechanisms. If your team is breaking glass every day and ticketing security, you’re probably failing to address both security and safety.. - related to the last point, but a lack of introspection as to the means of making changes which led to the incident. For example: user uses ssh to run a command and ran the wrong command -> we should eliminate ssh. Rather than asking why was ssh the best / only way the user could affect change to the system? Could we build an api for this with tooling and safeguards before cutting off ssh?

hinkley 1y ago
I've applied for a couple jobs like this and was somewhat relieved they didn't call me back. When you move thinking about reliability or safety outside of the teams that generate the problems, you replace self reflection with scolding, and you have to either cajole people to make changes for you or jump into code you're not spending enough time with to truly understand. And then if you make a mistake this is evidence that you shouldn't be touching it at all. See we told you this would end badly.
- captainkrtek 1y ago
  Yeah I think that’s accurate. The org had good intentions and owned some reasonable programs, but more and more became basically a security organization focused on cutting off hands to prevent people from touching keyboards, rather than addressing real systemic risks and patterns of operator behaviour leading to incidents.

hshdhdhehd 1y ago

Please keep working on that piece I think it will be very useful for incident reviewers. Someone said the quiet part loud! : """ Common circumstances missing from accident reports are: Pressures to cut costs or work quicker, Competing requests for colleagues, Unnecessarily complicated systems, Broken tools, Biological needs (e.g. sleep or hunger), Cumbersome enforced processes, Fear of being consequences of doing something out of the ordinary, and Shame of feeling in over one’s head. """

VirusNewbie 1y ago
>Pressures to cut costs or work quicker, >Unnecessarily complicated systems, >Broken tools, >cumbersome enforced processes, I have seen all of these specifically called out in Post Mortems at Google, so that's a plus in my book.
hinkley 1y ago
Saying "I was wrong" can be a very powerful debate tool, especially if you can follow it up with concrete actions to fix the problem. There were tools I wrote at my last job because basically an incident could IMO be tracked back to, "I was on step 5 and George interrupted me to ask a question of some other high priority effort, and when I got back I forgot to finish step 5". The more familiar you are with a runbook, the more your brain will confuse memories from an hour or a day ago with memories of the same routine at a different time. It's literally "did I turn off the stove". You remember turning off the stove. Many, many times. But did you turn off both burners you turned on for lunch? You're certain about one. But what about the other? That's a blur. But we're software developers. The more mundane a task, the more likely that we can replace it with a program to do the same thing. And as you get more familiar with the task you keep adding more to the automatic parts until one day it's just a couple buttons that can be reliably pushed during peak traffic on your services. Just make sure there isn't an active outage going on when you push the buttons.
- hshdhdhehd 1y ago
  Tip. Every runbook should have this first step: 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I am blessed with a bad memory so I do this anyway! But not everyone has that "advantage".

exmadscientist 1y ago

I agree with a lot of the statements at the top of the article, but some of them are just nonsense. This one, in particular: > If we analyse accidents more deeply, we can get by analysing fewer accidents and still learn more. Yeah, that's not how it works. The failure modes of your system might be concentrated in one particularly brittle area, but you really need as much breadth as you can get: the bullets are always fired at the entire plane. > An accident happens when a system in a hazardous state encounters unfavourable environmental conditions. We cannot control environmental conditions, so we need to prevent hazards. I mean, I'm an R&D guy, so my experience is biased, but... sometimes the system is just broke and no amount of saying "the system is in a hazardous state" can paper over the fact that you shipped (or, best-case, stress-tested) trash. You absolutely have to run these cases through the failure analysis pipeline, there's no out there, but the analysis flow looks a bit different for things that should-have worked versus things that could-never-have worked. And, yes, it will roll up on management, but... still.

kqr 1y ago
> you really need as much breadth as you can get Sure, more is always better. Practically, though, we are trading depth for breadth. In my experience, many problems that look dissimilar after a shallow analysis turn out to be caused by the same thing when analysed in depth. In that case, it is more economical to analyse fewer incidents in greater depth and actually find their common factors, rather than make a shallow pass over many incidents and continue to paper over symptoms of the undiscovered deeper problem.
- exmadscientist 1y ago
  I guess that's where our experience differs. I am 100% on board with you for chasing as many incidents to true root cause as possible and I agree that doing so can be extremely helpful in ways that might not be easy to foresee. But my experience is also that you cannot ignore anything. Even the little stuff. The number of difficult system-level bugs I have resolved by remembering "you know, two weeks ago, it briefly did this weird thing that really shouldn't have been possible, but this might be related to that if only..." is crazy. It's been a superpower for me through the years. However, I mostly work on hardware. Hardware's complexity envelope is straight-up different to software's. So that might explain some of our different perspective. Hardware absolutely never randomly misbehaves (which is to say that all its bad behavior has some kind of cause one might reasonably be able to ascertain, and that cause isn't from a level unreachable for a hardware engineer), but software carries enough state, and state from other levels of the stack, that I would not make the same statement. Thus the fault-chasing priorities aren't quite the same.

AstroJetson 1y ago

I did a very long RCA on a problem. My management at the time was really BIG into looking at ALL THE CAUSES. They wanted HUGE fishbone diagrams to show that we had looked at everything. This was in the days of having huge drum plotters, so the diagrams could be 36" and many feet long. So I did what they wanted and the root cause was: On December 11 1963 Mr and Mrs Stanley Smith had sexual intercourse. I got asked what that had to do with anything and I said, "If you look up a few lines you'll see that the issue was a human error caused by Bob Smith, if he hadn't been born we wouldn't have had this problem and I just went back to the actual conception date." I got asked how I was able to pin it to that date and said "I asked Bob what his father's birthday was and extrapolated that info" I was never asked to do a RCA again.

hinkley 1y ago
I model RCAs after my understanding of NTSB and incident response after my understanding of NASA command centers. They're both flawed but often replace something that works 3x worse than my caricature of both. The findings should always result in a material change that is worth at least the effort of having done it. Not just a checkbox that proves we did something. The investment in the mitigation should honor the consequences of the failure, and the uniqueness of the failure. Or rather, the lack of uniqueness. As a failure repeats in kind (eg, a bunch of 737 Maxes crashing), trust that the system is put in jeopardy. By the time a problem has happened three times, the response should begin to resemble penance. So how do we get the problem not to hit production again, or how do we at least keep it happening due to the exact same error? And for some failure modes, we need to project the consequences going forward. Let's say you find your app is occasionally crashing over a weekend because of memory leaks, plus the lack of Continuous Deployment forcibly restarting the services. We can predict this problem will happen reliably on Memorial Day, and Labor Day. So we need to do something relatively serious now. But it'll also get much worse on Thanksgiving weekend, and just stupid around Christmas, when we have code freezes. So we do something to get us through Memorial Day but we also need a second story near the top of the backlog that needs to be done by Labor Day, Thanksgiving at the latest. But we don't necessarily have to do that story next sprint.

protocolture 1y ago

Good RCA: Produce some useful documentation to prevent issue from recurring. Fantastic RCA: Remove requirement that caused the action that resulted in the problem occurring. Bad RCA: Lets get 12 non technical people on a call to ask the on call engineer who is tired from 6 hours managing the fault, a bunch of technical questions they don't understand the answers to anyway. (Worst possible fault practice is to bring in a bunch of stakeholders and force the engineer to be on a call with them while they try and investigate the fault) Worst RCA: A half paragraph describing the problem in the most general terms to meet a contractual RCA requirement.

0xbadcafebee 1y ago

A lot of this is based on heavy assumptions about systems and risk/safety analysis. The biggest assumption this post is making is that humans should be involved at all. Systems do not have to facilitate operators in building accurate mental models. In fact, safe systems disregard mental models, because a mental model is a human thing, and humans are fallible. Remove the human and you have a safer system. Safety is not a dynamic control problem, it's a quality-state management problem. You need to maintain a state of quality assurance/quality control to ensure safety. When it falters, so does safety. Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Also fwiw, there's often not a root cause, but instead multiple causes (or coincidental states). For getting better at tracking down causes of failures (and preventing them), I recommend learning the Toyota Production System, then reading Out Of The Crisis. That'll kickstart your brain enough to be more effective than 99.99% of people.

kqr 1y ago
> mental model It should be more clear in the article that this term is used more broadly. The "mental model" is the function that converts feedback into control actions. In that sense, even simple automated controllers have "mental models". > Dynamism is sometimes not a factor (although when it is, it's typically a harmful factor). Sometimes it isn't but often it is. Yes, that makes the system more complex, along with other properties such as organicism, non-linearity, parallelism, interactivity, long-term recurrences, etc. These properties are inconvenient but we cannot just wish them away. We have to design for the system we have. > Toyota, Deming, SPC I have read more books in this area than most people, and I don't really see how you came away without an appreciation for the importance of humans in the loop, and the dangers of overautomation. Could you illustrate more clearly what you mean?

o4c 1y ago

[dead]