3 ms·

I do wonder if people have any experience around doing root cause analysis that doesn't just end up in blame games? I've worked for three companies in a row tha

I do wonder if people have any experience around doing root cause analysis that doesn't just end up in blame games? I've worked for three companies in a row that claim to have blame-free cultures, and all of them did put work into it (structuring the documents to not assign an individual's name to any given misstep and telling people to be kind and understanding). But in every case, you can feel it in the air that everyone still understands that the RCA is a blame document/process and that management is keeping track of the individuals at fault and it still matters on their end of year eval. With the industry wide layoffs this feeling has only gotten worse now that there is a decent chance that accepting blame (or your manager deciding the blame is on you) will be the difference between having a job or not in 6 months. Maybe this is unavoidable given that these processes only kick in when something goes wrong, and you can only screw up at your job so much until you get shown the door?

hecanjog 3y ago
I really don't know either. When it works for me though, I think it looks like people going out of their way to point the finger at themselves, and nobody going out of their way to wag a follow-up finger at them. I'm not sure hiding it would be helpful, but I haven't tried that.
- jstarfish 3y ago
  > When it works for me though, I think it looks like people going out of their way to point the finger at themselves, and nobody going out of their way to wag a follow-up finger at them. This is how I prefer it too, but the problem is that honesty is a luxury when times are good. When times aren't so good, having a self-documented history of failure doesn't help your case for retention when the jerk next to you who deflects everything appears to be a golden boy. > I'm not sure hiding it would be helpful, but I haven't tried that. From the police playbook: after raising awareness of a problem, proactively make excuses for other teams and/or deflect blame to a vendor ("we don't know it's that team's fault; they're understaffed and we don't know what they're having to deal with over there; give them a break because nobody can understand the vendor's fucking incoherent documentation and their software is garbage anyway"). Confusing the issues and misdirecting blame gives the team time to address the problem while saving face. Everybody hates Microsoft anyway so they're an easy scapegoat. (Sorry, Microsoft employees.) For better or worse, it fosters a culture where people cover for each other, not throw each other under the bus. This is how you wrangle even the most toxic Narcissists-- they recognize and appreciate when someone's doing them a public favor, and (IME) this compels them to respond in kind because they'll want to one-up you as everyone's savior. (We're starting to see this with all the moral crusading and self-appointed "safety" workers. All Narcissists.)
sokoloff 3y ago
I think we ran an effective process to cover the RCA area of our operations. Importantly (IMO), we did go to lengths to understand and ascribe actions/activities that resulted in/contributed to an outage to a specific individual; we were just careful to not assign individual consequences to them. I think it's critically important to understand as precisely as possible what happened, who did it, what they were looking at that caused them to take that action, and which parts of that [if any] we'd change with the benefit of hindsight. I ran the Ops team at the time and it was easy for me to enforce the lack of consequences for anything short of an intentionally destructive act. If "blameless post-mortem" means "we want to make sure that no one has any idea who was responsible", you can achieve that but you probably won't like the results. If it instead means "we want to know why it happened, who contributed, and why, so that we can not repeat it", you have a fighting chance. I've written and published multiple RCAs that explain in detail why /u/sokoloff caused an outage, when it started, when it was contained, and how to avoid that mistake in the future. I think that trying to obscure who did something is not only not worth the effort, but is actively destructive to the learning and trust. If I can't trust that my name can appear next to an honest mistake, what else must I be distrustful of? If instead, I see respected, senior staff readily taking responsibility and sharing their mistakes without fear of consequences, I trust my company's leaders more, not less.

hecanjog 3y ago

I really don't know either. When it works for me though, I think it looks like people going out of their way to point the finger at themselves, and nobody going out of their way to wag a follow-up finger at them. I'm not sure hiding it would be helpful, but I haven't tried that.

jstarfish 3y ago
> When it works for me though, I think it looks like people going out of their way to point the finger at themselves, and nobody going out of their way to wag a follow-up finger at them. This is how I prefer it too, but the problem is that honesty is a luxury when times are good. When times aren't so good, having a self-documented history of failure doesn't help your case for retention when the jerk next to you who deflects everything appears to be a golden boy. > I'm not sure hiding it would be helpful, but I haven't tried that. From the police playbook: after raising awareness of a problem, proactively make excuses for other teams and/or deflect blame to a vendor ("we don't know it's that team's fault; they're understaffed and we don't know what they're having to deal with over there; give them a break because nobody can understand the vendor's fucking incoherent documentation and their software is garbage anyway"). Confusing the issues and misdirecting blame gives the team time to address the problem while saving face. Everybody hates Microsoft anyway so they're an easy scapegoat. (Sorry, Microsoft employees.) For better or worse, it fosters a culture where people cover for each other, not throw each other under the bus. This is how you wrangle even the most toxic Narcissists-- they recognize and appreciate when someone's doing them a public favor, and (IME) this compels them to respond in kind because they'll want to one-up you as everyone's savior. (We're starting to see this with all the moral crusading and self-appointed "safety" workers. All Narcissists.)

sokoloff 3y ago

I think we ran an effective process to cover the RCA area of our operations. Importantly (IMO), we did go to lengths to understand and ascribe actions/activities that resulted in/contributed to an outage to a specific individual; we were just careful to not assign individual consequences to them. I think it's critically important to understand as precisely as possible what happened, who did it, what they were looking at that caused them to take that action, and which parts of that [if any] we'd change with the benefit of hindsight. I ran the Ops team at the time and it was easy for me to enforce the lack of consequences for anything short of an intentionally destructive act. If "blameless post-mortem" means "we want to make sure that no one has any idea who was responsible", you can achieve that but you probably won't like the results. If it instead means "we want to know why it happened, who contributed, and why, so that we can not repeat it", you have a fighting chance. I've written and published multiple RCAs that explain in detail why /u/sokoloff caused an outage, when it started, when it was contained, and how to avoid that mistake in the future. I think that trying to obscure who did something is not only not worth the effort, but is actively destructive to the learning and trust. If I can't trust that my name can appear next to an honest mistake, what else must I be distrustful of? If instead, I see respected, senior staff readily taking responsibility and sharing their mistakes without fear of consequences, I trust my company's leaders more, not less.