5 ms·

Hah #7 really hits home. Every RCA I’ve been a part of always ends up pointing to systemic failures in the org at the top level, because walking the tree always

Hah #7 really hits home. Every RCA I’ve been a part of always ends up pointing to systemic failures in the org at the top level, because walking the tree always leads there. You can’t blame any one person or system for a failure in isolation. It’s usually some form of, “this is ultimately a consequence of miscalibrating risk associated to business/financial decisions.” I forget where I heard this but, “you manage risk, but risk cannot be managed.” Ie. there is no terminal state where it’s been “solved.” It’s much like “culture of safety.”

anonymars 1y ago
I was pondering this a bit recently while going through The Wire Though unsatisfying it feels like a lot boils down to "shit rolls downhill" or "fish rot from the head down"
- jaggederest 1y ago
  Ultimately this is what management is at least nominally supposed to do: decide values, set standards, and ensure that people down the chain are strategically aligned and have an environment where everyone is working together. That includes Andy Grove's "getting people to resolve their own differences" and other meta tasks. But fundamentally the person in charge is responsible for everything, even though they can't know everything. Any RCA that doesn't provide useful feedback to management up and down the chain is missing pieces, but there's lots of discussion elsewhere in this thread about that by people better at elaborating than I am.
crazygringo 1y ago
Of course, and that's very much the point. If you get a bunch of analyses that point to underinvestment in x in order to achieve y, and you can measure that this is losing money, then the top level recalibrates. It's not about blame, it's about course-correcting including at the top level. They can't do that course-correcting without these analyses.
- hinkley 1y ago
  As one of my early mentorish people used to say: sometimes you have to let the baby fall down. Before that I hated it when people confessed to me that they knew a problem was going to happen and they did nothing to stop it. But the problem with doing things right the first time is that nobody appreciates how hard it is. You spend all your political capital trying to stop the company from making a mistake it desperately wants to make. You get no credit when it doesn't happen because they already figured it wouldn't. And you don't get any buy-in for keeping it from repeating in the future. So now you have this plate you have to spin all by yourself, and nobody who can hand your more control gives a shit other than the fact that you seem like an asshole so we aren't going to promote you.
  - kqr 1y ago
    This is the hidden cost of heroics[1]: it severs an important information channel. [1]: https://entropicthoughts.com/hidden-cost-of-heroics https://entropicthoughts.com/hidden-cost-of-heroics
  - anonymars 1y ago
    I feel like that's what happening nowadays with vaccines and the like: victims of their own success
hinkley 1y ago
You can be clever though and put systems in place where effectively the people squished in the middle now have new places where they can just say 'no' because the process requires it. That is sort of how Agile got out there in the first place. Open feedback loops letting management be stupid for so long before the consequences were apparent that they could duck their vast contributions to it.
kqr 1y ago
> “you manage risk, but risk cannot be managed.” Ie. there is no terminal state where it’s been “solved.” I often hear people misinterpret "risk management" as if it means "risk minimisation", but this is the first time I hear of "risk elimination"! Risk management is about finding an appropriate level of risk. Could be lower, could be higher.

anonymars 1y ago

I was pondering this a bit recently while going through The Wire Though unsatisfying it feels like a lot boils down to "shit rolls downhill" or "fish rot from the head down"

jaggederest 1y ago
Ultimately this is what management is at least nominally supposed to do: decide values, set standards, and ensure that people down the chain are strategically aligned and have an environment where everyone is working together. That includes Andy Grove's "getting people to resolve their own differences" and other meta tasks. But fundamentally the person in charge is responsible for everything, even though they can't know everything. Any RCA that doesn't provide useful feedback to management up and down the chain is missing pieces, but there's lots of discussion elsewhere in this thread about that by people better at elaborating than I am.

crazygringo 1y ago

Of course, and that's very much the point. If you get a bunch of analyses that point to underinvestment in x in order to achieve y, and you can measure that this is losing money, then the top level recalibrates. It's not about blame, it's about course-correcting including at the top level. They can't do that course-correcting without these analyses.

hinkley 1y ago
As one of my early mentorish people used to say: sometimes you have to let the baby fall down. Before that I hated it when people confessed to me that they knew a problem was going to happen and they did nothing to stop it. But the problem with doing things right the first time is that nobody appreciates how hard it is. You spend all your political capital trying to stop the company from making a mistake it desperately wants to make. You get no credit when it doesn't happen because they already figured it wouldn't. And you don't get any buy-in for keeping it from repeating in the future. So now you have this plate you have to spin all by yourself, and nobody who can hand your more control gives a shit other than the fact that you seem like an asshole so we aren't going to promote you.
- kqr 1y ago
  This is the hidden cost of heroics[1]: it severs an important information channel. [1]: https://entropicthoughts.com/hidden-cost-of-heroics https://entropicthoughts.com/hidden-cost-of-heroics
- anonymars 1y ago
  I feel like that's what happening nowadays with vaccines and the like: victims of their own success

hinkley 1y ago

You can be clever though and put systems in place where effectively the people squished in the middle now have new places where they can just say 'no' because the process requires it. That is sort of how Agile got out there in the first place. Open feedback loops letting management be stupid for so long before the consequences were apparent that they could duck their vast contributions to it.

kqr 1y ago

> “you manage risk, but risk cannot be managed.” Ie. there is no terminal state where it’s been “solved.” I often hear people misinterpret "risk management" as if it means "risk minimisation", but this is the first time I hear of "risk elimination"! Risk management is about finding an appropriate level of risk. Could be lower, could be higher.