4 ms·

>The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. It's only an issue when people

by jasode 1mo ago

>The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that the failed O-rings were not the single root cause of the Challenger explosion. Another cause was management normalizing the deviations of previous unsafe datapoints of prior launches which let them greenlight the launch in freezing temperatures. Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. It wasn't The Rogers Commission that found the multiple causes; it was NASA engineers and management themselves explaining the multiple causes as they were interviewed by the Rogers Commission members. For whatever reason, alternative jargon such as "Root Causes Analysis" (plural) or "Proximate and Distal Causes Analysis" isn't as widely used.

jonahx 1mo ago
> Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. In the case of Challenger, I think it's pretty clear this was the "root cause": > According to testimony by Kilminster and Boisjoly, Mason finally turned to Bob Lund and said, "Take off your engineering hat and put on your management hat." Joe Kilminster wrote out the new recommendation and went back on line with the teleconference. > The new recommendation stated that the cold was still a safety concern, but their people had found that the original data was indeed inconclusive and their "engineering assessment" was that launch was recommended, even though the engineers had no part in writing the new recommendation and refused to sign it. -- https://onlineethics.virginia.edu/cases/engineering-ethics-cases-texas-am/space-shuttle-challenger-disaster?utm_source=chatgpt.com https://onlineethics.virginia.edu/cases/engineering-ethics-c... If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.
- otterley 1mo ago
  If you’re doing “five* whys” analyses correctly, you don’t stop at the technical causes. You continue further to analyze the causes that precipitated the technical errors, too. This exposes the business reasons behind them and forces management to face them. * the number five isn’t magical here. You don’t have to stop at five, and you often shouldn’t.
- majormajor 1mo ago
  A deeper cause there, as I understand it, is that the statistical analysis was incomplete/flawed but in a way that none of the technical practitioners caught and called out at the time. The way the data was presented was to the effect of "half of the o-ring damage incidents were in cold temp launches", which made things seem less bad. The way it could have been presented was more like "nearly every cold temp launch has resulted in damage, only like 10% of others do" (I forget the exact numbers) which is far more effective highlighting the impact of the temperature and the magnitude of the increased risk. And that would've let people make a stronger case "hey, we know temps in the 50s cause damage almost every time, and it's way colder today." There's a "system" aspect (outside of power structures and incentives) which is that prob/stats knowledge among almost all engineering disciplines (industrial engineering waves from the sidelines) is exceedingly poor and often viewed as "soft" and "less important" than calculus, linear algebra, etc. And so practitioners are ill-equipped at spotting things like that ("wait, is this the right denominator? what about frequency of incident?").
jdub 1mo ago
It's not just a matter of people taking the phrase literally: - by a plain reading it's clearly singular; seeing beyond that requires at least some effort (e.g. curiosity, education... but also authority, responsibility, time) - there are lots of people dealing with complex system failures that haven't been exposed to any of the relevant theory... tech folk, but also mgmt, comms (which can be more problematic) - concluding with a single cause is frequently less work - concluding with a single cause is frequently convenient (e.g. pinning the blame on a single vendor, component, person) That's why it's helpful to introduce terminology that points everyone's brains in the right direction from the very start, e.g. contributing factors. (also common sense isn't real)
- justsomehnguy 1mo ago
  - concluding what finding someone to blame not in my department is inherently preferential to anything else > (also common sense isn't real) I prefer Murphy's styled variant: - common sense isn't

by jasode 1mo ago

jonahx 1mo ago

> Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. In the case of Challenger, I think it's pretty clear this was the "root cause": > According to testimony by Kilminster and Boisjoly, Mason finally turned to Bob Lund and said, "Take off your engineering hat and put on your management hat." Joe Kilminster wrote out the new recommendation and went back on line with the teleconference. > The new recommendation stated that the cold was still a safety concern, but their people had found that the original data was indeed inconclusive and their "engineering assessment" was that launch was recommended, even though the engineers had no part in writing the new recommendation and refused to sign it. -- https://onlineethics.virginia.edu/cases/engineering-ethics-cases-texas-am/space-shuttle-challenger-disaster?utm_source=chatgpt.com https://onlineethics.virginia.edu/cases/engineering-ethics-c... If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.

otterley 1mo ago
If you’re doing “five* whys” analyses correctly, you don’t stop at the technical causes. You continue further to analyze the causes that precipitated the technical errors, too. This exposes the business reasons behind them and forces management to face them. * the number five isn’t magical here. You don’t have to stop at five, and you often shouldn’t.
majormajor 1mo ago
A deeper cause there, as I understand it, is that the statistical analysis was incomplete/flawed but in a way that none of the technical practitioners caught and called out at the time. The way the data was presented was to the effect of "half of the o-ring damage incidents were in cold temp launches", which made things seem less bad. The way it could have been presented was more like "nearly every cold temp launch has resulted in damage, only like 10% of others do" (I forget the exact numbers) which is far more effective highlighting the impact of the temperature and the magnitude of the increased risk. And that would've let people make a stronger case "hey, we know temps in the 50s cause damage almost every time, and it's way colder today." There's a "system" aspect (outside of power structures and incentives) which is that prob/stats knowledge among almost all engineering disciplines (industrial engineering waves from the sidelines) is exceedingly poor and often viewed as "soft" and "less important" than calculus, linear algebra, etc. And so practitioners are ill-equipped at spotting things like that ("wait, is this the right denominator? what about frequency of incident?").

jdub 1mo ago

It's not just a matter of people taking the phrase literally: - by a plain reading it's clearly singular; seeing beyond that requires at least some effort (e.g. curiosity, education... but also authority, responsibility, time) - there are lots of people dealing with complex system failures that haven't been exposed to any of the relevant theory... tech folk, but also mgmt, comms (which can be more problematic) - concluding with a single cause is frequently less work - concluding with a single cause is frequently convenient (e.g. pinning the blame on a single vendor, component, person) That's why it's helpful to introduce terminology that points everyone's brains in the right direction from the very start, e.g. contributing factors. (also common sense isn't real)

justsomehnguy 1mo ago
- concluding what finding someone to blame not in my department is inherently preferential to anything else > (also common sense isn't real) I prefer Murphy's styled variant: - common sense isn't