4 ms·

99% of folks here don’t deal with life and death matters. When you do, you get a lot of practice at canceling, which helps. There’s also the fact that you can s

99% of folks here don’t deal with life and death matters. When you do, you get a lot of practice at canceling, which helps. There’s also the fact that you can say, “if you do this, people will die,” which is a pretty bright line when you’ve got something to back it up. I actually found it more difficult to take a stand in less consequential fields. But it’s still very, very hard to look someone more powerful in the eye and tell them no. I’ve had to do it and it’s not fun. (Nothing on the scale of a Shuttle launch. I once had to effectively revoke a judge’s flying privileges, for instance.) McDonald is on a whole other level though. That’s not just your employer, that’s the entire space program. I hope I never have to find out whether I’m made of the same stuff he was.

bumby 6y ago
>if you do this, people will die In my experience, this is not how it works. Very rarely can you say definitively that “people will die”. Reliability is shrouded in uncertainty and the best you can say is, “people may die, eventually ”. If you make too absolute of a claim and it didn’t occur you will gradually erode credibility. Look at Columbia... they knew the foam shedding was out of spec but had so many instances of probability being on their side, if you claimed people would die act each of those launches soon you just become noise. Over a long enough timeline you may be right, but phrasing in that way is unlikely to help in low probability events.
- rdtwo 6y ago
  This man understands. It’s the .001% chance that kills people
  - inglor_cz 6y ago
    There was an anecdote from one of Czech researches in Papua-New Guinea who said he was surprised that local hunters absolutely refused to sleep under suspicious, not entirely healthy trees, and were very careful about examining the spot before they chose it for rest. He considered the chance of the trees falling down very low, perhaps less than one in thousand. He changed his attitude when the elders explained to him that while the chance was very low, the hunters took a lot of naps in the forest throughout their lives and over decades, the low chance translated to almost-certainty.
    - deleted 6y ago
      [deleted]
    - cafard 6y ago
      I think I read this in a column of Jared Diamond's.
  - gwd 6y ago
    No, it's the 10% that kills people. They knew the O-rings were failing because they had analyzed previously discarded rockets. So the engineers said, "We got lucky this time; eventually this is going to kill people." But after launching 4-5 times and having O-ring failures with no problems, management (and probably everyone) started thinking it wasn't really as dangerous as it was.
    - bumby 6y ago
      The percentage loses meaning without an expected number of opportunities to calculate the expected value. 10% risk x 1 flight might be deemed acceptable. 10% risk x 100 flights means you should plan on losing 10 shuttles. Neither statement about risk probability is wrong (0.001% vs 10%), but both are incomplete. The actual shuttle risk was calculated between 1/78 to 1/254, if I remember correctly
- vincnetas 6y ago
  aka Normalization of deviance https://en.wikipedia.org/wiki/Normalization_of_deviance https://en.wikipedia.org/wiki/Normalization_of_deviance And a short video in context of challenger accident : https://www.youtube.com/watch?v=tC0qkVf5SmE https://www.youtube.com/watch?v=tC0qkVf5SmE
  - interestica 6y ago
    Mullane was on STS-27 -- that was a disaster on the level of Challenger or Columbia but by pure luck they avoided a deadly ending. The failures were predictable. And this was the second flight after Challenger's last.
    - bumby 6y ago
      I think the underlying point is that there are cognitive biases that prevent decision makers from effectively understanding that risk. It’s very much like financial crashes; after the fact it becomes obvious the risk was there but our biases prevent most people from seeing/understanding it effectively in the moment
- jacquesm 6y ago
  It's enough that people may die (or even get injured) as a consequence of your work. That's the line you shouldn't cross and it actually helps to have such a clear and unambiguous definition of that line.
  - bumby 6y ago
    I understand that stance, but when humans are the decision makers and have all kinds of other pressures (cost, schedule, etc.) it’s often not enough that people “may” die. Especially if we’re talking about risky endeavors like human-rated space travel, “may” is only one level across the gradient, and that level could be considered an acceptable risk by some. You will never drive risk down to zero, so “people may die” will almost always be true. If that’s your threshold, you will never launch. The real difficulty is accurately gauging risk, especially when there isn’t substantial data.
- interestica 6y ago
  Sts-27 https://youtu.be/BswkvaAaqSM https://youtu.be/BswkvaAaqSM 1988. Damage to orbiter Atlantis. Not the same foam that shedded (pieces from SRB rather than external tank). But still demonstrated failure modes.
  - bumby 6y ago
    The failure mode was known (hence why it was considered out-of-spec) but it didn’t translate to an accurate risk assessment. It was deemed “in family” meaning it was known to be outside of spec but not a risk. Somehow, over many flights, something that was originally characterized as a safety risk lost its credibility as a risk. It’s not that it was unknown, but the understanding of the risk was changed. I wonder if STS-27 led to a false sense of confidence
tpaulin 6y ago
I'm curious to hear what you think about the Therac-25 situation because the line between a code change and fatalities isn't as clear cut as with Challenger, but definitely still there.

bumby 6y ago

>if you do this, people will die In my experience, this is not how it works. Very rarely can you say definitively that “people will die”. Reliability is shrouded in uncertainty and the best you can say is, “people may die, eventually ”. If you make too absolute of a claim and it didn’t occur you will gradually erode credibility. Look at Columbia... they knew the foam shedding was out of spec but had so many instances of probability being on their side, if you claimed people would die act each of those launches soon you just become noise. Over a long enough timeline you may be right, but phrasing in that way is unlikely to help in low probability events.

rdtwo 6y ago
This man understands. It’s the .001% chance that kills people
- inglor_cz 6y ago
  There was an anecdote from one of Czech researches in Papua-New Guinea who said he was surprised that local hunters absolutely refused to sleep under suspicious, not entirely healthy trees, and were very careful about examining the spot before they chose it for rest. He considered the chance of the trees falling down very low, perhaps less than one in thousand. He changed his attitude when the elders explained to him that while the chance was very low, the hunters took a lot of naps in the forest throughout their lives and over decades, the low chance translated to almost-certainty.
  - deleted 6y ago
    [deleted]
  - cafard 6y ago
    I think I read this in a column of Jared Diamond's.
- gwd 6y ago
  No, it's the 10% that kills people. They knew the O-rings were failing because they had analyzed previously discarded rockets. So the engineers said, "We got lucky this time; eventually this is going to kill people." But after launching 4-5 times and having O-ring failures with no problems, management (and probably everyone) started thinking it wasn't really as dangerous as it was.
  - bumby 6y ago
    The percentage loses meaning without an expected number of opportunities to calculate the expected value. 10% risk x 1 flight might be deemed acceptable. 10% risk x 100 flights means you should plan on losing 10 shuttles. Neither statement about risk probability is wrong (0.001% vs 10%), but both are incomplete. The actual shuttle risk was calculated between 1/78 to 1/254, if I remember correctly
vincnetas 6y ago
aka Normalization of deviance https://en.wikipedia.org/wiki/Normalization_of_deviance https://en.wikipedia.org/wiki/Normalization_of_deviance And a short video in context of challenger accident : https://www.youtube.com/watch?v=tC0qkVf5SmE https://www.youtube.com/watch?v=tC0qkVf5SmE
- interestica 6y ago
  Mullane was on STS-27 -- that was a disaster on the level of Challenger or Columbia but by pure luck they avoided a deadly ending. The failures were predictable. And this was the second flight after Challenger's last.
  - bumby 6y ago
    I think the underlying point is that there are cognitive biases that prevent decision makers from effectively understanding that risk. It’s very much like financial crashes; after the fact it becomes obvious the risk was there but our biases prevent most people from seeing/understanding it effectively in the moment
jacquesm 6y ago
It's enough that people may die (or even get injured) as a consequence of your work. That's the line you shouldn't cross and it actually helps to have such a clear and unambiguous definition of that line.
- bumby 6y ago
  I understand that stance, but when humans are the decision makers and have all kinds of other pressures (cost, schedule, etc.) it’s often not enough that people “may” die. Especially if we’re talking about risky endeavors like human-rated space travel, “may” is only one level across the gradient, and that level could be considered an acceptable risk by some. You will never drive risk down to zero, so “people may die” will almost always be true. If that’s your threshold, you will never launch. The real difficulty is accurately gauging risk, especially when there isn’t substantial data.
interestica 6y ago
Sts-27 https://youtu.be/BswkvaAaqSM https://youtu.be/BswkvaAaqSM 1988. Damage to orbiter Atlantis. Not the same foam that shedded (pieces from SRB rather than external tank). But still demonstrated failure modes.
- bumby 6y ago
  The failure mode was known (hence why it was considered out-of-spec) but it didn’t translate to an accurate risk assessment. It was deemed “in family” meaning it was known to be outside of spec but not a risk. Somehow, over many flights, something that was originally characterized as a safety risk lost its credibility as a risk. It’s not that it was unknown, but the understanding of the risk was changed. I wonder if STS-27 led to a false sense of confidence

tpaulin 6y ago

I'm curious to hear what you think about the Therac-25 situation because the line between a code change and fatalities isn't as clear cut as with Challenger, but definitely still there.