5 ms·

I'm the author of this piece, happy to answer questions. I grew up with stories of the Challenger after my father - a statistician - and 2 of his co-authors we

by nemild 10y ago

I'm the author of this piece, happy to answer questions. I grew up with stories of the Challenger after my father - a statistician - and 2 of his co-authors were selected by National Academy of Sciences to study if the danger could have been predicted beforehand. They showed that the likelihood of failure was 13% at the launch temperature, but would have been negligible if NASA had waited just a few hours. (His co-author, Ed Fowlkes, was dying of AIDS at the time - and considered this paper one of his life's great achievements) Bad statistical inferences were a huge part of the launch story, and you can see more in Richard Feynman's critiques: https://en.wikipedia.org/wiki/Rogers_Commission_Report https://en.wikipedia.org/wiki/Rogers_Commission_Report Secondly, the effect I highlight (a biased data sample) is a key issue with news/social media - and can lead us to heavily flawed inferences if we don't correct for it. I'll dig deep into this in future posts with a substantial amount of data and visualizations.

kevinpet 10y ago
Feynman's actually observations are well worth reading for anyone who builds anything that may be vaguely considered engineering. http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger... The key items for me were: 1) While they had no expectation of erosion, and the design did not call for the o-rings to erode, once they observed them eroding, they retroactively invented a "margin of error" based on what fraction the o-rings eroded. This was not based on an actual understood process, and is akin to saying "well, the bridge didn't break when we drove that truck over it, so it must be okay" 2) The engineers actually knew the risk (~1% chance of loss per launch, not specific to the o-rings, compared with two actual losses of the shuttle over ~130 missions). Management used entirely invented numbers for the risk which were not justified.
- JohnJamesRambo 10y ago
  Since 1 part in 100,000 would imply that one could put a Shuttle up each day for 300 years expecting to lose only one, we could properly ask "What is the cause of management's fantastic faith in the machinery?" I wonder if you told the management that, in those words, if they would have still believed such a ludicrous idea.
- smallnamespace 10y ago
  There's a lot of blame that (deservedly) gets pinned on the NASA administrators, but this fails to ask the really important question -- what sort of political and other pressures were put on the administrators such that they felt compelled to make shit up? Seems like it was politically impossible for NASA to say outright that there was a 1% chance of failure for every launch -- it would have led to loss of public support for the Shuttle program. So we have a systems failure where both politicians and the public contributed by making NASA admins feel compelled to lie and cover up to make the launches work. I mean, the courageous thing to do would be to stand up and say space travel is inherently risky, people might die, but it's still worth it. But courageous politicians regularly get voted out of office, and I would bet a courageous NASA admin who said that would end up fired.
  - nemild 10y ago
    You're absolutely right, but after so much success there also seems to be a fair amount of confidence/arrogance that set in at NASA regardless of other pressures. Additionally, when you have a civilian on board, I think it really changes how you think about what an appropriate level of risk is (13% might have been ok with professional astronauts who knew the risks beforehand, but likely was too high for a civilian). And the line engineers at Morton Thiokol fought back pretty hard on the decision, even if it might have impacted their careers negatively.
    - vkou 10y ago
      At that point in space exploration, 13% was way too high a risk, even for professional astronauts.
- avar 10y ago
  Your paraphrasing of Feynman's bridge quote is inaccurate. From Appendix F[1] of the report: [..] In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights. For example. in determining if flight 51-L was safe to fly in the face of ring erosion in flight 51-C, it was noted that the erosion depth was only one-third of the radius. It had been noted in an experiment cutting the ring that cutting it as deep as one radius was necessary before the ring failed. Instead of being very concerned that variations of poorly understood conditions might reasonably create a deeper erosion this time, it was asserted, there was "a safety factor of three." This is a strange use of the engineer's term ,"safety factor." If a bridge is built to withstand a certain load without the beams permanently deforming, cracking, or breaking, it may be designed for the materials used to actually stand up under three times the load. This "safety factor" is to allow for uncertain excesses of load, or unknown extra loads, or weaknesses in the material that might have unexpected flaws, etc. If now the expected load comes on to the new bridge and a crack appears in a beam, this is a failure of the design. There was no safety factor at all; even though the bridge did not actually collapse because the crack went only one-third of the way through the beam. The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred. His point about NASA's nonsensical use of the "safety factor" is not that you could drive over a bridge, and look, it didn't break, so it must be OK! It's even worse, you drive a truck over it, afterwards 1/3 of the steel is cracked, so you conclude that it must be able to safely accept 3x the weight. Nonesense! This is the sort of moronic engineering that killed the crew of the Challenger. 1. http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger...
  - masklinn 10y ago
    > This is the sort of moronic engineering that killed the crew of the Challenger. Note that it wasn't even engineering, reading the story in full (it's great, and covers the software side for which Feynman had nothing but praise) Feynman repeatedly noted that engineers were fairly realistic[0] and had been ringing alarms pretty much all along, this was entirely manglement mangling. [0] unless the spectre of manglement was involved, at least for some of them
nickff 10y ago
What about using similar techniques on other problematic parts which did not have disastrous failures? If you had used the same reasoning and analysis, how many flights (which ended up successful) would have been delayed or cancelled? Are you not falling victim to a sort of survivor-ship bias in only applying these analyses to 'failed' missions? It seems obvious in hindsight that the 'whistle-blowers' were right, but how many people voiced concerns which turned out to be erroneous?
- nemild 10y ago
  While the graphs I highlight don't show it, the broader paper uses both actual launch data AND experimental data to derive the regression. This experimental data presumably should be available regardless of if this reasoning had been used earlier on launch data. This is one good reason why experimental data is so critical, as real world data can be biased. You're absolutely right though about this being "Monday morning quarterbacking" with any analysis after the events. That said, two things are clear: 1. The most knowledgeable engineers were strongly against launch (see Ebeling and Boisjoly at Morton Thiokol) 2. There were huge statistical flaws made by decision makers at NASA before launch (see Feynman as noted in this thread)
- byoung2 10y ago
  The entire shuttle program was plagued by a pattern of ignoring safety objections. It was widely known ahead of disasters that the whistle-blowers were right about many issues, but were ignored. I made this point in another HN thread about Challenger, but if you drive drunk on a regular basis, it is not safe and it is only a matter of time before you crash and kill a family of four. The fact that you have driven drunk for 20 years and haven't killed anyone yet doesn't change the fact that driving drunk was dangerous. Evidence of this phenomenon in the shuttle program was that foam shedding (insulation for the main fuel tank falling off during launch) was observed as early as 1983, and had been noted as a substantial risk many times over the years. Engineers pressed for high resolution images to inspect damage, but those requests were denied. NASA continued to "drive drunk" over the years, even after Challenger, and inevitably, disaster struck when Columbia was damaged by a piece of foam that broke off during launch, resulting in disintegration during reentry.[1] After grounding the fleet and improving safety, the very next launch suffered similar foam shedding, though from a different part of the tank. The launch after that was also pressed, after delays and objections from the chief engineer and safety officer. 1. https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste...
kordless 10y ago
I'd suggest mentioning Feynman in the article, given his contributions were what made them face the truth of the matter.
- nemild 10y ago
  Thanks - I had put it in a footnote that got cut, but will edit.
davidf18 10y ago
The paper is discussing denominators. They weren't taking into account total missions at a certain temperature (the denominator), only the numerator (# failures). One should always think of denominators (really ratio of numerator over denominator). In health, public heath in particular is "about denominators."
deleted 10y ago
[deleted]
mturmon 10y ago
A bit OT, but is Charless Fowlkes (now at UC Irvine) a relation of the Fowlkes who was your father's collaborator? A long time ago he interned in my group at JPL.

by nemild 10y ago

kevinpet 10y ago

Feynman's actually observations are well worth reading for anyone who builds anything that may be vaguely considered engineering. http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger... The key items for me were: 1) While they had no expectation of erosion, and the design did not call for the o-rings to erode, once they observed them eroding, they retroactively invented a "margin of error" based on what fraction the o-rings eroded. This was not based on an actual understood process, and is akin to saying "well, the bridge didn't break when we drove that truck over it, so it must be okay" 2) The engineers actually knew the risk (~1% chance of loss per launch, not specific to the o-rings, compared with two actual losses of the shuttle over ~130 missions). Management used entirely invented numbers for the risk which were not justified.

JohnJamesRambo 10y ago
Since 1 part in 100,000 would imply that one could put a Shuttle up each day for 300 years expecting to lose only one, we could properly ask "What is the cause of management's fantastic faith in the machinery?" I wonder if you told the management that, in those words, if they would have still believed such a ludicrous idea.
smallnamespace 10y ago
There's a lot of blame that (deservedly) gets pinned on the NASA administrators, but this fails to ask the really important question -- what sort of political and other pressures were put on the administrators such that they felt compelled to make shit up? Seems like it was politically impossible for NASA to say outright that there was a 1% chance of failure for every launch -- it would have led to loss of public support for the Shuttle program. So we have a systems failure where both politicians and the public contributed by making NASA admins feel compelled to lie and cover up to make the launches work. I mean, the courageous thing to do would be to stand up and say space travel is inherently risky, people might die, but it's still worth it. But courageous politicians regularly get voted out of office, and I would bet a courageous NASA admin who said that would end up fired.
- nemild 10y ago
  You're absolutely right, but after so much success there also seems to be a fair amount of confidence/arrogance that set in at NASA regardless of other pressures. Additionally, when you have a civilian on board, I think it really changes how you think about what an appropriate level of risk is (13% might have been ok with professional astronauts who knew the risks beforehand, but likely was too high for a civilian). And the line engineers at Morton Thiokol fought back pretty hard on the decision, even if it might have impacted their careers negatively.
  - vkou 10y ago
    At that point in space exploration, 13% was way too high a risk, even for professional astronauts.
avar 10y ago
Your paraphrasing of Feynman's bridge quote is inaccurate. From Appendix F[1] of the report: [..] In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights. For example. in determining if flight 51-L was safe to fly in the face of ring erosion in flight 51-C, it was noted that the erosion depth was only one-third of the radius. It had been noted in an experiment cutting the ring that cutting it as deep as one radius was necessary before the ring failed. Instead of being very concerned that variations of poorly understood conditions might reasonably create a deeper erosion this time, it was asserted, there was "a safety factor of three." This is a strange use of the engineer's term ,"safety factor." If a bridge is built to withstand a certain load without the beams permanently deforming, cracking, or breaking, it may be designed for the materials used to actually stand up under three times the load. This "safety factor" is to allow for uncertain excesses of load, or unknown extra loads, or weaknesses in the material that might have unexpected flaws, etc. If now the expected load comes on to the new bridge and a crack appears in a beam, this is a failure of the design. There was no safety factor at all; even though the bridge did not actually collapse because the crack went only one-third of the way through the beam. The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred. His point about NASA's nonsensical use of the "safety factor" is not that you could drive over a bridge, and look, it didn't break, so it must be OK! It's even worse, you drive a truck over it, afterwards 1/3 of the steel is cracked, so you conclude that it must be able to safely accept 3x the weight. Nonesense! This is the sort of moronic engineering that killed the crew of the Challenger. 1. http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger...
- masklinn 10y ago
  > This is the sort of moronic engineering that killed the crew of the Challenger. Note that it wasn't even engineering, reading the story in full (it's great, and covers the software side for which Feynman had nothing but praise) Feynman repeatedly noted that engineers were fairly realistic[0] and had been ringing alarms pretty much all along, this was entirely manglement mangling. [0] unless the spectre of manglement was involved, at least for some of them

nickff 10y ago

What about using similar techniques on other problematic parts which did not have disastrous failures? If you had used the same reasoning and analysis, how many flights (which ended up successful) would have been delayed or cancelled? Are you not falling victim to a sort of survivor-ship bias in only applying these analyses to 'failed' missions? It seems obvious in hindsight that the 'whistle-blowers' were right, but how many people voiced concerns which turned out to be erroneous?

nemild 10y ago
While the graphs I highlight don't show it, the broader paper uses both actual launch data AND experimental data to derive the regression. This experimental data presumably should be available regardless of if this reasoning had been used earlier on launch data. This is one good reason why experimental data is so critical, as real world data can be biased. You're absolutely right though about this being "Monday morning quarterbacking" with any analysis after the events. That said, two things are clear: 1. The most knowledgeable engineers were strongly against launch (see Ebeling and Boisjoly at Morton Thiokol) 2. There were huge statistical flaws made by decision makers at NASA before launch (see Feynman as noted in this thread)
byoung2 10y ago
The entire shuttle program was plagued by a pattern of ignoring safety objections. It was widely known ahead of disasters that the whistle-blowers were right about many issues, but were ignored. I made this point in another HN thread about Challenger, but if you drive drunk on a regular basis, it is not safe and it is only a matter of time before you crash and kill a family of four. The fact that you have driven drunk for 20 years and haven't killed anyone yet doesn't change the fact that driving drunk was dangerous. Evidence of this phenomenon in the shuttle program was that foam shedding (insulation for the main fuel tank falling off during launch) was observed as early as 1983, and had been noted as a substantial risk many times over the years. Engineers pressed for high resolution images to inspect damage, but those requests were denied. NASA continued to "drive drunk" over the years, even after Challenger, and inevitably, disaster struck when Columbia was damaged by a piece of foam that broke off during launch, resulting in disintegration during reentry.[1] After grounding the fleet and improving safety, the very next launch suffered similar foam shedding, though from a different part of the tank. The launch after that was also pressed, after delays and objections from the chief engineer and safety officer. 1. https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste...

kordless 10y ago

I'd suggest mentioning Feynman in the article, given his contributions were what made them face the truth of the matter.

nemild 10y ago
Thanks - I had put it in a footnote that got cut, but will edit.

davidf18 10y ago

The paper is discussing denominators. They weren't taking into account total missions at a certain temperature (the denominator), only the numerator (# failures). One should always think of denominators (really ratio of numerator over denominator). In health, public heath in particular is "about denominators."

deleted 10y ago

[deleted]

mturmon 10y ago

A bit OT, but is Charless Fowlkes (now at UC Irvine) a relation of the Fowlkes who was your father's collaborator? A long time ago he interned in my group at JPL.