28 ms·

The result would be a catastrophe (1985)

381 points by moviuro 11y ago

tokenadult 11y ago
I remember the Challenger explosion like it was yesterday. I'll link here to a scan of the pages from Edward Tufte's book Visual Explanations about what went wrong with the data analysis during the launch planning, which understated the risk of a launch in cold weather. http://williamwolff.org/wp-content/uploads/2013/01/tufte-challenger-1997.pdf http://williamwolff.org/wp-content/uploads/2013/01/tufte-cha...
- nehushtan 11y ago
  But Boisjoly rejected Tufte's analysis http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i...
  - pmahoney 11y ago
    Can you summarize one or two of Boisjoly's main points? Tufte's pictures of the (superfluous) rocket graphics showing the temperature vs. his redrawing, placing temperature on the x-axis are very convincing.
    - jasode 11y ago
      Boisjoly's rebuttal is that 1) Tufte had temperature datapoints for the previous launches that the Boisjoly's team didn't know[1]. I assume "know" to mean the complete historical data wasn't all consolidated conveniently at his fingertips before the disaster. Presumably, Boisjoly's team could have gotten it if it had occurred to them to gather it. Essentially, Tufte had the benefit of "hindsight" to make his compelling diagram showing cause & effect. 2) GIGO garbage in is garbage out. The Tufte temperature datapoint assumes that the outside ambient temperature is equal to the O-ring temperature[2] so substituting one for the other is wrong. (E.g. It's wrong to put them both on the same X-axis.) Quotes excerpted from http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i...: [1]"He thus supposes that they knew the temperatures at launch of all the shuttles and, assuming they acted voluntarily, infers they were incompetent." [2]", in addition, mixes O-ring temperatures and ambient air temperature as though the two were the same."
      - m741 11y ago
        I thought the rebuttal was very well written, if a bit dense. For (1), you're correct that the engineers did not have the historical data. More than that, though, it did occur to them to request the data, but they were stifled by Morton Thiokol Management (and NASA). To start with, there were a variety of previous problems with the O-rings caused by variables that appear unrelated to temperature. These problems were resolved, but prevented anyone from seeing a pattern. It was only on the basis of the single data point in SRM 15 that Boisjoly requested temperature data in advance of the launch. Obtaining such data was far from simple, because, as you mention, the temperature of the O-ring isn't the same as the ambient air temperature. Thus, obtaining the data was relatively involved and required knowing many variables: time on the pad, the gradient of ambient temperature, the temperature at which testing was conducted, and so forth. For this reason, the engineers didn't compile the data themselves (unclear what process they'd need to get the data). The engineers thus requested the data in advance, but had not received it. They had precise data on only two data points (at 53 degrees and 75 degrees), so the rest of the data in the chart was compiled after the fact. "The data necessary for a calculation of O-ring temperatures was thus not collected all along during the shuttle history. And when Boisjoly asked for that data in September, along with much other data, any one of which might have been the crucial missing piece to explain the anomalous cause, it was not supplied. In fact, the engineers received none of the data they requested."
    - m741 11y ago
      I'm no engineer, but I'll take a crack at it: * Both axes of the chart are inaccurate. The temperature on the X axis intermingles ambient and O-ring temperature, but they're not the same (imagine ocean temperature vs air temperature). The vertical axis, O-ring damage, is semi-relevant, but the important question is whether the O-rings held a seal (degree of blow-by). O-ring damage probably correlates, but seems like a proxy that may be misleading. * The data available in the chart was not available to the engineers at the time. There was only one previous failure (SRM 15) that led anyone to believe temperature itself was an issue. They had two valid temperature data points (SRM 15 and SRM 22), and correctly pointed out that another launch at 29 degrees was completely outside their tested range and would be inherently risky. It's not clear to me that a scatterplot with two datapoints would be that effective. Furthermore Boisjoly criticizes that Tufte presumed to judge the engineers, for the following reasons: * The engineers previously recommended no launch should occur, months in advance (due to previous O-ring issues), but NASA overruled them. * The engineers recommended, and their managers accepted, that the low launch temperature was outside their test database. NASA came back and requested proof that the temperature was dangerous, but the engineers could not comply, because the parameters were extreme enough that they had not tested them. They couldn't prove a negative. The managers at Morton Thiokol and NASA then jointly overruled the engineers. * The idea of putting together a chart was not something the engineers considered, because there were a variety of previous unrelated problems in O-rings (each resolved afterwards) that muddied that data, and muddies the data that Tufte presents. * Tufte himself failed to research or note the pivotal information, and thus misrepresented the situation the engineers were in, and the data available, while imputing that the engineers should be held morally responsible for not presenting data they didn't have (and then going on to present such data himself, compiled after the fact and at his own leisure, incorrectly).
  - ricardobeat 11y ago
    That reads like a knee-jerk reaction to the (implied) accusation that the engineers were at fault for not presenting their case correctly. It doesn't dismiss Tufte's arguments for clear visual presentation of data; you could say that both of them are right.
- MCRed 11y ago
  I walked in to the TV showing the launch. I hadn't known it had exploded. I didn't realize I was seeing a replay. I spent my time as a kid living on Vandenberg Air Force base, and my father would drag me out to Minute Men missile launches early in the mornings. I saw several launches. Even two launched at once. I had seen many launches before, so I had no expectation that they could go wrong. Watching the shuttle explode like that was shocking.
  - ascagnel_ 11y ago
    > I had seen many launches before, so I had no expectation that they could go wrong. This mindset is interesting. Issues like plane and train crashes grab headlines and are typically catastrophic, but end up being far more rare than, say, a fatal car crash. Similarly, the Space Shuttle had two major incidents in 30 years, and the Concord had only one, but they were very public and very catastrophic. Is it the rarity or severity that make them such big news?
    - mikeash 11y ago
      Rarity is certainly a big factor even from an objective viewpoint. Concorde instantaneously went from the safest airliner flying to the most dangerous with that crash, just because it didn't fly much. Flying on a Space Shuttle was one of the most dangerous activities out there, judging by the historical failure rate. It helps just in terms of grabbing your attention, too. "Man drives into lake" doesn't make the news. "Man drives Bugatti Veyron into lake" does, just because the unusual car makes it more interesting. In the case of Concorde and the Shuttle, the machines in question were not only rare, but beloved by many, and seen as a national symbol. It's almost like assassinating the country's leader.
- jqm 11y ago
  I saw the Challenger explosion live. I was a sophmore in High School in Florida and standing outside at lunch. Shuttle launches were a fairly regular thing and I'd seem many. But this time I could see the trails as parts fell away to earth. It wasn't like the fuel tanks the fell away usually. I was talking with a guy named Wes. He said "that doesn't look right". It didn't. After lunch there was announcement that the shuttle had exploded. It doesn't seem that long ago really. The weird thing is.. in that same amount of time in the future I'm going to be a 75 year old man:)
  - JustSomeNobody 11y ago
    Same. I was home sick and, as usual for a launch, went out to see it. Saw the split smoke plumes and knew it was wrong, having seen enough launches.
  - ck2 11y ago
    I still remember the words the announcer said just before it happened, it was that deeply etched into my brain. go for throttle up and whenever the news shows that Y trail plume it still makes me ill to this day
samwiseg 11y ago
As someone born in 1995, I don't have any real emotional connection to the Challenger explosion. It's pretty bizarre to think that children born after the 9/11 attacks will probably feel the same indifference.
- mturmon 11y ago
  If you are an engineer, it would be good to try to learn from this and similar accidents. I.e., noting your lack of emotional connection is beside the point, the question is, what can you take away from what happened?
  - samwiseg 11y ago
    Of course, it's extremely important. I'm currently listening to the Freakonomics podcast on the Challenger explosion[0]. I was only noting the difference being born a few years apart can make when it comes to significant shared cultural memories. [0]http://freakonomics.com/2015/05/20/failure-is-your-friend-a-freakonomics-radio-rebroadcast/ http://freakonomics.com/2015/05/20/failure-is-your-friend-a-...
    - btilly 11y ago
      You will notice this sort of thing more and more as time goes on. For me the "ah hah" moment was when it hit me that "Where were you when you heard about Challenger?" was my generation's version of, "Where were you when JFK was assassinated?" Another easily observed one is music. Very few people are aware of much that happened in popular music after they hit 25.
      - gradi3nt 11y ago
        I'm turning 25 in two weeks. So this is it? 2015 was my last year of music? Damn...it went by so fast!
        vlehto 11y ago
shams93 11y ago
I doubt anyone in the reagan admin could even understand this note its highly technical it was likely tossed out without ever being read.
- crystalmeph 11y ago
  This memo was not sent to the administration, it was sent to the management of Morton Thiokol, who could reasonably be expected to understand basic engineering phrases like "loss of human life." It's a real PITA to engineer safe systems (see e.g. IEC 61508 and ISO 13849 in industrial automation), but it saves lives, and if you're in a position where Sales, Production, etc. is trying to get you to rush the Engineering work so they can make deadline, you've got to find the backbone to say "no," even if it hurts you professionally.
  - madaxe_again 11y ago
    You'd think so, but they probably saw "loss of human life" and went "hysterical. ignore.". It's actually really hard to warn people. Warn them loudly and strongly and they think you're scaremongering and being unnecessarily negative. Warn them quietly, and nobody listens. Warn them just right, and they'll take it under consideration, but go ahead anyway. When of course the inevitable happens, as the guy who predicted doom, you'll be blamed, because "it wouldn't have happened if you hadn't made a self fulfilling prophecy".
- stevegoober 11y ago
  > I doubt anyone in the reagan admin could even understand this note Right... and maybe this is a problem with the authors of the note rather than the recipients? Know how to communicate to your target audience.
  - URSpider94 11y ago
    That's why he sent it to the VP of Engineering, who I'm sure understood everything in the letter.
DickingAround 11y ago
Today, there's no way I could get away with writing an entire page of context before getting to the impact. I wonder how much context the recipient had and what reading expectations were like back then...
- wylee 11y ago
  I would think that when someone writes a document like this that the context is really important for the record and that they're not so concerned with making an "impact". Edit: Grammar
- dtparr 11y ago
  Leaving aside the fact that the subject includes 'Potential Failure Criticality', the key impact line "The result would be a catastrophe of the highest order - loss of human life" occurs less than 150 words into this. It's the 5th sentence (though they're admittedly long sentences). Are you having serious engineering-related discussions with your management where they can't read 150 words before getting to the hook? (serious question, not snark) Is everyone communicating via twitter or something? (sorry, that was a bit of snark)
  - frandroid 11y ago
    They're really "thick" sentences. By the time you get there, your brain has had to process so much that you go "huh".
    - dtparr 11y ago
      Interesting. I'm curious, do you find the technical content to be the 'thickening agent' or the style? I work in a related field, and am vaguely familiar with both the general makeup of this type of rocket and this accident in general. I also tend to write longer, more complex sentences. I did not find it to be particularly 'thick', and I'm curious whether it's because I'm more familiar with the technical details and so avoided the glossy-eyed stare from that aspect or from the fact that a few dozen words in a sentence doesn't faze me. The former would mean this shouldn't bother anyone practiced in the arts of spaceflight (such as the VP of engineering), but the latter would have been a problem. Edit: stupid homophones.
viach 11y ago
If you don't stop the CO2 emission, the result would be a catastrophe. Lets see how it will turn out now. Edit: And yes, it is sarcasm, nothing changed or is going to change.
DanielBMarkham 11y ago
This was a terrible thing. So many saw this coming and nobody listened to them. In my mind, what made this tragedy even worse was the way the program itself was conducted. You learn new modes of transportation and hardware by using it, many times to the point of exhaustion. We should have built a dozen orbiters and flown the shit out of them through hell or high water, learning as we went along. Instead we built 5 and every time something happened we backed farther away from the entire manned space program. A lot of time was wasted, and the lessons we didn't learn? Somebody else is still going to have to learn them someday.
- mikeash 11y ago
  I think it's emblematic of the whole Shuttle program to look at the flight test program. There basically wasn't one. In particular, look at the various abort scenarios, and then look at the abort testing. Don't look too hard for the abort testing, because they didn't do any. There was pretty much zero resiliency in the system, and they knew it.
throwaway2380 11y ago
Boisjoly later revealed this memo to the presidential commission investigating the disaster and was then forced to leave Morton Thioklol after been shunned by disgruntled colleagues. I find this phenomenon so difficult to understand.
- csense 11y ago
  You just found out your company screwed up big, and it turned into people dying and a bunch of geopolitical egg on your country's face, but if everybody keeps quiet, then maybe you'll get lucky and nobody will figure out it was your fault. But one guy tells the world exactly how you screwed up. Now you're in fear of your reputation and job -- but if he'd just kept his mouth shut, then you might be okay.
- poof131 11y ago
  Seems fairly standard. He provided feedback that could have prevented a disaster and the managers ignored it. So at the very least, the managers would dislike him for the fact that he illuminated their incompetence, but even more so for the fact that he would be a reminder of the massive tragedy that ensued and for which they were partially to blame. With management's resentment, his peers would have a choice, maintain a friendly relationship with him and risk their careers and management’s wrath, or distance themselves from him. It’s sad, but the natural tendency of organizations is to close ranks in situations like this. It is incredibly tough to perform an open, honest, self-assessment. The procedures need to be in place to force the self-examination and strong leaders need to push for it. Even then, politics sometimes place the blame on the wrong people. I’ve seen this happen time and time again in the military from working with SF in Iraq [1] to joining a squadron just after a disaster [2]. [1] http://www.nytimes.com/2008/08/20/world/africa/20iht-20iraq.15446434.html http://www.nytimes.com/2008/08/20/world/africa/20iht-20iraq.... [2] https://en.wikipedia.org/wiki/2008_San_Diego_F/A-18_crash https://en.wikipedia.org/wiki/2008_San_Diego_F/A-18_crash
- camperman 11y ago
  You need to read a book called Systemantics by John Gall. It's a hilarious and accurate look at how large systems fail. Although parts are somewhat dated, the principles are the same. In particular his material on bureaucracies is relevant: when threatened, bureaucracies immediately fight for self-preservation. Boisjoly was threatening the bureaucracy and so had to be eliminated for it to survive. Its actions makes perfect sense in the light of this. Ugly, yes but understandable.
maho 11y ago
It seems almost unbelievable how such a stern, concrete warning can be ignored. Of course, we look at this memo with perfect 20/20 hindsight, knowing that the O-ring issue caused a disaster. I am genuinely curious why the warning was ignored. I am hesistant to believe it was through malice or sheer incompetence. Does upper managment get two overblown warnings per week by engineering, so that this critical warning was "drowned by the noise"? Or were there good (at the time!) reasons to focus on other issues first?
- tahoeskibum 11y ago
  That's a good point. It is easy to just say that the others were incompetent without knowing the rate of false positives ie. how frequently were such memos written talking about catastrophes. The issue of rate of false positives vs true positives (i.e.,real problem) is something that most engineers and managers don't have a good grasp of.
  - LoSboccacc 11y ago
    this is true for this early warning from the manufacturing company, but the launch day reasoning was chilling. I'll be back with links to the worse parts. https://news.ycombinator.com/item?id=9277408 https://news.ycombinator.com/item?id=9277408
- cmurf 11y ago
  I could see managers at NASA seeing their role as a union or labor leader, trying to keep their budget and their people employed at least as much a priority as safety. So they're willing to take more risks because they're in a position of having to balance an equation. And it may be that the flaw is this uni-dimensional hierarchy that we seem to like, that seem efficient most of the time, but in reality not only miss edge cases but can cause them. The managers simply did not always 100% trust the engineers. Ultimately, the people with the power over go/no-go were more managerial than engineers, were more concerned with the politics, cost, perception of delay than they were at risk assessment. All spending of money in any company or government is basically the same: someone at the top delegates the vast majority to someone else. And they choose that someone else based on trust. And the longer than chain is, the greater the likelihood someone is put in position where the chain of trust is just flawed. And you hope it gets exposed where people don't die, obviously. But that didn't happen here. And that's why a bunch of people got angry at Feynman as well as Boisjoly. They exposed that the wrong people were making key decisions. Had they been the right people, then loss of life wouldn't have happened. Anyone that wrong is going to be angry, it's a personality flip of the coin whether they get angry with themselves for their mistake or with others for exposing it.
sgentle 11y ago
If you haven't read Richard Feynman's appendix to the Rogers Commission report on the Challenger disaster, I really recommend it: http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger... It's very readable and goes into a bunch of other issues at NASA at the time beyond the O-ring failure, which Feynman treats as a symptom of more general engineering and cultural problems.
- danso 11y ago
  Feynman is well-known for his TV appearance where he dramatically demonstrated what happened with the O-rings, live...but in his memoir, "What Do You Care What Other People Think?", he credits being inspired by General Kutyna, who had asked him to join the commission. Feynman describes that a random comment from Kutyna led him down the path of revelation: > Then he says, “I was working on my carburetor this morning, and I was thinking: the shuttle took off when the temperature was 28 or 29 degrees. The coldest temperature previous to that was 53 degrees. You’re a professor; what, sir, is the effect of cold on the O-rings?” “Oh!” I said. “It makes them stiff. Yes, of course!” That’s all he had to tell me. It was a clue for which I got a lot of credit later, but it was his observation. A professor of theoretical physics always has to be told what to look for. He just uses his knowledge to explain the observations of the experimenters! However, Feynman admits at the end of his multi-chapter account of serving on the commission that Kutyna had coyly played Feynman: > Another thing I understand better now has to do with where the idea came from that cold affects the O-rings. It was General Kutyna who called me up and said, “I was working on my carburetor, and I was thinking: what is the effect of cold on the O-rings?” Well, it turns out that one of NASA’s own astronauts told him there was information, somewhere in the works of NASA, that the O-rings had no resilience whatever at low temperatures—and NASA wasn’t saying anything about it. But General Kutyna had the career of that astronaut to worry about, so the real question the General was thinking about while he was working on his carburetor was, “How can I get this information out without jeopardizing my astronaut friend?” His solution was to get the professor excited about it, and his plan worked perfectly. If we take Feynman at his word, to me this is a great description of how science and bureaucracy and invention (and, sometimes, disaster) works: it's not always just random eurekas, or study-real-hard-and-you'll-figure-it-out...sometimes the information is already well-known, and obvious, but for various logistical/political/bureaucratic reasons, it isn't immediately disseminated or explored. via: Feynman, Richard P. (2011-02-14). "What Do You Care What Other People Think?": Further Adventures of a Curious Character (Kindle Locations 2923-2930). W. W. Norton & Company. Kindle Edition.
stepanhruda 11y ago
Here is Richard Feynman talking about the issues https://www.youtube.com/watch?v=4kpDg7MjHps https://www.youtube.com/watch?v=4kpDg7MjHps Seems not very different from how companies are managed 30 years later.
madaxe_again 11y ago
"Ach, damned engineers, what do they know about engineering!?" ~ Management, upon reading the memo. Actually, more realistically: "Brinton, do you know what this Boisjoly guy is talking about?" "No Lund, I don't - his attitude is disappointing, we need team players." "Thanks Brinton - I'll bring it up with his manager for his next performance review."
- krschultz 11y ago
  It's easy to blame it on 'management', but it is important to note that author of the memo is himself a 'manager' at the time of writing the memo. There is way more depth to this situation than the simple hand waiving 'everyone that supervises me is stupid'.
- sjm-lbm 11y ago
  To be honest, I think you are - to an extent - falling into the same trap that the managers fell into at Thiokol: namely, assuming that your view of the situation is complete and that input from other work groups can be disregarded. In reality, it's a very big problem for companies to disappoint their large customers (as NASA certainly was for Morton-Thiokol at the time), and I suspect that the engineers would have been quite annoyed at management if the ultimate result of a launch no-go was fewer contracts, lower pay and/or job losses. This isn't to say that management did the right thing, of course, just that "lol management just throws some buzzwords around and never attempts to understand the problem" is basically the same attitude that caused this disaster - just seen from an engineering viewpoint.
  - madaxe_again 11y ago
    I am "management" where I work, and I'd drive my company into the ground and make everyone unemployed, before I allowed my defective product to kill someone.
    - sebastos 11y ago
      Duh. You think a single person at NASA would have reported otherwise had you asked them before the Challenger disaster? Social pressure can influence people into engaging in wishful thinking. They made a horrible judgment call, and we should remember that that tragedy was authored by them. But if you think you NEVER would have fallen for it... you're not being self aware. There's a pretty decent chance you would have.
jrcii 11y ago
Can you imagine this poor man's reaction as he watched the launch?
draw_down 11y ago
I think many of us here have been Boisjoly at some point, though with much lower stakes. This is how people are, they want to do the thing. Doesn't matter how much you warn them.
metaprinter 11y ago
National Geographic has an awesome video called "Challenger, The Untold Story" which highlights the communications breakdowns. google it
steven777400 11y ago
Nothing I've worked on has been nearly to this scale, but having management understand and ignore warnings is commonplace. Just today, I pointed out that a backend web service system for an important (HR) function was so insecure that simply putting a single quote in an end-user input field would crash it, and that a SQL injection would be trivial. After a little arguing from them ("You don't know what a SQL injection is") they changed their tune to "Just put a limit on the frontend", and when I told them that could be trivially bypassed, they said "No one will ever think to try that." When I pressed further, they continued to "It's an internal app, no one would attack it" then to "And do you want me to put locks on all the doors and cabinets around the office too?" So that's a case where there's some reasonable level of understanding and just a refusal to act. I also recall a time (at a different company) when I discovered a public read/write anonymous FTP server setup externally facing with no firewall rules. When I pointed out the problem, they said, "that's what our customers are used to, we're not going to change it." and when I pushed, "No one will ever find it anyways." Literally less than a month later that server was overrun with illicit content. It happens. Unfortunately, as we've seen with the Volkswagen debacle as well, it happens even when regulation or (in the Challenger case) human life is on the line.
- chapium 11y ago
  Honestly, if its concerning personal data held by hr I would expect locks on the cabinets. At most places you cannot sneeze in HR without running into security.
- heywire 11y ago
  They'll pay attention when someone emails them their own salary information. I'm kidding, that would be career suicide. Printing it out and leaving it on the printer would be much better.
  - jayhuang 11y ago
    That's of course, assuming that there isn't a record of who is printing what. Which would probably be a dangerous assumption.
phkahler 11y ago
Isn't something missing? I was under the impression that the accident happened due to the cold. Nowhere in the memo does it say anything about how to avoid the problem, nor does it call for a full stop of launches until its fixed. It seems to be referring to a known issue without explaining what that is or what to do - specifically. Or did I miss that?
- JshWright 11y ago
  Simplifying a bit... They knew the o-rings were being eroded away during the flight, but the secondary o-ring was 'squishy' enough to fill in the gap and prevent the erosion from actually destroying the vehicle. While the erosion was unexpected, they figured the 'backup' was doing its job, and actually ended up increasing the predicted safety margin (i.e. it's only working 1/5th of the way through, so we have a 500% safety margin, yay! despite the fact that any erosion was unexpected in first place) The problem was, the cold made the secondary o-ring stiff enough that it didn't 'squish' as much as it had in previous launches, so the o-ring failed completely.
Houshalter 11y ago
I'm not saying NASA didn't fuck up by any means. I'd just like to point out that hindsight bias is a very real effect that can have huge consequences when planning for future events: http://lesswrong.com/lw/il/hindsight_bias/ http://lesswrong.com/lw/il/hindsight_bias/ >Viewing history through the lens of hindsight, we vastly underestimate the cost of effective safety precautions. In 1986, the Challenger exploded for reasons traced to an O-ring losing flexibility at low temperature. There were warning signs of a problem with the O-rings. But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight. It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings. >Shortly after September 11th 2001, I thought to myself, and now someone will turn up minor intelligence warnings of something-or-other, and then the hindsight will begin. Yes, I'm sure they had some minor warnings of an al Qaeda plot, but they probably also had minor warnings of mafia activity, nuclear material for sale, and an invasion from Mars. >Because we don't see the cost of a general policy, we learn overly specific lessons. After September 11th, the FAA prohibited box-cutters on airplanes—as if the problem had been the failure to take this particular "obvious" precaution. We don't learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.
- yosefk 11y ago
  The letter demonstrates foresight, not hindsight; a very long list of equally severe and plausible warnings is needed to make the point that one is unlikely to single out this one letter without the benefit of hindsight.
  - jsprogrammer 11y ago
    Yes, there is a serious gap of logic in the linked article. In fact, it appears that if the O-Rings were better checked and improved, as the letter indicated they must be, the Challenger would not have exploded as it did.
  - banku_brougham 11y ago
thoughtsimple 11y ago
I worked in a company that made CNC machine tools at the time of the Challenger disaster. The actions of Roger Boisjoly gave me the courage to reject a demand from management that I add a "feature" to a control system I was working on to disable all safety lockouts from a single virtual switch. I refused the assignment. When they didn't fire me on the spot I followed the managers who made the request and when they would ask another engineer to do it, I would warn the engineer that they could be held personally responsible for any injuries or fatalities that could occur. I fully expected to be fired but surprisingly, management just gave up on the idea. Perhaps my insistence that it was a bad idea convinced them.
- JustSomeNobody 11y ago
  Perhaps they consulted with HR/Legal and were told firing you would be a bad idea.
  - meric 11y ago
    Or Legal told them it was bad idea to ignore safety advice from their engineers?
- pmarreck 11y ago
  Good for you, man, for having the balls to stick up for your principles!
sixdimensional 11y ago
For those interested, social scientists often refer to what happened with the Challenger as a "normal accident". https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
mmaunder 11y ago
My parents in law worked with Greg Jarvis at Hughes who was on that mission. They had a touching memorial at Hughes for his wife according to my mom in law. Very sad that it was preventable.
ergothus 11y ago
It is perhaps sad, but I'm heartened by this letter. I had heard that warnings were given in advance, but this is clear (in fact, I find the "catastrophe of the highest order" language to be less impressive than the clear and defined "loss of human life"). Why am I heartened to see that someone foresaw a lethal accident and was ignored? Because it was foreseen, and the consequences understood, with clarity. Getting people to take clear warnings seriously seems a more easily solved problem than getting us to be able to recognize the danger in the first place. A single person in management, or a few people, being dense is fixable. Groupthink where everyone assumes someone else is checking for problems and no one does is harder to fix. Now, just because it's an easier problem doesn't make it an EASY problem, but still, easier.
lutusp 11y ago
A classic tale of engineering gone wrong. Apropos, I designed high-efficiency 20 KHz inverters for the Space Shuttle crew quarters, flight hardware that ended up working flawlessly. But during the design phase, the primary Shuttle contractor revealed that my inverters would be exposed to much higher voltages than they had originally supposed. When I saw this, I realized we would have to start over and choose different components able to tolerate the higher voltages. My managers disagreed, arguing that if we caused problems, we would be denied a follow-on contract for similar inverters to be deployed in the payload bay. The managers composed a reply to NASA in which there was no problem, things were fine, let's forge ahead. I was a mere engineer, I had no management authority, and I hadn't been consulted about the reply. When I heard about it, I sat down and wrote a letter of resignation and pushed my letter into more hands than was absolutely necessary. I made some comparisons that at the time might have seemed over the top (like the Apollo file that killed three astronauts, a disaster resulting from lax oversight). In my case, because of my having distributed the letter farther than was absolutely necessary, my managers were forced to reverse themselves, I was able to redesign my inverters in a safe way, and we got the follow-on contract in spite of not being seen as "team players". Many years later, at the time of the Challenger disaster, it finally dawned on me that, had I disregarded the overvoltage issue as my managers had wanted me to, and if something had gone wrong, I would have been held personally responsible, because I was the only person with the level of technical knowledge required to make the call, and my managers could disavow any responsibility. At the time, I made the right decision, but for reasons that I hadn't fully thought out -- if my equipment had failed in-flight, I would have been held responsible, and that would have been a perfectly just outcome.
jldugger 11y ago
Engineering degrees universally require a technical writing course. Engineers universally revile it as busy work. This memo is Exhibit A for why written communication is critical for engineering, and I imagine an assignment to rewrite this memo to be more effective would be a good way to underscore the importance of clear and concise writing.
- brlewis 11y ago
  I upvoted this comment because the point about the importance of writing. However, I don't think conciseness and clarity was the problem here. In 1985 if an engineer personally composed a long letter you should automatically know it's important. If you read this letter at all the danger is made clear.
  - banku_brougham 11y ago
    The call to action was on the second page though - definitely try to keep it to one page in corporate communications class.
- gradi3nt 11y ago
  What would us change about the letter? Commenter djcapelis above disagrees.
  - jldugger 11y ago
    First fix the grammar mistake ("insure"). Then put the Bottom Line Up Front. Bold it if possible. Cut out the fluff like 'this letter is written' and 'a jump ball as to the success or failure.' Rewrite the second paragraph with some verbs and a proper subject. Not that any of this really matters in the end. The final decision was fully in the hands of management and NASA during a teleconference, during which argument was made.
bluedino 11y ago
Imagine being on the team that discovered and warned about this issue, and then a year later watching the replays of the shuttle exploding on television. Just knowing that's what had happened and your worst fears at come true.
jsprogrammer 11y ago
The transcript is incorrect. It uses "Manager" under Kapp's signature, instead of the original "Manger".
thescriptkiddie 11y ago
Here's an interesting thing about the Challenger (and Columbia) disaster: We find it to be particularly devastating, the same as a major natural disaster or terrorist attack. But on paper, they don't even compare. The explosion of the Challenger killed 7 people and cost NASA around $40 billion. The destruction of the world trade center killed over 2600 and the cost to the private insurers alone was over $40 billion. Hurricane Katrina killed over 1800 and cost at least $100 billion. Not even the same ballpark. So why the big emotional impact? Because they stood for something important.
mirceal 11y ago
In hindsight things are obvious. I think the way to look at it is from the point of view that people have before the event happens. You also weigh in all the warnings that you are receiving. No matter what you build, if it's complex enough you're always going to have individual predicting doom. The challenging part is filtering the signal from the noise and owning the decisions you make.
Pitarou 11y ago
This letter is an excellent illustration of why good writing matters, and why bad writing can be disastrous. The most important sentence in the letter ("The result would be a catastrophe of the highest order - loss of human life.") is at the end of the third paragraph, and is effectively hidden by two paragraphs of dense, technical jargon that I, as a layman, cannot understand at all. I honestly think that if he had just taken that sentence and moved it to the end of the first paragraph, 7 astronauts' lives could have been saved.
- banku_brougham 11y ago
  yeah - management only reads the first sentence and first bullet point in my experience. sick.
- djcapelis 11y ago
  I strongly doubt it and I don't think you've worked in an engineering environment if you think this is true. This document is written as a credible engineering analysis of a distinct problem and uses extremely strong terms. It isn't addressed to the general public, or an article on Buzzfeed, it is an interdepartmental memo from an engineer signed by his manager to the Vice President of Engineering. Your example of "clear writing" would have just made this engineer look hyperbolic and likely undermined the issue even more. In a document like this, stating the physical problem up front is clear writing, and then, once the engineering problem is stated, he immediately states the possible outcome and then describes what he says as the management failure to allocate the appropriate resources, and how to fix it. How it comes off to you as a layman doesn't dictate whether or not this is good writing. He wasn't writing it to you.
  - Pitarou 11y ago
    I respectfully disagree. Safety is of the highest priority, so if there's a serious risk to human life, there's nothing hyperbolic about drawing attention to it. If you can't do that, there's something very wrong with your engineering culture. To help the busy reader, the first paragraph of a memo should summarize the whole document. (Like the abstract of a technical paper.) This document is about an engineering problem and it's serious consequences, so they should both be mentioned in the first paragraph.
sopooneo 11y ago
I bet there were other such warnings, many of them valid, that just didn't turn out to be catastrophic. If there are any high level managers here, please tell me, does it seem like your people are constantly alerting you of dire risks? Does it feel like you are inundated with worriers?
js2 11y ago
An piece NPR did on Boisjoly after he passed away: http://www.npr.org/sections/thetwo-way/2012/02/06/146490064/remembering-roger-boisjoly-he-tried-to-stop-shuttle-challenger-launch http://www.npr.org/sections/thetwo-way/2012/02/06/146490064/... And an interview with one of Boisjoly's colleauges: http://www.npr.org/sections/thetwo-way/2016/01/28/464744781/30-years-after-disaster-challenger-engineer-still-blames-himself http://www.npr.org/sections/thetwo-way/2016/01/28/464744781/...
hackuser 11y ago
What stands out to me the impact of loyalty on corruption. Despite doing obviously the right thing: Boisjoly later revealed this memo to the presidential commission investigating the disaster and was then forced to leave Morton Thioklol after been shunned by disgruntled colleagues. Elsewhere in the discussion[1], we learn that key information about the O-rings had to be carefully, anonymously disclosed to protect the jobs of several people, including a prominent astronaut. How do we avoid corruption when loyalty uber alles is the rule of almost all organizations? [1] https://news.ycombinator.com/item?id=10992224 https://news.ycombinator.com/item?id=10992224
- vlehto 11y ago
  For small enough organization that "loyalty uber alles" prevents corruption. What is small enough? For a military company it's around Dunbar number. Squad that actually works together on single issue is often optimal around 8 individuals. Company, squad and brigade are most important formations, as they have dis-proportionally high number of things expected to handle independently. For this NASA o-ring bullshit, simplest solution seems to be having the technical managers and the astronauts in the same in-group. Now your loyalty is about not getting your mates killed. If you need a big organization, the trick is to have that internal network of squads and companies to work somehow non-corrupt and nice manner. This is where it gets hairy. Basically you can go with assumption of corruption and apply "transparency" or monetary incentives (sub contractors). Or you can assume that corruption does not happen. Sometimes the mere assumption that corruption does not happen stifles it. Practically for big companies failing less than your competitors is adequate for success.
  - hackuser 11y ago
    Some interesting ideas; thanks. I don't quite grasp a couple of them: > For small enough organization that "loyalty uber alles" prevents corruption. How? There are many, many cases of people covering up for their fellow squad members. > Or you can assume that corruption does not happen. Sometimes the mere assumption that corruption does not happen stifles it. Hmmm ... that seems like a well-known recipe for encouraging corruption. How would that approach stifle it?

381 points by moviuro 11y ago

I remember the Challenger explosion like it was yesterday. I'll link here to a scan of the pages from Edward Tufte's book Visual Explanations about what went wrong with the data analysis during the launch planning, which understated the risk of a launch in cold weather. http://williamwolff.org/wp-content/uploads/2013/01/tufte-challenger-1997.pdf http://williamwolff.org/wp-content/uploads/2013/01/tufte-cha...

nehushtan 11y ago
But Boisjoly rejected Tufte's analysis http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i...
- pmahoney 11y ago
  Can you summarize one or two of Boisjoly's main points? Tufte's pictures of the (superfluous) rocket graphics showing the temperature vs. his redrawing, placing temperature on the x-axis are very convincing.
  - jasode 11y ago
    Boisjoly's rebuttal is that 1) Tufte had temperature datapoints for the previous launches that the Boisjoly's team didn't know[1]. I assume "know" to mean the complete historical data wasn't all consolidated conveniently at his fingertips before the disaster. Presumably, Boisjoly's team could have gotten it if it had occurred to them to gather it. Essentially, Tufte had the benefit of "hindsight" to make his compelling diagram showing cause & effect. 2) GIGO garbage in is garbage out. The Tufte temperature datapoint assumes that the outside ambient temperature is equal to the O-ring temperature[2] so substituting one for the other is wrong. (E.g. It's wrong to put them both on the same X-axis.) Quotes excerpted from http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i...: [1]"He thus supposes that they knew the temperatures at launch of all the shuttles and, assuming they acted voluntarily, infers they were incompetent." [2]", in addition, mixes O-ring temperatures and ambient air temperature as though the two were the same."
    - m741 11y ago
      I thought the rebuttal was very well written, if a bit dense. For (1), you're correct that the engineers did not have the historical data. More than that, though, it did occur to them to request the data, but they were stifled by Morton Thiokol Management (and NASA). To start with, there were a variety of previous problems with the O-rings caused by variables that appear unrelated to temperature. These problems were resolved, but prevented anyone from seeing a pattern. It was only on the basis of the single data point in SRM 15 that Boisjoly requested temperature data in advance of the launch. Obtaining such data was far from simple, because, as you mention, the temperature of the O-ring isn't the same as the ambient air temperature. Thus, obtaining the data was relatively involved and required knowing many variables: time on the pad, the gradient of ambient temperature, the temperature at which testing was conducted, and so forth. For this reason, the engineers didn't compile the data themselves (unclear what process they'd need to get the data). The engineers thus requested the data in advance, but had not received it. They had precise data on only two data points (at 53 degrees and 75 degrees), so the rest of the data in the chart was compiled after the fact. "The data necessary for a calculation of O-ring temperatures was thus not collected all along during the shuttle history. And when Boisjoly asked for that data in September, along with much other data, any one of which might have been the crucial missing piece to explain the anomalous cause, it was not supplied. In fact, the engineers received none of the data they requested."
  - m741 11y ago
    I'm no engineer, but I'll take a crack at it: * Both axes of the chart are inaccurate. The temperature on the X axis intermingles ambient and O-ring temperature, but they're not the same (imagine ocean temperature vs air temperature). The vertical axis, O-ring damage, is semi-relevant, but the important question is whether the O-rings held a seal (degree of blow-by). O-ring damage probably correlates, but seems like a proxy that may be misleading. * The data available in the chart was not available to the engineers at the time. There was only one previous failure (SRM 15) that led anyone to believe temperature itself was an issue. They had two valid temperature data points (SRM 15 and SRM 22), and correctly pointed out that another launch at 29 degrees was completely outside their tested range and would be inherently risky. It's not clear to me that a scatterplot with two datapoints would be that effective. Furthermore Boisjoly criticizes that Tufte presumed to judge the engineers, for the following reasons: * The engineers previously recommended no launch should occur, months in advance (due to previous O-ring issues), but NASA overruled them. * The engineers recommended, and their managers accepted, that the low launch temperature was outside their test database. NASA came back and requested proof that the temperature was dangerous, but the engineers could not comply, because the parameters were extreme enough that they had not tested them. They couldn't prove a negative. The managers at Morton Thiokol and NASA then jointly overruled the engineers. * The idea of putting together a chart was not something the engineers considered, because there were a variety of previous unrelated problems in O-rings (each resolved afterwards) that muddied that data, and muddies the data that Tufte presents. * Tufte himself failed to research or note the pivotal information, and thus misrepresented the situation the engineers were in, and the data available, while imputing that the engineers should be held morally responsible for not presenting data they didn't have (and then going on to present such data himself, compiled after the fact and at his own leisure, incorrectly).
- ricardobeat 11y ago
  That reads like a knee-jerk reaction to the (implied) accusation that the engineers were at fault for not presenting their case correctly. It doesn't dismiss Tufte's arguments for clear visual presentation of data; you could say that both of them are right.
MCRed 11y ago
I walked in to the TV showing the launch. I hadn't known it had exploded. I didn't realize I was seeing a replay. I spent my time as a kid living on Vandenberg Air Force base, and my father would drag me out to Minute Men missile launches early in the mornings. I saw several launches. Even two launched at once. I had seen many launches before, so I had no expectation that they could go wrong. Watching the shuttle explode like that was shocking.
- ascagnel_ 11y ago
  > I had seen many launches before, so I had no expectation that they could go wrong. This mindset is interesting. Issues like plane and train crashes grab headlines and are typically catastrophic, but end up being far more rare than, say, a fatal car crash. Similarly, the Space Shuttle had two major incidents in 30 years, and the Concord had only one, but they were very public and very catastrophic. Is it the rarity or severity that make them such big news?
  - mikeash 11y ago
    Rarity is certainly a big factor even from an objective viewpoint. Concorde instantaneously went from the safest airliner flying to the most dangerous with that crash, just because it didn't fly much. Flying on a Space Shuttle was one of the most dangerous activities out there, judging by the historical failure rate. It helps just in terms of grabbing your attention, too. "Man drives into lake" doesn't make the news. "Man drives Bugatti Veyron into lake" does, just because the unusual car makes it more interesting. In the case of Concorde and the Shuttle, the machines in question were not only rare, but beloved by many, and seen as a national symbol. It's almost like assassinating the country's leader.
jqm 11y ago
I saw the Challenger explosion live. I was a sophmore in High School in Florida and standing outside at lunch. Shuttle launches were a fairly regular thing and I'd seem many. But this time I could see the trails as parts fell away to earth. It wasn't like the fuel tanks the fell away usually. I was talking with a guy named Wes. He said "that doesn't look right". It didn't. After lunch there was announcement that the shuttle had exploded. It doesn't seem that long ago really. The weird thing is.. in that same amount of time in the future I'm going to be a 75 year old man:)
- JustSomeNobody 11y ago
  Same. I was home sick and, as usual for a launch, went out to see it. Saw the split smoke plumes and knew it was wrong, having seen enough launches.
- ck2 11y ago
  I still remember the words the announcer said just before it happened, it was that deeply etched into my brain. go for throttle up and whenever the news shows that Y trail plume it still makes me ill to this day

samwiseg 11y ago

As someone born in 1995, I don't have any real emotional connection to the Challenger explosion. It's pretty bizarre to think that children born after the 9/11 attacks will probably feel the same indifference.

mturmon 11y ago
If you are an engineer, it would be good to try to learn from this and similar accidents. I.e., noting your lack of emotional connection is beside the point, the question is, what can you take away from what happened?
- samwiseg 11y ago
  Of course, it's extremely important. I'm currently listening to the Freakonomics podcast on the Challenger explosion[0]. I was only noting the difference being born a few years apart can make when it comes to significant shared cultural memories. [0]http://freakonomics.com/2015/05/20/failure-is-your-friend-a-freakonomics-radio-rebroadcast/ http://freakonomics.com/2015/05/20/failure-is-your-friend-a-...
  - btilly 11y ago
    You will notice this sort of thing more and more as time goes on. For me the "ah hah" moment was when it hit me that "Where were you when you heard about Challenger?" was my generation's version of, "Where were you when JFK was assassinated?" Another easily observed one is music. Very few people are aware of much that happened in popular music after they hit 25.
    - gradi3nt 11y ago
      I'm turning 25 in two weeks. So this is it? 2015 was my last year of music? Damn...it went by so fast!
      - vlehto 11y ago

shams93 11y ago

I doubt anyone in the reagan admin could even understand this note its highly technical it was likely tossed out without ever being read.

crystalmeph 11y ago
This memo was not sent to the administration, it was sent to the management of Morton Thiokol, who could reasonably be expected to understand basic engineering phrases like "loss of human life." It's a real PITA to engineer safe systems (see e.g. IEC 61508 and ISO 13849 in industrial automation), but it saves lives, and if you're in a position where Sales, Production, etc. is trying to get you to rush the Engineering work so they can make deadline, you've got to find the backbone to say "no," even if it hurts you professionally.
- madaxe_again 11y ago
  You'd think so, but they probably saw "loss of human life" and went "hysterical. ignore.". It's actually really hard to warn people. Warn them loudly and strongly and they think you're scaremongering and being unnecessarily negative. Warn them quietly, and nobody listens. Warn them just right, and they'll take it under consideration, but go ahead anyway. When of course the inevitable happens, as the guy who predicted doom, you'll be blamed, because "it wouldn't have happened if you hadn't made a self fulfilling prophecy".
stevegoober 11y ago
> I doubt anyone in the reagan admin could even understand this note Right... and maybe this is a problem with the authors of the note rather than the recipients? Know how to communicate to your target audience.
- URSpider94 11y ago
  That's why he sent it to the VP of Engineering, who I'm sure understood everything in the letter.

DickingAround 11y ago

Today, there's no way I could get away with writing an entire page of context before getting to the impact. I wonder how much context the recipient had and what reading expectations were like back then...

wylee 11y ago
I would think that when someone writes a document like this that the context is really important for the record and that they're not so concerned with making an "impact". Edit: Grammar
dtparr 11y ago
Leaving aside the fact that the subject includes 'Potential Failure Criticality', the key impact line "The result would be a catastrophe of the highest order - loss of human life" occurs less than 150 words into this. It's the 5th sentence (though they're admittedly long sentences). Are you having serious engineering-related discussions with your management where they can't read 150 words before getting to the hook? (serious question, not snark) Is everyone communicating via twitter or something? (sorry, that was a bit of snark)
- frandroid 11y ago
  They're really "thick" sentences. By the time you get there, your brain has had to process so much that you go "huh".
  - dtparr 11y ago
    Interesting. I'm curious, do you find the technical content to be the 'thickening agent' or the style? I work in a related field, and am vaguely familiar with both the general makeup of this type of rocket and this accident in general. I also tend to write longer, more complex sentences. I did not find it to be particularly 'thick', and I'm curious whether it's because I'm more familiar with the technical details and so avoided the glossy-eyed stare from that aspect or from the fact that a few dozen words in a sentence doesn't faze me. The former would mean this shouldn't bother anyone practiced in the arts of spaceflight (such as the VP of engineering), but the latter would have been a problem. Edit: stupid homophones.

viach 11y ago

If you don't stop the CO2 emission, the result would be a catastrophe. Lets see how it will turn out now. Edit: And yes, it is sarcasm, nothing changed or is going to change.

DanielBMarkham 11y ago

This was a terrible thing. So many saw this coming and nobody listened to them. In my mind, what made this tragedy even worse was the way the program itself was conducted. You learn new modes of transportation and hardware by using it, many times to the point of exhaustion. We should have built a dozen orbiters and flown the shit out of them through hell or high water, learning as we went along. Instead we built 5 and every time something happened we backed farther away from the entire manned space program. A lot of time was wasted, and the lessons we didn't learn? Somebody else is still going to have to learn them someday.

mikeash 11y ago
I think it's emblematic of the whole Shuttle program to look at the flight test program. There basically wasn't one. In particular, look at the various abort scenarios, and then look at the abort testing. Don't look too hard for the abort testing, because they didn't do any. There was pretty much zero resiliency in the system, and they knew it.

throwaway2380 11y ago

Boisjoly later revealed this memo to the presidential commission investigating the disaster and was then forced to leave Morton Thioklol after been shunned by disgruntled colleagues. I find this phenomenon so difficult to understand.

csense 11y ago
You just found out your company screwed up big, and it turned into people dying and a bunch of geopolitical egg on your country's face, but if everybody keeps quiet, then maybe you'll get lucky and nobody will figure out it was your fault. But one guy tells the world exactly how you screwed up. Now you're in fear of your reputation and job -- but if he'd just kept his mouth shut, then you might be okay.
poof131 11y ago
Seems fairly standard. He provided feedback that could have prevented a disaster and the managers ignored it. So at the very least, the managers would dislike him for the fact that he illuminated their incompetence, but even more so for the fact that he would be a reminder of the massive tragedy that ensued and for which they were partially to blame. With management's resentment, his peers would have a choice, maintain a friendly relationship with him and risk their careers and management’s wrath, or distance themselves from him. It’s sad, but the natural tendency of organizations is to close ranks in situations like this. It is incredibly tough to perform an open, honest, self-assessment. The procedures need to be in place to force the self-examination and strong leaders need to push for it. Even then, politics sometimes place the blame on the wrong people. I’ve seen this happen time and time again in the military from working with SF in Iraq [1] to joining a squadron just after a disaster [2]. [1] http://www.nytimes.com/2008/08/20/world/africa/20iht-20iraq.15446434.html http://www.nytimes.com/2008/08/20/world/africa/20iht-20iraq.... [2] https://en.wikipedia.org/wiki/2008_San_Diego_F/A-18_crash https://en.wikipedia.org/wiki/2008_San_Diego_F/A-18_crash
camperman 11y ago
You need to read a book called Systemantics by John Gall. It's a hilarious and accurate look at how large systems fail. Although parts are somewhat dated, the principles are the same. In particular his material on bureaucracies is relevant: when threatened, bureaucracies immediately fight for self-preservation. Boisjoly was threatening the bureaucracy and so had to be eliminated for it to survive. Its actions makes perfect sense in the light of this. Ugly, yes but understandable.

maho 11y ago

It seems almost unbelievable how such a stern, concrete warning can be ignored. Of course, we look at this memo with perfect 20/20 hindsight, knowing that the O-ring issue caused a disaster. I am genuinely curious why the warning was ignored. I am hesistant to believe it was through malice or sheer incompetence. Does upper managment get two overblown warnings per week by engineering, so that this critical warning was "drowned by the noise"? Or were there good (at the time!) reasons to focus on other issues first?

tahoeskibum 11y ago
That's a good point. It is easy to just say that the others were incompetent without knowing the rate of false positives ie. how frequently were such memos written talking about catastrophes. The issue of rate of false positives vs true positives (i.e.,real problem) is something that most engineers and managers don't have a good grasp of.
- LoSboccacc 11y ago
  this is true for this early warning from the manufacturing company, but the launch day reasoning was chilling. I'll be back with links to the worse parts. https://news.ycombinator.com/item?id=9277408 https://news.ycombinator.com/item?id=9277408
cmurf 11y ago
I could see managers at NASA seeing their role as a union or labor leader, trying to keep their budget and their people employed at least as much a priority as safety. So they're willing to take more risks because they're in a position of having to balance an equation. And it may be that the flaw is this uni-dimensional hierarchy that we seem to like, that seem efficient most of the time, but in reality not only miss edge cases but can cause them. The managers simply did not always 100% trust the engineers. Ultimately, the people with the power over go/no-go were more managerial than engineers, were more concerned with the politics, cost, perception of delay than they were at risk assessment. All spending of money in any company or government is basically the same: someone at the top delegates the vast majority to someone else. And they choose that someone else based on trust. And the longer than chain is, the greater the likelihood someone is put in position where the chain of trust is just flawed. And you hope it gets exposed where people don't die, obviously. But that didn't happen here. And that's why a bunch of people got angry at Feynman as well as Boisjoly. They exposed that the wrong people were making key decisions. Had they been the right people, then loss of life wouldn't have happened. Anyone that wrong is going to be angry, it's a personality flip of the coin whether they get angry with themselves for their mistake or with others for exposing it.

sgentle 11y ago

If you haven't read Richard Feynman's appendix to the Rogers Commission report on the Challenger disaster, I really recommend it: http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger... It's very readable and goes into a bunch of other issues at NASA at the time beyond the O-ring failure, which Feynman treats as a symptom of more general engineering and cultural problems.

danso 11y ago
Feynman is well-known for his TV appearance where he dramatically demonstrated what happened with the O-rings, live...but in his memoir, "What Do You Care What Other People Think?", he credits being inspired by General Kutyna, who had asked him to join the commission. Feynman describes that a random comment from Kutyna led him down the path of revelation: > Then he says, “I was working on my carburetor this morning, and I was thinking: the shuttle took off when the temperature was 28 or 29 degrees. The coldest temperature previous to that was 53 degrees. You’re a professor; what, sir, is the effect of cold on the O-rings?” “Oh!” I said. “It makes them stiff. Yes, of course!” That’s all he had to tell me. It was a clue for which I got a lot of credit later, but it was his observation. A professor of theoretical physics always has to be told what to look for. He just uses his knowledge to explain the observations of the experimenters! However, Feynman admits at the end of his multi-chapter account of serving on the commission that Kutyna had coyly played Feynman: > Another thing I understand better now has to do with where the idea came from that cold affects the O-rings. It was General Kutyna who called me up and said, “I was working on my carburetor, and I was thinking: what is the effect of cold on the O-rings?” Well, it turns out that one of NASA’s own astronauts told him there was information, somewhere in the works of NASA, that the O-rings had no resilience whatever at low temperatures—and NASA wasn’t saying anything about it. But General Kutyna had the career of that astronaut to worry about, so the real question the General was thinking about while he was working on his carburetor was, “How can I get this information out without jeopardizing my astronaut friend?” His solution was to get the professor excited about it, and his plan worked perfectly. If we take Feynman at his word, to me this is a great description of how science and bureaucracy and invention (and, sometimes, disaster) works: it's not always just random eurekas, or study-real-hard-and-you'll-figure-it-out...sometimes the information is already well-known, and obvious, but for various logistical/political/bureaucratic reasons, it isn't immediately disseminated or explored. via: Feynman, Richard P. (2011-02-14). "What Do You Care What Other People Think?": Further Adventures of a Curious Character (Kindle Locations 2923-2930). W. W. Norton & Company. Kindle Edition.

stepanhruda 11y ago

Here is Richard Feynman talking about the issues https://www.youtube.com/watch?v=4kpDg7MjHps https://www.youtube.com/watch?v=4kpDg7MjHps Seems not very different from how companies are managed 30 years later.

madaxe_again 11y ago

"Ach, damned engineers, what do they know about engineering!?" ~ Management, upon reading the memo. Actually, more realistically: "Brinton, do you know what this Boisjoly guy is talking about?" "No Lund, I don't - his attitude is disappointing, we need team players." "Thanks Brinton - I'll bring it up with his manager for his next performance review."

krschultz 11y ago
It's easy to blame it on 'management', but it is important to note that author of the memo is himself a 'manager' at the time of writing the memo. There is way more depth to this situation than the simple hand waiving 'everyone that supervises me is stupid'.
sjm-lbm 11y ago
To be honest, I think you are - to an extent - falling into the same trap that the managers fell into at Thiokol: namely, assuming that your view of the situation is complete and that input from other work groups can be disregarded. In reality, it's a very big problem for companies to disappoint their large customers (as NASA certainly was for Morton-Thiokol at the time), and I suspect that the engineers would have been quite annoyed at management if the ultimate result of a launch no-go was fewer contracts, lower pay and/or job losses. This isn't to say that management did the right thing, of course, just that "lol management just throws some buzzwords around and never attempts to understand the problem" is basically the same attitude that caused this disaster - just seen from an engineering viewpoint.
- madaxe_again 11y ago
  I am "management" where I work, and I'd drive my company into the ground and make everyone unemployed, before I allowed my defective product to kill someone.
  - sebastos 11y ago
    Duh. You think a single person at NASA would have reported otherwise had you asked them before the Challenger disaster? Social pressure can influence people into engaging in wishful thinking. They made a horrible judgment call, and we should remember that that tragedy was authored by them. But if you think you NEVER would have fallen for it... you're not being self aware. There's a pretty decent chance you would have.

jrcii 11y ago

Can you imagine this poor man's reaction as he watched the launch?

draw_down 11y ago

I think many of us here have been Boisjoly at some point, though with much lower stakes. This is how people are, they want to do the thing. Doesn't matter how much you warn them.

metaprinter 11y ago

National Geographic has an awesome video called "Challenger, The Untold Story" which highlights the communications breakdowns. google it

steven777400 11y ago

Nothing I've worked on has been nearly to this scale, but having management understand and ignore warnings is commonplace. Just today, I pointed out that a backend web service system for an important (HR) function was so insecure that simply putting a single quote in an end-user input field would crash it, and that a SQL injection would be trivial. After a little arguing from them ("You don't know what a SQL injection is") they changed their tune to "Just put a limit on the frontend", and when I told them that could be trivially bypassed, they said "No one will ever think to try that." When I pressed further, they continued to "It's an internal app, no one would attack it" then to "And do you want me to put locks on all the doors and cabinets around the office too?" So that's a case where there's some reasonable level of understanding and just a refusal to act. I also recall a time (at a different company) when I discovered a public read/write anonymous FTP server setup externally facing with no firewall rules. When I pointed out the problem, they said, "that's what our customers are used to, we're not going to change it." and when I pushed, "No one will ever find it anyways." Literally less than a month later that server was overrun with illicit content. It happens. Unfortunately, as we've seen with the Volkswagen debacle as well, it happens even when regulation or (in the Challenger case) human life is on the line.

chapium 11y ago
Honestly, if its concerning personal data held by hr I would expect locks on the cabinets. At most places you cannot sneeze in HR without running into security.
heywire 11y ago
They'll pay attention when someone emails them their own salary information. I'm kidding, that would be career suicide. Printing it out and leaving it on the printer would be much better.
- jayhuang 11y ago
  That's of course, assuming that there isn't a record of who is printing what. Which would probably be a dangerous assumption.

phkahler 11y ago

Isn't something missing? I was under the impression that the accident happened due to the cold. Nowhere in the memo does it say anything about how to avoid the problem, nor does it call for a full stop of launches until its fixed. It seems to be referring to a known issue without explaining what that is or what to do - specifically. Or did I miss that?

JshWright 11y ago
Simplifying a bit... They knew the o-rings were being eroded away during the flight, but the secondary o-ring was 'squishy' enough to fill in the gap and prevent the erosion from actually destroying the vehicle. While the erosion was unexpected, they figured the 'backup' was doing its job, and actually ended up increasing the predicted safety margin (i.e. it's only working 1/5th of the way through, so we have a 500% safety margin, yay! despite the fact that any erosion was unexpected in first place) The problem was, the cold made the secondary o-ring stiff enough that it didn't 'squish' as much as it had in previous launches, so the o-ring failed completely.

Houshalter 11y ago

I'm not saying NASA didn't fuck up by any means. I'd just like to point out that hindsight bias is a very real effect that can have huge consequences when planning for future events: http://lesswrong.com/lw/il/hindsight_bias/ http://lesswrong.com/lw/il/hindsight_bias/ >Viewing history through the lens of hindsight, we vastly underestimate the cost of effective safety precautions. In 1986, the Challenger exploded for reasons traced to an O-ring losing flexibility at low temperature. There were warning signs of a problem with the O-rings. But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight. It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings. >Shortly after September 11th 2001, I thought to myself, and now someone will turn up minor intelligence warnings of something-or-other, and then the hindsight will begin. Yes, I'm sure they had some minor warnings of an al Qaeda plot, but they probably also had minor warnings of mafia activity, nuclear material for sale, and an invasion from Mars. >Because we don't see the cost of a general policy, we learn overly specific lessons. After September 11th, the FAA prohibited box-cutters on airplanes—as if the problem had been the failure to take this particular "obvious" precaution. We don't learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.

yosefk 11y ago
The letter demonstrates foresight, not hindsight; a very long list of equally severe and plausible warnings is needed to make the point that one is unlikely to single out this one letter without the benefit of hindsight.
- jsprogrammer 11y ago
  Yes, there is a serious gap of logic in the linked article. In fact, it appears that if the O-Rings were better checked and improved, as the letter indicated they must be, the Challenger would not have exploded as it did.
- banku_brougham 11y ago

thoughtsimple 11y ago

I worked in a company that made CNC machine tools at the time of the Challenger disaster. The actions of Roger Boisjoly gave me the courage to reject a demand from management that I add a "feature" to a control system I was working on to disable all safety lockouts from a single virtual switch. I refused the assignment. When they didn't fire me on the spot I followed the managers who made the request and when they would ask another engineer to do it, I would warn the engineer that they could be held personally responsible for any injuries or fatalities that could occur. I fully expected to be fired but surprisingly, management just gave up on the idea. Perhaps my insistence that it was a bad idea convinced them.

JustSomeNobody 11y ago
Perhaps they consulted with HR/Legal and were told firing you would be a bad idea.
- meric 11y ago
  Or Legal told them it was bad idea to ignore safety advice from their engineers?
pmarreck 11y ago
Good for you, man, for having the balls to stick up for your principles!

sixdimensional 11y ago

For those interested, social scientists often refer to what happened with the Challenger as a "normal accident". https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents

mmaunder 11y ago

My parents in law worked with Greg Jarvis at Hughes who was on that mission. They had a touching memorial at Hughes for his wife according to my mom in law. Very sad that it was preventable.

ergothus 11y ago

It is perhaps sad, but I'm heartened by this letter. I had heard that warnings were given in advance, but this is clear (in fact, I find the "catastrophe of the highest order" language to be less impressive than the clear and defined "loss of human life"). Why am I heartened to see that someone foresaw a lethal accident and was ignored? Because it was foreseen, and the consequences understood, with clarity. Getting people to take clear warnings seriously seems a more easily solved problem than getting us to be able to recognize the danger in the first place. A single person in management, or a few people, being dense is fixable. Groupthink where everyone assumes someone else is checking for problems and no one does is harder to fix. Now, just because it's an easier problem doesn't make it an EASY problem, but still, easier.

lutusp 11y ago

A classic tale of engineering gone wrong. Apropos, I designed high-efficiency 20 KHz inverters for the Space Shuttle crew quarters, flight hardware that ended up working flawlessly. But during the design phase, the primary Shuttle contractor revealed that my inverters would be exposed to much higher voltages than they had originally supposed. When I saw this, I realized we would have to start over and choose different components able to tolerate the higher voltages. My managers disagreed, arguing that if we caused problems, we would be denied a follow-on contract for similar inverters to be deployed in the payload bay. The managers composed a reply to NASA in which there was no problem, things were fine, let's forge ahead. I was a mere engineer, I had no management authority, and I hadn't been consulted about the reply. When I heard about it, I sat down and wrote a letter of resignation and pushed my letter into more hands than was absolutely necessary. I made some comparisons that at the time might have seemed over the top (like the Apollo file that killed three astronauts, a disaster resulting from lax oversight). In my case, because of my having distributed the letter farther than was absolutely necessary, my managers were forced to reverse themselves, I was able to redesign my inverters in a safe way, and we got the follow-on contract in spite of not being seen as "team players". Many years later, at the time of the Challenger disaster, it finally dawned on me that, had I disregarded the overvoltage issue as my managers had wanted me to, and if something had gone wrong, I would have been held personally responsible, because I was the only person with the level of technical knowledge required to make the call, and my managers could disavow any responsibility. At the time, I made the right decision, but for reasons that I hadn't fully thought out -- if my equipment had failed in-flight, I would have been held responsible, and that would have been a perfectly just outcome.

jldugger 11y ago

Engineering degrees universally require a technical writing course. Engineers universally revile it as busy work. This memo is Exhibit A for why written communication is critical for engineering, and I imagine an assignment to rewrite this memo to be more effective would be a good way to underscore the importance of clear and concise writing.

brlewis 11y ago
I upvoted this comment because the point about the importance of writing. However, I don't think conciseness and clarity was the problem here. In 1985 if an engineer personally composed a long letter you should automatically know it's important. If you read this letter at all the danger is made clear.
- banku_brougham 11y ago
  The call to action was on the second page though - definitely try to keep it to one page in corporate communications class.
gradi3nt 11y ago
What would us change about the letter? Commenter djcapelis above disagrees.
- jldugger 11y ago
  First fix the grammar mistake ("insure"). Then put the Bottom Line Up Front. Bold it if possible. Cut out the fluff like 'this letter is written' and 'a jump ball as to the success or failure.' Rewrite the second paragraph with some verbs and a proper subject. Not that any of this really matters in the end. The final decision was fully in the hands of management and NASA during a teleconference, during which argument was made.

bluedino 11y ago

Imagine being on the team that discovered and warned about this issue, and then a year later watching the replays of the shuttle exploding on television. Just knowing that's what had happened and your worst fears at come true.

jsprogrammer 11y ago

The transcript is incorrect. It uses "Manager" under Kapp's signature, instead of the original "Manger".

thescriptkiddie 11y ago

Here's an interesting thing about the Challenger (and Columbia) disaster: We find it to be particularly devastating, the same as a major natural disaster or terrorist attack. But on paper, they don't even compare. The explosion of the Challenger killed 7 people and cost NASA around $40 billion. The destruction of the world trade center killed over 2600 and the cost to the private insurers alone was over $40 billion. Hurricane Katrina killed over 1800 and cost at least $100 billion. Not even the same ballpark. So why the big emotional impact? Because they stood for something important.

mirceal 11y ago

In hindsight things are obvious. I think the way to look at it is from the point of view that people have before the event happens. You also weigh in all the warnings that you are receiving. No matter what you build, if it's complex enough you're always going to have individual predicting doom. The challenging part is filtering the signal from the noise and owning the decisions you make.

Pitarou 11y ago

This letter is an excellent illustration of why good writing matters, and why bad writing can be disastrous. The most important sentence in the letter ("The result would be a catastrophe of the highest order - loss of human life.") is at the end of the third paragraph, and is effectively hidden by two paragraphs of dense, technical jargon that I, as a layman, cannot understand at all. I honestly think that if he had just taken that sentence and moved it to the end of the first paragraph, 7 astronauts' lives could have been saved.

banku_brougham 11y ago
yeah - management only reads the first sentence and first bullet point in my experience. sick.
djcapelis 11y ago
I strongly doubt it and I don't think you've worked in an engineering environment if you think this is true. This document is written as a credible engineering analysis of a distinct problem and uses extremely strong terms. It isn't addressed to the general public, or an article on Buzzfeed, it is an interdepartmental memo from an engineer signed by his manager to the Vice President of Engineering. Your example of "clear writing" would have just made this engineer look hyperbolic and likely undermined the issue even more. In a document like this, stating the physical problem up front is clear writing, and then, once the engineering problem is stated, he immediately states the possible outcome and then describes what he says as the management failure to allocate the appropriate resources, and how to fix it. How it comes off to you as a layman doesn't dictate whether or not this is good writing. He wasn't writing it to you.
- Pitarou 11y ago
  I respectfully disagree. Safety is of the highest priority, so if there's a serious risk to human life, there's nothing hyperbolic about drawing attention to it. If you can't do that, there's something very wrong with your engineering culture. To help the busy reader, the first paragraph of a memo should summarize the whole document. (Like the abstract of a technical paper.) This document is about an engineering problem and it's serious consequences, so they should both be mentioned in the first paragraph.

sopooneo 11y ago

I bet there were other such warnings, many of them valid, that just didn't turn out to be catastrophic. If there are any high level managers here, please tell me, does it seem like your people are constantly alerting you of dire risks? Does it feel like you are inundated with worriers?

js2 11y ago

An piece NPR did on Boisjoly after he passed away: http://www.npr.org/sections/thetwo-way/2012/02/06/146490064/remembering-roger-boisjoly-he-tried-to-stop-shuttle-challenger-launch http://www.npr.org/sections/thetwo-way/2012/02/06/146490064/... And an interview with one of Boisjoly's colleauges: http://www.npr.org/sections/thetwo-way/2016/01/28/464744781/30-years-after-disaster-challenger-engineer-still-blames-himself http://www.npr.org/sections/thetwo-way/2016/01/28/464744781/...

hackuser 11y ago

What stands out to me the impact of loyalty on corruption. Despite doing obviously the right thing: Boisjoly later revealed this memo to the presidential commission investigating the disaster and was then forced to leave Morton Thioklol after been shunned by disgruntled colleagues. Elsewhere in the discussion[1], we learn that key information about the O-rings had to be carefully, anonymously disclosed to protect the jobs of several people, including a prominent astronaut. How do we avoid corruption when loyalty uber alles is the rule of almost all organizations? [1] https://news.ycombinator.com/item?id=10992224 https://news.ycombinator.com/item?id=10992224

vlehto 11y ago
For small enough organization that "loyalty uber alles" prevents corruption. What is small enough? For a military company it's around Dunbar number. Squad that actually works together on single issue is often optimal around 8 individuals. Company, squad and brigade are most important formations, as they have dis-proportionally high number of things expected to handle independently. For this NASA o-ring bullshit, simplest solution seems to be having the technical managers and the astronauts in the same in-group. Now your loyalty is about not getting your mates killed. If you need a big organization, the trick is to have that internal network of squads and companies to work somehow non-corrupt and nice manner. This is where it gets hairy. Basically you can go with assumption of corruption and apply "transparency" or monetary incentives (sub contractors). Or you can assume that corruption does not happen. Sometimes the mere assumption that corruption does not happen stifles it. Practically for big companies failing less than your competitors is adequate for success.
- hackuser 11y ago
  Some interesting ideas; thanks. I don't quite grasp a couple of them: > For small enough organization that "loyalty uber alles" prevents corruption. How? There are many, many cases of people covering up for their fellow squad members. > Or you can assume that corruption does not happen. Sometimes the mere assumption that corruption does not happen stifles it. Hmmm ... that seems like a well-known recipe for encouraging corruption. How would that approach stifle it?