5 ms·

While watching the new iPad UI course from CMU ( http://www.cmu.edu/homepage/computing/2012/winter/ipad-course.shtml http://www.cmu.edu/homepage/computing/2012/

While watching the new iPad UI course from CMU ( http://www.cmu.edu/homepage/computing/2012/winter/ipad-course.shtml http://www.cmu.edu/homepage/computing/2012/winter/ipad-cours... ), the lecturer touched on what happened with the Challenger, using it as an example in a point he was trying to make. However, I was struck by wrong he got it. He was making a point about how just having the data available isnt enough - the modern challenge is to make sense of big data in a meaningful way through visualization. And to back this point up, he made it sound like a bunch of engineers didn't look closely at the data sheet for the rubber used in their o-rings and oopsy! The challenger blew up. Shrug, Who knew? I'll tell you who knew. Roger. "The engineers" knew perfectly well what was going to happen. It was management who refused to listen to them and launched anyway. Feyman's account of his involvement in investigating this disaster in "the pleasure of finding things out" was excellent. In one exercise, he bad the engineers and management write down what they honestly thought the failure rate of the shuttle was. Management quoted 1 in 100,000, while engineering quoted 1 in 100. The challenger disaster was a symptom of a complete breakdown in communication between engineering and management at NASA.

gallamine 15y ago
I don't dispute what you say, but I think it's too easy to absolve the engineers of any fault. If they knew that the public was being lied to, and that there was a good chance of people being killed, they should have been more vocal. When lives are on the line, you can't just say, "well, I told management ... "
- hga 15y ago
  Don't forget the additional step of "Find a new career", for doing that will get you blacklisted just as the much milder actions of Boisjoly did.
ThomPete 15y ago
I think the point with that was that those who needed to know didn't know because they hadn't been presented the data in a meaningful way. So you are correct that he knew, but you are wrong in claiming the lecturer got it wrong. It is exactly because those who makes the decisions don't have the necessary know-how that visualizing the data in a meaningful way is so much more important.
- cellularmitosis 15y ago
  After reading Feyman's account, I believe that no amount of visualization nor compelling case making would have made any difference. 1 in 100,000 means launching the shuttle 3 times a day for nearly 100 years and experiencing only one failure. If the logical side of their brain is that far out to lunch, why would you think that the problem was simply an insufficient appeal to logic?
  - ThomPete 15y ago
    Visualization is not as such about logic, its about interpretation and communication. So the problem was more about insufficient appeal to emotion than to logic. Whether that would have been possible in the case of Challenger is another question, obviously retrospective analysis is always easier. But in my mind there is no doubt that visualization are able to change peoples opinions and understanding. I have seen that first hand on some projects.
- afterburner 15y ago
  The decision makers have a very, very strong incentive to keep things going. And the PR circus surrounding the teachers made it worse in the Challenger case. Meanwhile, everyone else has a strong incentive to shut the hell up. Besides, saying the engineers didn't "try hard enough" is essentially claiming the engineers were in charge. They were not. Managers are paid to manage their department, not just please the higher ups. When a claim is made that failure to address a problem will result in catastrophe and loss of life... really what more needs to be said, or drawn, to make the point? If you're not going to trust your own experts, why have them at all?
  - ThomPete 15y ago
    There aren't enough engineers in this world to oversee all potential dangers and errors.
    - afterburner 15y ago
      I really don't know what your point is. And how does the Challenger explosion in any way suggest that there aren't enough engineers? Certainly this one major issue had engineers urging for delay, so there doesn't seem to have been a lack of them. You know that all of the shuttles risks are well documented, right? It's a mountain of issues that basically gets the "ok to be ignored for now" stamp at every launch.
po 15y ago
The idea that the challenger accident came from poor data analysis comes from Tufte's (well-read) Visual Explanations: http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i... His argument (which the linked article takes to task) is that the engineers knew but were unable to convince due to their lack of ability to put together a convincing case for it. While Tufte's argument might have some merit, there was probably a lot more going on than just that. Perhaps even with a very compelling case presented, the launch would have gone through.
krschultz 15y ago
One of my college mechanical engineering professors was a former NASA engineer involved in determining risk and investigating problems. He gave a gave a lecture & led a discussion on the Challenger O-Ring case (and I have since read a lot of the reports). As you mentioned, the CMU lecturer wasn't even close to the real story. But it also ins't so simple as to say 'managment didn't listen'. The engineers knew what was going to happen with a fairly high degree of certainty - enough that the launch shouldn't hvae happened. Then they turned to management and tried to present their findings. Clearly the business guys didn't want to be the ones to hold up the launch, they had a strong incentive not to. So the engineers had to convince them why the company should basically stick its neck out as one of hundreds and hundreds of vendors and stop this launch. The engineers didn't do the best job of communicating the certainty of the problem, the magnitude of the problem, etc. So it was a little bit of both. That lecture was probably 8 years ago for me, but it definitely left a lasting impression. Today I work on systems that truthfully are more complicated than the space shuttle and also have a lot more lives at stake. "Intellectual honesty" is a phrase I live by. There have been times I fucked up some math and knew we needed to fix something - even though it would money and delay, but I stuck to my guns and made sure it happened. The alternative is too likely to lead to disaster. It keeps me up at night, and those are the near misses. I can't imagine what that guy went through knowing they almost stopped it, but didn't.
- ak217 15y ago
  Interesting perspective, thanks for sharing. Can you say what you work on, or at least give us an idea? Systems more complicated than the space shuttle - that sounds really interesting and I'm not sure which ones those are.
  - krschultz 15y ago
    I'd rather not share on HN because of how often it gets crawled by Google. You can generally find it through my LinkedIn which is easy enough to find from my HN handle.
- mduerksen 15y ago
  The engineers didn't do the best job of communicating the certainty of the problem, the magnitude of the problem, etc. That's a recurring theme when management screws up badly. Intellectual honesty applied to this particular situation: 1. The management should know that the engineers are more profound in the topic than they are. 2. The management should know that lives are at stake. 3. The management should notice that the engineers take their time and argue until late at night, mentioning "death", "explosion" 4. The management should know that communication skills may vary. But management still did not act upon it, because they did not know their limits. That's their fault, nobody else's. I'm glad you take your responsibility seriously.
lutorm 15y ago
I just read "The Challenger Launch Decision" (after someone here recommended it) and it has a more nuanced view of what went wrong. While there was a long history of concern about the O-rings, there was also a history of getting conflicting data, of implementing "fixes" that appeared to work, only to come back later. Even the engineers themselves, while expressing concern like the memo mentioned in the article, they did not, until the launch decision conference the night before, voice anything as strong as "stop the launch". During reading the book, I was struck by two things: one was the "slippery slope" introduced by first seeing something and arguing it was not a launch risk. With the decision making system in place, that meant it was very difficult to then, when another datum came in, to say that the two data together implied something was wrong, because they had already argued that the first datum was not a risk. This is of course a well-known phenomenon, but it didn't seem like the system was well-equipped to deal with sparse and sometimes contradictory data. The second was the feeling that what was happening was that the engineers were observing a random process and post-rationalizing cause and effect. (I had just read "Fooled by Randomness", which may have contributed to this.) Every piece of information was made to fit into some model, but it seemed like no one was considering that maybe what they were seeing was just inherent randomness. With that view, the progressively bad outcomes that happened before indicated that what was observed was a poorly-characterized random process with a fat tail. At some point, one rationalization was that they had a "safety factor of 4 left", but if you have indications of a fat-tailed process, that's not much to bank on.
- mturmon 15y ago
  Nice comment. As an engineer, I enjoyed reading the Columbia accident report also. It's available free at: http://www.nasa.gov/columbia/home/CAIB_Vol1.html http://www.nasa.gov/columbia/home/CAIB_Vol1.html It's detailed, but well-written and the quality of the analysis is extraordinary. If you want a preview, take a glance at the debris analysis on page 40. If you like understated drama, start at the comm transcript on page 42. ("Lock the doors.") Everyone at NASA got a hard copy of this report.
  - lutorm 15y ago
    Yeah, I remember reading the CAIB report when it came out. Scott Hubbard, one of the board members, also came by UCSC and gave a talk about the investigation, like how everyone had assumed that a piece of foam couldn't do anything serious to the RCC leading edge until they shot a foam piece of similar size and speed at a spare with an air gun and it went clean through.

gallamine 15y ago

I don't dispute what you say, but I think it's too easy to absolve the engineers of any fault. If they knew that the public was being lied to, and that there was a good chance of people being killed, they should have been more vocal. When lives are on the line, you can't just say, "well, I told management ... "

hga 15y ago
Don't forget the additional step of "Find a new career", for doing that will get you blacklisted just as the much milder actions of Boisjoly did.

ThomPete 15y ago

I think the point with that was that those who needed to know didn't know because they hadn't been presented the data in a meaningful way. So you are correct that he knew, but you are wrong in claiming the lecturer got it wrong. It is exactly because those who makes the decisions don't have the necessary know-how that visualizing the data in a meaningful way is so much more important.

cellularmitosis 15y ago
After reading Feyman's account, I believe that no amount of visualization nor compelling case making would have made any difference. 1 in 100,000 means launching the shuttle 3 times a day for nearly 100 years and experiencing only one failure. If the logical side of their brain is that far out to lunch, why would you think that the problem was simply an insufficient appeal to logic?
- ThomPete 15y ago
  Visualization is not as such about logic, its about interpretation and communication. So the problem was more about insufficient appeal to emotion than to logic. Whether that would have been possible in the case of Challenger is another question, obviously retrospective analysis is always easier. But in my mind there is no doubt that visualization are able to change peoples opinions and understanding. I have seen that first hand on some projects.
afterburner 15y ago
The decision makers have a very, very strong incentive to keep things going. And the PR circus surrounding the teachers made it worse in the Challenger case. Meanwhile, everyone else has a strong incentive to shut the hell up. Besides, saying the engineers didn't "try hard enough" is essentially claiming the engineers were in charge. They were not. Managers are paid to manage their department, not just please the higher ups. When a claim is made that failure to address a problem will result in catastrophe and loss of life... really what more needs to be said, or drawn, to make the point? If you're not going to trust your own experts, why have them at all?
- ThomPete 15y ago
  There aren't enough engineers in this world to oversee all potential dangers and errors.
  - afterburner 15y ago
    I really don't know what your point is. And how does the Challenger explosion in any way suggest that there aren't enough engineers? Certainly this one major issue had engineers urging for delay, so there doesn't seem to have been a lack of them. You know that all of the shuttles risks are well documented, right? It's a mountain of issues that basically gets the "ok to be ignored for now" stamp at every launch.

po 15y ago

The idea that the challenger accident came from poor data analysis comes from Tufte's (well-read) Visual Explanations: http://www.onlineethics.org/CMS/profpractice/exempindex/RB-intro/RepMisrep.aspx http://www.onlineethics.org/CMS/profpractice/exempindex/RB-i... His argument (which the linked article takes to task) is that the engineers knew but were unable to convince due to their lack of ability to put together a convincing case for it. While Tufte's argument might have some merit, there was probably a lot more going on than just that. Perhaps even with a very compelling case presented, the launch would have gone through.

krschultz 15y ago

One of my college mechanical engineering professors was a former NASA engineer involved in determining risk and investigating problems. He gave a gave a lecture & led a discussion on the Challenger O-Ring case (and I have since read a lot of the reports). As you mentioned, the CMU lecturer wasn't even close to the real story. But it also ins't so simple as to say 'managment didn't listen'. The engineers knew what was going to happen with a fairly high degree of certainty - enough that the launch shouldn't hvae happened. Then they turned to management and tried to present their findings. Clearly the business guys didn't want to be the ones to hold up the launch, they had a strong incentive not to. So the engineers had to convince them why the company should basically stick its neck out as one of hundreds and hundreds of vendors and stop this launch. The engineers didn't do the best job of communicating the certainty of the problem, the magnitude of the problem, etc. So it was a little bit of both. That lecture was probably 8 years ago for me, but it definitely left a lasting impression. Today I work on systems that truthfully are more complicated than the space shuttle and also have a lot more lives at stake. "Intellectual honesty" is a phrase I live by. There have been times I fucked up some math and knew we needed to fix something - even though it would money and delay, but I stuck to my guns and made sure it happened. The alternative is too likely to lead to disaster. It keeps me up at night, and those are the near misses. I can't imagine what that guy went through knowing they almost stopped it, but didn't.

ak217 15y ago
Interesting perspective, thanks for sharing. Can you say what you work on, or at least give us an idea? Systems more complicated than the space shuttle - that sounds really interesting and I'm not sure which ones those are.
- krschultz 15y ago
  I'd rather not share on HN because of how often it gets crawled by Google. You can generally find it through my LinkedIn which is easy enough to find from my HN handle.
mduerksen 15y ago
The engineers didn't do the best job of communicating the certainty of the problem, the magnitude of the problem, etc. That's a recurring theme when management screws up badly. Intellectual honesty applied to this particular situation: 1. The management should know that the engineers are more profound in the topic than they are. 2. The management should know that lives are at stake. 3. The management should notice that the engineers take their time and argue until late at night, mentioning "death", "explosion" 4. The management should know that communication skills may vary. But management still did not act upon it, because they did not know their limits. That's their fault, nobody else's. I'm glad you take your responsibility seriously.

lutorm 15y ago

I just read "The Challenger Launch Decision" (after someone here recommended it) and it has a more nuanced view of what went wrong. While there was a long history of concern about the O-rings, there was also a history of getting conflicting data, of implementing "fixes" that appeared to work, only to come back later. Even the engineers themselves, while expressing concern like the memo mentioned in the article, they did not, until the launch decision conference the night before, voice anything as strong as "stop the launch". During reading the book, I was struck by two things: one was the "slippery slope" introduced by first seeing something and arguing it was not a launch risk. With the decision making system in place, that meant it was very difficult to then, when another datum came in, to say that the two data together implied something was wrong, because they had already argued that the first datum was not a risk. This is of course a well-known phenomenon, but it didn't seem like the system was well-equipped to deal with sparse and sometimes contradictory data. The second was the feeling that what was happening was that the engineers were observing a random process and post-rationalizing cause and effect. (I had just read "Fooled by Randomness", which may have contributed to this.) Every piece of information was made to fit into some model, but it seemed like no one was considering that maybe what they were seeing was just inherent randomness. With that view, the progressively bad outcomes that happened before indicated that what was observed was a poorly-characterized random process with a fat tail. At some point, one rationalization was that they had a "safety factor of 4 left", but if you have indications of a fat-tailed process, that's not much to bank on.

mturmon 15y ago
Nice comment. As an engineer, I enjoyed reading the Columbia accident report also. It's available free at: http://www.nasa.gov/columbia/home/CAIB_Vol1.html http://www.nasa.gov/columbia/home/CAIB_Vol1.html It's detailed, but well-written and the quality of the analysis is extraordinary. If you want a preview, take a glance at the debris analysis on page 40. If you like understated drama, start at the comm transcript on page 42. ("Lock the doors.") Everyone at NASA got a hard copy of this report.
- lutorm 15y ago
  Yeah, I remember reading the CAIB report when it came out. Scott Hubbard, one of the board members, also came by UCSC and gave a talk about the investigation, like how everyone had assumed that a piece of foam couldn't do anything serious to the RCC leading edge until they shot a foam piece of similar size and speed at a spare with an air gun and it went clean through.