4 ms·
Parts installed “upside down” caused Russian rocket to explode last week
- smoyer 13y ago"young technician" I sure hope he doesn't end up being both a scapegoat and deemed a criminal. Problems like this are exactly why you let younger technicians be mentored by older (and ostensibly wiser) ones. It's also why you have visual inspections done by a different party.
- bjz_ 13y agoIt's also why you design the parts so that they are impossible to install upside down. No matter how clearly things are labelled, even the most experienced people will make mistakes.
- crististm 13y agoThe shuttle had a similar problem with a large piece of round metal that had more than one way to install of which only one was right. The mechanics knew this and asked for some markers on the piece to ease the process. The markers were never added not because of the cost of four paint points but because of the documentation that had to be updated. So there is your problem - bureaucracy. I'm sure that an engineer knew about the sensor mounting problem but nobody cared enough to make the change because of the paperwork.
- ryandrake 13y agoAnd why you design such parts so they cannot be physically installed upside down. http://en.wikipedia.org/wiki/Poka-yoke http://en.wikipedia.org/wiki/Poka-yoke
- jrarredondo 13y agoOr why this goes into configuration management and is verified before shipping. Seems not just the "young technician" failed but the process failed to catch the defect.
- compumike 13y agohttp://www.youtube.com/watch?v=ipNjYMJo79o#t=77s http://www.youtube.com/watch?v=ipNjYMJo79o#t=77s -- ignition begins at about 1:20
- andrewflnr 13y agoAnd now we know why they insist on such large perimeters. Imagine being on the side it was tipping toward.
- InclinedPlane 13y agoYou mean like these folks? http://www.youtube.com/watch?v=rfuXUr-_Rns http://www.youtube.com/watch?v=rfuXUr-_Rns
- jasonjackson 13y agoNote to self: when you see the rocket crash with your eyes, count to 5 seconds then cover your ears.
- erenemre 13y agoReminds me of https://kerbalspaceprogram.com/ https://kerbalspaceprogram.com/
- SEJeff 13y agoThis is why (in C code) I prefer what some people call "yoda conditions"[1]. It prevents problems where you accidentally assign when you mean to check equality. Like mentioned above, you should engineer everything, software and hardware, defensively. If the design simply forbade installing a part upside down, it would make up for the pathetic QA/QE which caused this issue in the first place. [1] http://en.m.wikipedia.org/wiki/Yoda_Conditions http://en.m.wikipedia.org/wiki/Yoda_Conditions
- krasin 13y agoClang (and GCC 4.7, I believe) would issue a warning for that. Instead of Yoda Conditions, it's recommended to use -Werror.
- jussij 13y agoThis is one (of many) reasons I have a dislike for Java: String myString = null; if (myString.equals("foobar")) { /* ... */ } // This causes a NullPointerException in Java Here is some equivalent C# code: using System; namespace SampleApplication { static class Program { static void Main(string[] args) { String myString = null; if (myString == "foobar") { Console.WriteLine("???"); } else { Console.WriteLine("No good!!"); } } } } and sure enough it produces what I consider the correct logical output: No good!!
- deleted 13y ago
- onan_barbarian 13y agoThis is an illustration of Murphy's Law, almost perfectly: "If there's more than one way to do a job, and one of those ways will result in disaster, then he will do it that way." Defensive design would be to make sure, as others have state on this thread, that it's impossible to make a mistake (e.g. an asymmetrical mount for the part).
- itcmcgrath 13y agoIf something wrong happens, don't punish the one who did it wrong - work out why the hell it was possible to do it wrong in the first place and solve that.
- mikeash 13y agoAnd Murphy coined his Law after being screwed by a part that was installed upside down!
- waster 13y agoThis reminds me of a comment under the Asiana Flight 214 discussion at https://news.ycombinator.com/item?id=6012214 https://news.ycombinator.com/item?id=6012214, that outsider views can help. In this case, as in the case of the Mars Climate Orbiter (http://en.wikipedia.org/wiki/Mars_Climate_Orbiter http://en.wikipedia.org/wiki/Mars_Climate_Orbiter), perhaps review of the work by someone who hasn't been scrutinizing it all day would have caught the errors? You stare at the same thing all day, and it's easy to get lost/not see your own mistakes. This can apply to supervisors, who might be inclined not to review their supervisees' work carefully because they always do such good work that it really doesn't occur to them that something this big could go wrong. And at the same time, they may well have been staring at it too long themselves. And as an aside, isn't the Russian space program always "beleaguered"?
- VladRussian2 13y ago>And as an aside, isn't the Russian space program always "beleaguered"? a decade old story about one of the unsuccessful launches goes this way - "the salary wasn't paid for several month, so not much work was done, once the launch date came within a month the government finally paid the salary and the satellite was built really fast, though some testing and some other things didn't really make it ..."
- robomartin 13y ago"each of those sensors had an arrow that was suppose to point towards the top of the vehicle" It's a little unbelievable to me that aerospace designers today would design a part who's only means of ensuring proper installation and alignment is a sticker with an arrow. If it was truly the case the error was one of engineering and not the fault of the technician who installed the parts. Engineers working on such projects should not make these kinds of mistakes. As an EE I had to learn mechanical design on my own. I quickly learned that it is paramount to have DFM (design for manufacturing) become a part of your design DNA. You need to think DFM for every little part you design. You need to think assembly for every little part you are designing. You need to ensure that parts or assemblies can be put together almost by a blind person and do so correctly. The tools available are simple: Alignment pins, unequally spaced holes, alignment tabs or machined features, unidirectional mating or clamping methods, non-rotatable insertion of sliding elements (d-pin into d-hole), etc. The same is true of EE design. Don't make a bunch of connectors exactly the same unless swapping the cables or daughterboards that mate with them is OK. One common technique is to use different pin count connectors. If a connector is reversible (such as a pin header used for, say, an RC servo) you need to make sure that reversing whatever mates to it will not cause damage. A more sophisticated approach, when it can be justified, is to make smart connectors that can deal with reversals by actively flipping signals around as required --common on ethernet switches and routers that can deal with straight and flipped signal pair cables. Again, I find it hard to believe that aerospace engineers would make such a basic error. You never know.
- Zigurd 13y agoIt seemed like a pretty resilient system. Those parts were right side up by the time the flight ended.
- ajross 13y ago> Engineers working on such projects should not make these kinds of mistakes. Sounds like a plan. In fact, it's inspired me in my own professional life. Henceforth, I shall never write another software bug again. The problem with the idea of this kind of thought is that it's turtles all the way down. You have an assembly mistake that could have been prevented by engineering. But of course now it's an engineering mistake. And you can't simply legislate the absence of mistakes. You can try to treat that with "engineering process", I guess, which mandates the use of funny gadgets or asymmetry or automated software checkers or whatever. And that probably helps. But then you can apply that process mistakenly... At some point you have to cut bait and ship (or launch, rather).
- natch 13y agoOriginal source article: http://www.russianspaceweb.com/proton_glonass49.html http://www.russianspaceweb.com/proton_glonass49.html
- danso 13y agoAs I've gotten more experienced, I've found it hard to work with people who don't optimize for safety and efficency...not because I'm a worrywart, but because experience had shown that fucking up (royally) is much less likely caused by lack of skill, industriousness, or moral character, than by unforeseen weaknesses, stress, and cascading errors. Virtually every ER surgeon has spent more energy and time into their work than I have in mine, and every day they deal with life and death, and yet failing to wash their hands is not a rare occurrence...why should I think I'm any more foolproof?
- iano 13y agoA similar issue was responsible for one of the V22 crashes ([1], page 4). In that case the gyro wires were hooked up backwards, instead of the unit installed backward. Both cases are good examples of small design details that make or break a project. [1] - http://www.fas.org/man/dod-101/sys/ac/v22-report.pdf http://www.fas.org/man/dod-101/sys/ac/v22-report.pdf
- InclinedPlane 13y agoLest you think this sort of thing is something that could only happen to the Ruskies, consider that the same sort of mistake has happened on NASA space probes, twice. The Galileo Jupiter atmospheric probe deployed its parachutes late because the accelerometer designed to trigger the pyro charges to release the parachutes was wired backwards. There were two different "g-switches" which provided acceleration data at different scales. When the probe entered Jupiter's atmosphere the software control system saw that the acceleration data it was getting was all wrong, it marked one of the channels as failed and attempted to use just the other channel, though the scale was all wrong so the data was bogus (leading the software to think that the probe was in a different stage of its entry than it actually was). The result was that the main parachute was deployed very late and the only reason the parachute successfully deployed at all at the speed and atmospheric density it was released into is by more or less sheer luck. The same mis-wiring happened on the Genesis sample return probe, which caused it to collide with the ground rather than be captured in mid-air as per the plan. The funny part is that the accelerometers on both probes were tested in a centrifuge, but the wiring harnesses for the tests were also wired backwards as well.
- grinich 13y agoThe lack of integrated testing was by far the most impressive thing about the Curiosity Rover/MSL landing to me. Most people who followed the news had no idea that the skycrane deployment setup was never actually tested in full. They validated components, like the hydrazine engines, radar, and bridle release mechanism, but not the combination of all. Software and circuits scale through abstraction and the static discipline. But combining atoms just gets harder and harder. (This sort of thing also happens when constructing buildings or bridges, but there's usually not a million different points for potential catastrophic failure.)