4 ms·

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Eithe

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming individuals only gets them fired and weakens the entire organization. You just fired the one person who learned an expensive lesson. The only time when an individual should be blamed is when they intended harm, at which case the law could kick in

idlewords 3y ago
You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.
- hughes 3y ago
  Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.
  - lisper 3y ago
    Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move in the right direction as one of the last checklist items before you take off. Every. Single. Time.)
    - maxerickson 3y ago
      So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
      - amluto 3y ago
        I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors. All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
        maxerickson 3y ago
        A break out box could very sensibly have both sides of the connector on it and then have the various pins broken out into individual connections for flexibility. In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
        jeffreygoesto 3y ago
        One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.
      - numpad0 3y ago
  - car 3y ago
    This is also the case for medical gas connectors in operating rooms, at least in Europe.
- n8cpdx 3y ago
  The one process part that can be controlled and jumped out from the first paragraphs is not letting people touch billion dollar equipment at the tail end of a 12-hour shift. If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued. I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.
- astrolx 3y ago
  I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ... Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.
  - Dioxide2119 3y ago
    The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it. "It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone. I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
    - astrolx 3y ago
      Exactly :). The funny part is, the thing actually crashed! [1] Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie ! [1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-schiaparelli-anomaly-inquiry https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...
  - zilti 3y ago
    And we all know how reliable ESA landers are. The laughing stock of the industry.
deepsun 3y ago
> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
- ycombobreaker 3y ago
  Firing somebody for a simple mistake with grave consequences doesn't make your organization stronger. There will be plenty of better examples to make.
- skrebbel 3y ago
  Firing people for incidental mistakes instead of overall bad performance is pretty shitty management.
  - deepsun 3y ago
    For one incidental mistake of course not. For repeated inattention (like plugging mars rover's cables wrongly several times) at an attention-demanding job -- yes.
- dividedbyzero 3y ago
  So you'd not blame them for their simple mistake, but still fire them for it?

idlewords 3y ago

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.

hughes 3y ago
Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.
- lisper 3y ago
  Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move in the right direction as one of the last checklist items before you take off. Every. Single. Time.)
  - maxerickson 3y ago
    So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
    - amluto 3y ago
      I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors. All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
      - maxerickson 3y ago
        A break out box could very sensibly have both sides of the connector on it and then have the various pins broken out into individual connections for flexibility. In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
      - jeffreygoesto 3y ago
        One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.
    - numpad0 3y ago
- car 3y ago
  This is also the case for medical gas connectors in operating rooms, at least in Europe.
n8cpdx 3y ago
The one process part that can be controlled and jumped out from the first paragraphs is not letting people touch billion dollar equipment at the tail end of a 12-hour shift. If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued. I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.
astrolx 3y ago
I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ... Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.
- Dioxide2119 3y ago
  The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it. "It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone. I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
  - astrolx 3y ago
    Exactly :). The funny part is, the thing actually crashed! [1] Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie ! [1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-schiaparelli-anomaly-inquiry https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...
- zilti 3y ago
  And we all know how reliable ESA landers are. The laughing stock of the industry.

deepsun 3y ago

> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

ycombobreaker 3y ago
Firing somebody for a simple mistake with grave consequences doesn't make your organization stronger. There will be plenty of better examples to make.
skrebbel 3y ago
Firing people for incidental mistakes instead of overall bad performance is pretty shitty management.
- deepsun 3y ago
  For one incidental mistake of course not. For repeated inattention (like plugging mars rover's cables wrongly several times) at an attention-demanding job -- yes.
dividedbyzero 3y ago
So you'd not blame them for their simple mistake, but still fire them for it?