6 ms·

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example,

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.

hughes 3y ago
Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.
- lisper 3y ago
  Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move in the right direction as one of the last checklist items before you take off. Every. Single. Time.)
  - maxerickson 3y ago
    So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
    - amluto 3y ago
      I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors. All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
      - maxerickson 3y ago
        A break out box could very sensibly have both sides of the connector on it and then have the various pins broken out into individual connections for flexibility. In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
      - jeffreygoesto 3y ago
        One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.
    - numpad0 3y ago
      Looks like the author tried to double-power the motor with both the spacecraft motor driver and through the breakout box that MITMs the driver and the motor. In such event, the free-wheeling diode in the driver will allow reverse current to be fed back to the driver's power supply up to certain amounts. This will absorb back-EMF, or energy from "regen" from the motor. I'm suspecting the breakout wasn't literally sitting between the driver and the motor, but rather all internal connections are broken out to the box for testing; and likely the author's mistake was to not mess with the spacecraft to temporarily disconnect the driver. But I'm not sure if I'd "just" made the right call and done so nonchalantly on a Mars rover to launch in few weeks.
  - markus92 3y ago
    How not to check your flight surfaces: Air Astana 1388.
    - 6LLvveMx2koXfwn 3y ago
      From the Wikipedia article of that flight: "The incident was featured in season 23, episode 5 of the Canadian documentary series Mayday . . ." [1] Season 23 - I'm glad I don't fly! 1. https://en.wikipedia.org/wiki/Air_Astana_Flight_1388 https://en.wikipedia.org/wiki/Air_Astana_Flight_1388
  - kristjansson 3y ago
    > flight control surfaces move in the right direction How … how often does that go wrong?!
    - lisper 3y ago
      It depends on what you mean by "that". Getting control surfaces actually reversed is not very common, but it does happen, typically after maintenance when a mechanic inadvertently re-connects a control cable backwards. Control cables also can and do break, but that too is fairly rare. What is not rare is control mechanisms jamming. Here is an example: https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.aspx https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.a...
    - ahazred8ta 3y ago
      There have been several cases of the landing gear up/down lever getting wired backwards during maintenance. Not to worry, the gear has a 'squat switch' sensor that prevents the gear from being raised when the plane is on the ground. Unless you taxi over a bump and the switch decides it's now airborne. Crunch.
- car 3y ago
  This is also the case for medical gas connectors in operating rooms, at least in Europe.
n8cpdx 3y ago
The one process part that can be controlled and jumped out from the first paragraphs is not letting people touch billion dollar equipment at the tail end of a 12-hour shift. If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued. I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.
astrolx 3y ago
I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ... Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.
- Dioxide2119 3y ago
  The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it. "It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone. I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
  - astrolx 3y ago
    Exactly :). The funny part is, the thing actually crashed! [1] Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie ! [1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-schiaparelli-anomaly-inquiry https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...
- zilti 3y ago
  And we all know how reliable ESA landers are. The laughing stock of the industry.

hughes 3y ago

Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.

lisper 3y ago
Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move in the right direction as one of the last checklist items before you take off. Every. Single. Time.)
- maxerickson 3y ago
  So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
  - amluto 3y ago
    I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors. All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
    - maxerickson 3y ago
      A break out box could very sensibly have both sides of the connector on it and then have the various pins broken out into individual connections for flexibility. In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
    - jeffreygoesto 3y ago
      One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.
  - numpad0 3y ago
    Looks like the author tried to double-power the motor with both the spacecraft motor driver and through the breakout box that MITMs the driver and the motor. In such event, the free-wheeling diode in the driver will allow reverse current to be fed back to the driver's power supply up to certain amounts. This will absorb back-EMF, or energy from "regen" from the motor. I'm suspecting the breakout wasn't literally sitting between the driver and the motor, but rather all internal connections are broken out to the box for testing; and likely the author's mistake was to not mess with the spacecraft to temporarily disconnect the driver. But I'm not sure if I'd "just" made the right call and done so nonchalantly on a Mars rover to launch in few weeks.
- markus92 3y ago
  How not to check your flight surfaces: Air Astana 1388.
  - 6LLvveMx2koXfwn 3y ago
    From the Wikipedia article of that flight: "The incident was featured in season 23, episode 5 of the Canadian documentary series Mayday . . ." [1] Season 23 - I'm glad I don't fly! 1. https://en.wikipedia.org/wiki/Air_Astana_Flight_1388 https://en.wikipedia.org/wiki/Air_Astana_Flight_1388
- kristjansson 3y ago
  > flight control surfaces move in the right direction How … how often does that go wrong?!
  - lisper 3y ago
    It depends on what you mean by "that". Getting control surfaces actually reversed is not very common, but it does happen, typically after maintenance when a mechanic inadvertently re-connects a control cable backwards. Control cables also can and do break, but that too is fairly rare. What is not rare is control mechanisms jamming. Here is an example: https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.aspx https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.a...
  - ahazred8ta 3y ago
    There have been several cases of the landing gear up/down lever getting wired backwards during maintenance. Not to worry, the gear has a 'squat switch' sensor that prevents the gear from being raised when the plane is on the ground. Unless you taxi over a bump and the switch decides it's now airborne. Crunch.
car 3y ago
This is also the case for medical gas connectors in operating rooms, at least in Europe.

n8cpdx 3y ago

The one process part that can be controlled and jumped out from the first paragraphs is not letting people touch billion dollar equipment at the tail end of a 12-hour shift. If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued. I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.

astrolx 3y ago

I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ... Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.

Dioxide2119 3y ago
The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it. "It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone. I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
- astrolx 3y ago
  Exactly :). The funny part is, the thing actually crashed! [1] Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie ! [1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-schiaparelli-anomaly-inquiry https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...
zilti 3y ago
And we all know how reliable ESA landers are. The laughing stock of the industry.