7 ms·

My intuition from experience is that in sufficiently complex systems (basically all modern hardware + software systems), things break from humans misunderstandi

by ethbro 7y ago

My intuition from experience is that in sufficiently complex systems (basically all modern hardware + software systems), things break from humans misunderstanding or not knowing the rules far more often than from physical failure. If there's a reason for using an electronic system, by all means. But for God's sake air-gap it, compartmentalize functionality with clearly defined interfaces, build simple fault-tolerant monitoring, and make final controls hand-on-metal. When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

qserasera 7y ago
>When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity. My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.
- agoodthrowaway 7y ago
  I wonder what would happen in an actual battle with the fog of war when people are pushed beyond exhaustion. What happens when senior command is dead or missing?
  - metaphor 7y ago
    There exists a very distinct line between routine/exercise and war. You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.
- dvdbloc 7y ago
  What event was this?
  - garrettgrimsley 7y ago
    Sounds like the US Navy destroyer that collided [0] with a cargo ship. [0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crystal_collision https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...
  - bin0 7y ago
    Based on google results, it appears to be an incident where the USS McCain collided with the Alnic MC near Singapore. She was getting off course, and poor UI made it so hard to steer her back that she crashed. Injuries and deaths resulted. https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroyer-6232e8c4e59e https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...
- Johnny555 7y ago
  Though one could say that this was due to too little automation rather than too much -- the default behavior could be for the ship to steer itself away from another vessel or shallow water unless overridden (setting off alarms that this action is being taken) My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.
  - qserasera 7y ago
    Unfortunately military ships are expensive and have strict schedules. You may be able to drive your car to a shop any day to diagnose a sensor error but military vessels dont have that luxury. To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.
  - joejerryronnie 7y ago
    The problem with automation is that it works perfectly until it runs into that edge case which nobody thought about. The problem with the real world is that there are infinity edge cases.
    - pferde 7y ago
      I'd like to use this in my .signature.
    - RomanBob 7y ago
      > there are infinity edge cases. this is simply not true. there are finite particles in discrete space so there's no infinity here.
    - Nasrudith 7y ago
- yason 7y ago
  A lot of problems with automation vs manual controls is that the line between those tends to be blurry. It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it. It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations. Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact. In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that. Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more. If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.
  - homonculus1 7y ago
    Sounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.
    - ethbro 7y ago
      It's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us. Electricity, radiation, remote or sealed mechanical systems, deeply layered software. In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.
russdill 7y ago
More than one large power outage has been caused by a human operater manually grounding live transmission lines by mistake.
- NegativeLatency 7y ago
  Sounds like an interesting situation, got any good links on the topic?
  - howard941 7y ago
    It's an extremely common situation with interesting solutions. If you've experienced the power going out, then coming on a few seconds later, then going out, and coming back on a few seconds later, then going out and staying out you've experienced a "recloser" attempting to clear a power fault (tree falling on power line perhaps, the recloser goes through the motions in an attempt to burn the thing away and clear the fault). A quick google turned up this fairly lengthy page https://www.electricaltechnology.org/2018/03/electrical-protection-systems-and-units.html#protection-devices https://www.electricaltechnology.org/2018/03/electrical-prot... but the protective systems are covered in depth in basic power transmission tech texts. Also here https://en.wikipedia.org/wiki/Recloser https://en.wikipedia.org/wiki/Recloser
    - NegativeLatency 7y ago
      Cool, thanks!

by ethbro 7y ago

qserasera 7y ago

>When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity. My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.

agoodthrowaway 7y ago
I wonder what would happen in an actual battle with the fog of war when people are pushed beyond exhaustion. What happens when senior command is dead or missing?
- metaphor 7y ago
  There exists a very distinct line between routine/exercise and war. You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.
dvdbloc 7y ago
What event was this?
- garrettgrimsley 7y ago
  Sounds like the US Navy destroyer that collided [0] with a cargo ship. [0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crystal_collision https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...
- bin0 7y ago
  Based on google results, it appears to be an incident where the USS McCain collided with the Alnic MC near Singapore. She was getting off course, and poor UI made it so hard to steer her back that she crashed. Injuries and deaths resulted. https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroyer-6232e8c4e59e https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...
Johnny555 7y ago
Though one could say that this was due to too little automation rather than too much -- the default behavior could be for the ship to steer itself away from another vessel or shallow water unless overridden (setting off alarms that this action is being taken) My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.
- qserasera 7y ago
  Unfortunately military ships are expensive and have strict schedules. You may be able to drive your car to a shop any day to diagnose a sensor error but military vessels dont have that luxury. To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.
- joejerryronnie 7y ago
  The problem with automation is that it works perfectly until it runs into that edge case which nobody thought about. The problem with the real world is that there are infinity edge cases.
  - pferde 7y ago
    I'd like to use this in my .signature.
  - RomanBob 7y ago
    > there are infinity edge cases. this is simply not true. there are finite particles in discrete space so there's no infinity here.
  - Nasrudith 7y ago
yason 7y ago
A lot of problems with automation vs manual controls is that the line between those tends to be blurry. It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it. It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations. Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact. In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that. Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more. If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.
- homonculus1 7y ago
  Sounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.
  - ethbro 7y ago
    It's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us. Electricity, radiation, remote or sealed mechanical systems, deeply layered software. In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.

russdill 7y ago

More than one large power outage has been caused by a human operater manually grounding live transmission lines by mistake.

NegativeLatency 7y ago
Sounds like an interesting situation, got any good links on the topic?
- howard941 7y ago
  It's an extremely common situation with interesting solutions. If you've experienced the power going out, then coming on a few seconds later, then going out, and coming back on a few seconds later, then going out and staying out you've experienced a "recloser" attempting to clear a power fault (tree falling on power line perhaps, the recloser goes through the motions in an attempt to burn the thing away and clear the fault). A quick google turned up this fairly lengthy page https://www.electricaltechnology.org/2018/03/electrical-protection-systems-and-units.html#protection-devices https://www.electricaltechnology.org/2018/03/electrical-prot... but the protective systems are covered in depth in basic power transmission tech texts. Also here https://en.wikipedia.org/wiki/Recloser https://en.wikipedia.org/wiki/Recloser
  - NegativeLatency 7y ago
    Cool, thanks!