5 ms·

Metrorail Crash May Exemplify Automation Paradox

19 points by bootload 17y ago

ZeroGravitas 17y ago
Most of these just seem like standard bad design or human stupidity. I think there may be a "paradox" but I'm not sure it's well sketched out here.
- ars 17y ago
  The paradox is that the better the automation, the more humans rely on the machine. But when it fails, humans trust it too much. Or you can make it less reliable, and cause the human to trust it less, but that has issues too. It's hard to find a balance point, neither direction is good. A solution from outside the box is needed here - make the automation good, but in a way that the human input is encouraged.
  - randallsquared 17y ago
    Make failure immediately visible.
    - skolor 17y ago
      Make failure immediately visible in a non-destructive way. A nuclear reactor that fails and explodes immediately is not a good thing. A nuclear reactor that fails and notifies someone qualified to fix the problem immediately is a very good thing.
      - randallsquared 17y ago
        Well, the details are application-specific, yeah. The metrorail problem could have been "solved" by making the system refuse to allow a train to travel on track that didn't have a currently working train sensor, but such failures are so common in the system that the whole Metro would grind to a halt, from my reading this summer about this problem (mostly while on the Metro, fittingly).
    - ars 17y ago
      > Make failure immediately visible. If you are aware of failure, then why fail? The entire problem is that the computer is not aware that it failed. You need the human to constantly check that everything is OK. But if the computer is always working fine, then human nature is such that eventually the human will stop paying such close attention.
      - randallsquared 17y ago
        There's a pretty large class of problems where the computer could be aware of failure, and not able to do anything about it. Just because you know that a sensor is failing doesn't mean that you can fix it in software; the best you can do force a human to acknowledge the failure before they continue. I think a bigger problem than humans relying on things that don't fail is humans getting used to ignoring failure because the failures typically don't matter.
        ars 17y ago
        But are there problems where the computer is aware of failure and requires the human to do something about it? I think that's pretty rare. Usually if it fails, it can handle that on it's own. (For example stop the train.) How often would you have a failure where the human must intervene, and the computer can do nothing? That's what I meant by don't fail, have the computer handle the problem. If you handled it, it's not a failure (from a global perspective). The real problem is where the computer does not know it failed.
  - jamesbritt 17y ago
    "The paradox is that the better the automation, the more humans rely on the machine." But how is that a paradox? It may be somewhat ironic, but there's no inherent contradiction or impossibilty.
    - ars 17y ago
      The contradiction is that the better the computer becomes, the less the human pays attention. So when the computer does fail the human doesn't notice. The other option is to make the computer less good, so the human is forced to pay attention, but that has it's own problems. So how do you make a system where the computer is very good, and the human pays attention? (And saying force the human, doesn't count.)
      - ZeroGravitas 17y ago
        The other option is to fail-safe: http://en.wikipedia.org/wiki/Fail-safe http://en.wikipedia.org/wiki/Fail-safe
devicenull 17y ago
I'm not convinced that the ship that crashed was a good design. When GPS fails, it seems logical to switch to dead reckoning only if there is no human available to take over. If you've got a human crew available, it seems like a better choice to just stop the ship and let them handle it.
- lnguyen 17y ago
  The design issue in this case is not making it blatantly and unavoidably obvious that the GPS had failed. The dead reckoning system should probably also give an error range instead of keeping any appearance of the same "pinpoint" accuracy.
  - ars 17y ago
    The system was self steering (i.e. driving the ship) - there was no one to give an error range to. No one was watching it because it normally works so well that there is no reason to double check it. An error indicator is no help if no one goes to look at it.

19 points by bootload 17y ago

Most of these just seem like standard bad design or human stupidity. I think there may be a "paradox" but I'm not sure it's well sketched out here.

ars 17y ago
The paradox is that the better the automation, the more humans rely on the machine. But when it fails, humans trust it too much. Or you can make it less reliable, and cause the human to trust it less, but that has issues too. It's hard to find a balance point, neither direction is good. A solution from outside the box is needed here - make the automation good, but in a way that the human input is encouraged.
- randallsquared 17y ago
  Make failure immediately visible.
  - skolor 17y ago
    Make failure immediately visible in a non-destructive way. A nuclear reactor that fails and explodes immediately is not a good thing. A nuclear reactor that fails and notifies someone qualified to fix the problem immediately is a very good thing.
    - randallsquared 17y ago
      Well, the details are application-specific, yeah. The metrorail problem could have been "solved" by making the system refuse to allow a train to travel on track that didn't have a currently working train sensor, but such failures are so common in the system that the whole Metro would grind to a halt, from my reading this summer about this problem (mostly while on the Metro, fittingly).
  - ars 17y ago
    > Make failure immediately visible. If you are aware of failure, then why fail? The entire problem is that the computer is not aware that it failed. You need the human to constantly check that everything is OK. But if the computer is always working fine, then human nature is such that eventually the human will stop paying such close attention.
    - randallsquared 17y ago
      There's a pretty large class of problems where the computer could be aware of failure, and not able to do anything about it. Just because you know that a sensor is failing doesn't mean that you can fix it in software; the best you can do force a human to acknowledge the failure before they continue. I think a bigger problem than humans relying on things that don't fail is humans getting used to ignoring failure because the failures typically don't matter.
      - ars 17y ago
        But are there problems where the computer is aware of failure and requires the human to do something about it? I think that's pretty rare. Usually if it fails, it can handle that on it's own. (For example stop the train.) How often would you have a failure where the human must intervene, and the computer can do nothing? That's what I meant by don't fail, have the computer handle the problem. If you handled it, it's not a failure (from a global perspective). The real problem is where the computer does not know it failed.
- jamesbritt 17y ago
  "The paradox is that the better the automation, the more humans rely on the machine." But how is that a paradox? It may be somewhat ironic, but there's no inherent contradiction or impossibilty.
  - ars 17y ago
    The contradiction is that the better the computer becomes, the less the human pays attention. So when the computer does fail the human doesn't notice. The other option is to make the computer less good, so the human is forced to pay attention, but that has it's own problems. So how do you make a system where the computer is very good, and the human pays attention? (And saying force the human, doesn't count.)
    - ZeroGravitas 17y ago
      The other option is to fail-safe: http://en.wikipedia.org/wiki/Fail-safe http://en.wikipedia.org/wiki/Fail-safe

devicenull 17y ago

I'm not convinced that the ship that crashed was a good design. When GPS fails, it seems logical to switch to dead reckoning only if there is no human available to take over. If you've got a human crew available, it seems like a better choice to just stop the ship and let them handle it.

lnguyen 17y ago
The design issue in this case is not making it blatantly and unavoidably obvious that the GPS had failed. The dead reckoning system should probably also give an error range instead of keeping any appearance of the same "pinpoint" accuracy.
- ars 17y ago
  The system was self steering (i.e. driving the ship) - there was no one to give an error range to. No one was watching it because it normally works so well that there is no reason to double check it. An error indicator is no help if no one goes to look at it.