3 ms·
The paradox is that the better the automation, the more humans rely on the machine. But when it fails, humans trust it too much. Or you can make it less relia
by ars 17y ago
The paradox is that the better the automation, the more humans rely on the machine.
But when it fails, humans trust it too much.
Or you can make it less reliable, and cause the human to trust it less, but that has issues too.
It's hard to find a balance point, neither direction is good.
A solution from outside the box is needed here - make the automation good, but in a way that the human input is encouraged.
- randallsquared 17y agoMake failure immediately visible.
- skolor 17y agoMake failure immediately visible in a non-destructive way. A nuclear reactor that fails and explodes immediately is not a good thing. A nuclear reactor that fails and notifies someone qualified to fix the problem immediately is a very good thing.
- randallsquared 17y agoWell, the details are application-specific, yeah. The metrorail problem could have been "solved" by making the system refuse to allow a train to travel on track that didn't have a currently working train sensor, but such failures are so common in the system that the whole Metro would grind to a halt, from my reading this summer about this problem (mostly while on the Metro, fittingly).
- ars 17y ago> Make failure immediately visible. If you are aware of failure, then why fail? The entire problem is that the computer is not aware that it failed. You need the human to constantly check that everything is OK. But if the computer is always working fine, then human nature is such that eventually the human will stop paying such close attention.
- randallsquared 17y agoThere's a pretty large class of problems where the computer could be aware of failure, and not able to do anything about it. Just because you know that a sensor is failing doesn't mean that you can fix it in software; the best you can do force a human to acknowledge the failure before they continue. I think a bigger problem than humans relying on things that don't fail is humans getting used to ignoring failure because the failures typically don't matter.
- ars 17y agoBut are there problems where the computer is aware of failure and requires the human to do something about it? I think that's pretty rare. Usually if it fails, it can handle that on it's own. (For example stop the train.) How often would you have a failure where the human must intervene, and the computer can do nothing? That's what I meant by don't fail, have the computer handle the problem. If you handled it, it's not a failure (from a global perspective). The real problem is where the computer does not know it failed.
- jamesbritt 17y ago"The paradox is that the better the automation, the more humans rely on the machine." But how is that a paradox? It may be somewhat ironic, but there's no inherent contradiction or impossibilty.
- ars 17y agoThe contradiction is that the better the computer becomes, the less the human pays attention. So when the computer does fail the human doesn't notice. The other option is to make the computer less good, so the human is forced to pay attention, but that has it's own problems. So how do you make a system where the computer is very good, and the human pays attention? (And saying force the human, doesn't count.)
- ZeroGravitas 17y agoThe other option is to fail-safe: http://en.wikipedia.org/wiki/Fail-safe http://en.wikipedia.org/wiki/Fail-safe