4 ms·
A lot of problems with automation vs manual controls is that the line between those tends to be blurry. It's not that automation wouldn't work in the absolute
by yason 7y ago
A lot of problems with automation vs manual controls is that the line between those tends to be blurry.
It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it.
It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations.
Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact.
In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that.
Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more.
If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.
- homonculus1 7y agoSounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.
- ethbro 7y agoIt's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us. Electricity, radiation, remote or sealed mechanical systems, deeply layered software. In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.
- qserasera 7y agoOperators may cause slight SOP deviations in emergency situations. Anticipating this in design between human controls and automated function is going to be very hard. I hope we can get voice input or other advanced input devices setup for communication between systems.
- ethbro 7y agoThe input isn't the problem IMHO, it's the output. If a human is in the loop at all then it's because they may be expected to make a decision. And that decision is based on information. And that information needs to be presented unambiguously. One of the smartest things I read on feedback here was to start with a balance-in, balance-out counting system. It's unambiguous: if the number of things being processed and output is different than the number of things input, that tells me something. And that's where automated systems get dangerous. It's incredibly risky if instead of informing and shrinking their response space, they expand it in an attempt to auto-correct the issue.
- deleted 7y ago[deleted]