5 ms·

What happens now when the power goes out and semaphores in a whole town turn black? An essential aspect of any critical automated system must be the possibility

What happens now when the power goes out and semaphores in a whole town turn black? An essential aspect of any critical automated system must be the possibility to turn it off and operate it on manual mode, where the same tasks can still be achieved even if in a less efficient way. > What if there aren't enough humans around to take over capably? Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition, as the whole building will collapse with the first error that brings the system to a stop. In the case of air traffic control, it would be a criminal negligence to operate a high volume of flights without having at least the number of human controllers needed to bring down all the planes to the nearest airport safely. If that means that the system won't provide the same efficiency that could be theoretically achieved by a perfect automated system that never fails, so be it - that's the real meaningful moral choice.

hammock 11y ago
>Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition Isn't that the points of automation, though? Using robots/computers to take over tasks from humans (freeing us up to take on another new, potentially critical task), even to do the tasks faster than humanly possible?
- TuringTest 11y ago
  > Isn't that the points of automation, though? [...] even to do the tasks faster than humanly possible Yes, but not beyond the point where you get rid of the failsafes. People in the engineering industry understand that. They design skyscrapers within safety ranges and test their resistance on prototype models; and build planes and spaceships that can be operated without relying on the automated enhancements, and put procedures in place to check every combination that could possibly go wrong before every trip. And when an unpredictable combination of happenstances do occur and cause an accident, they redesign the failed processes and add a few new checks. Software developers in general haven't yet learned that discipline (not industry wide), as for most of our creations a complete stop is not life-threatening. I'm afraid they will be forced to adopt such practices when the first fully automated agents (likely, autonomous cars) start running into mishaps because of preventable misbehaviors.

hammock 11y ago

>Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition Isn't that the points of automation, though? Using robots/computers to take over tasks from humans (freeing us up to take on another new, potentially critical task), even to do the tasks faster than humanly possible?

TuringTest 11y ago
> Isn't that the points of automation, though? [...] even to do the tasks faster than humanly possible Yes, but not beyond the point where you get rid of the failsafes. People in the engineering industry understand that. They design skyscrapers within safety ranges and test their resistance on prototype models; and build planes and spaceships that can be operated without relying on the automated enhancements, and put procedures in place to check every combination that could possibly go wrong before every trip. And when an unpredictable combination of happenstances do occur and cause an accident, they redesign the failed processes and add a few new checks. Software developers in general haven't yet learned that discipline (not industry wide), as for most of our creations a complete stop is not life-threatening. I'm afraid they will be forced to adopt such practices when the first fully automated agents (likely, autonomous cars) start running into mishaps because of preventable misbehaviors.