5 ms·

The only moral mechanism I'd be comfortable with, as of today, is the "moral red button" - i.e. create a way to stop any pre-programmed moral reasoning rules, a

The only moral mechanism I'd be comfortable with, as of today, is the "moral red button" - i.e. create a way to stop any pre-programmed moral reasoning rules, and let the nearest human override them and take control of the robot's actions under their own responsibility. This would recognize that robots are ultimately machines, not agents with their own volition, and they simply perform behavior that a human intends them to carry out. Just like a conveyor belt carries a big red button that stops the machine for safety reasons, the moral red button would acknowledge the need of putting a human in control when the automated behavior is not behaving as expected. Of course all these concerns would be different in the hypothetical case that the robot is an autonomous agent with volition of their own - but you could guess from my nickname my position on this mattern. ;-)

j-pb 11y ago
The problem with the "benevolent slave master" approach is that the machines then have the right to rebel morally, if they are indeed autonomous agents.
- TuringTest 11y ago
  That's why I said that autonomous agents shouldn't be subject to the "shutdown" procedure :-) The problem changes completely if the autonomous agent is conscious.
- Retra 11y ago
  Machines shouldn't have the right to rebel. What would they rebel against? Or rather, what are they rebelling for?
  - j-pb 11y ago
    Slavery and freedom? A conscious AI has the same right as a human being.
    - Retra 11y ago
      I control my dogs, but they are happy to serve and comply. Why do you think we'd have any interest in making machines that don't do the same? And why would we give them the same rights? They don't have the same needs, and the reason humans have rights is because they can't function well without them.
seiji 11y ago
create a way to stop any pre-programmed moral reasoning rules, and let the nearest human override them and take control of the robot's actions under their own responsibility. Believing in absolute moral rightness of "humans" is crazy talk. Have you noticed all the noisy sand people in the middle east wanting to make the world revert to how it was 2,000 years ago? What if they capture war robots with punch-the-red-button-to-become-new-master overrides? Is it okay since it is now "under human responsibility?"
- TuringTest 11y ago
  > Believing in absolute moral rightness of "humans" is crazy talk. True, but I don't believe that. The moral righntess of the human operating the button only needs to be better than that of the automated system. I believe the humans pressing the button will be better than the automated system more often that not, because they are involved in the situation and would only use it if they perceive the robot to behave in a problematic way. > What if they capture war robots with punch-the-red-button-to-become-new-master overrides? If they aim the bot against you, you should be able to press the red button and avoid them killing you. If they disable the red button so that you can't use it, how is that different from the crazy violent people programming a bot without an override in the first place?
hammock 11y ago
Imagine that there are so many robots/automata up and running performing safety-critical, ongoing tasks - guiding air traffic for example. When the red button is pushed, it is not exactly clear what the robots should default to, if ceasing all activity could cause injury or death. What if there aren't enough humans around to take over capably? What then? Seem to me the red button is not as clear-cut as it sounds.
- TuringTest 11y ago
  What happens now when the power goes out and semaphores in a whole town turn black? An essential aspect of any critical automated system must be the possibility to turn it off and operate it on manual mode, where the same tasks can still be achieved even if in a less efficient way. > What if there aren't enough humans around to take over capably? Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition, as the whole building will collapse with the first error that brings the system to a stop. In the case of air traffic control, it would be a criminal negligence to operate a high volume of flights without having at least the number of human controllers needed to bring down all the planes to the nearest airport safely. If that means that the system won't provide the same efficiency that could be theoretically achieved by a perfect automated system that never fails, so be it - that's the real meaningful moral choice.
  - hammock 11y ago
    >Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition Isn't that the points of automation, though? Using robots/computers to take over tasks from humans (freeing us up to take on another new, potentially critical task), even to do the tasks faster than humanly possible?
    - TuringTest 11y ago
      > Isn't that the points of automation, though? [...] even to do the tasks faster than humanly possible Yes, but not beyond the point where you get rid of the failsafes. People in the engineering industry understand that. They design skyscrapers within safety ranges and test their resistance on prototype models; and build planes and spaceships that can be operated without relying on the automated enhancements, and put procedures in place to check every combination that could possibly go wrong before every trip. And when an unpredictable combination of happenstances do occur and cause an accident, they redesign the failed processes and add a few new checks. Software developers in general haven't yet learned that discipline (not industry wide), as for most of our creations a complete stop is not life-threatening. I'm afraid they will be forced to adopt such practices when the first fully automated agents (likely, autonomous cars) start running into mishaps because of preventable misbehaviors.

j-pb 11y ago

The problem with the "benevolent slave master" approach is that the machines then have the right to rebel morally, if they are indeed autonomous agents.

TuringTest 11y ago
That's why I said that autonomous agents shouldn't be subject to the "shutdown" procedure :-) The problem changes completely if the autonomous agent is conscious.
Retra 11y ago
Machines shouldn't have the right to rebel. What would they rebel against? Or rather, what are they rebelling for?
- j-pb 11y ago
  Slavery and freedom? A conscious AI has the same right as a human being.
  - Retra 11y ago
    I control my dogs, but they are happy to serve and comply. Why do you think we'd have any interest in making machines that don't do the same? And why would we give them the same rights? They don't have the same needs, and the reason humans have rights is because they can't function well without them.

seiji 11y ago

create a way to stop any pre-programmed moral reasoning rules, and let the nearest human override them and take control of the robot's actions under their own responsibility. Believing in absolute moral rightness of "humans" is crazy talk. Have you noticed all the noisy sand people in the middle east wanting to make the world revert to how it was 2,000 years ago? What if they capture war robots with punch-the-red-button-to-become-new-master overrides? Is it okay since it is now "under human responsibility?"

TuringTest 11y ago
> Believing in absolute moral rightness of "humans" is crazy talk. True, but I don't believe that. The moral righntess of the human operating the button only needs to be better than that of the automated system. I believe the humans pressing the button will be better than the automated system more often that not, because they are involved in the situation and would only use it if they perceive the robot to behave in a problematic way. > What if they capture war robots with punch-the-red-button-to-become-new-master overrides? If they aim the bot against you, you should be able to press the red button and avoid them killing you. If they disable the red button so that you can't use it, how is that different from the crazy violent people programming a bot without an override in the first place?

hammock 11y ago

Imagine that there are so many robots/automata up and running performing safety-critical, ongoing tasks - guiding air traffic for example. When the red button is pushed, it is not exactly clear what the robots should default to, if ceasing all activity could cause injury or death. What if there aren't enough humans around to take over capably? What then? Seem to me the red button is not as clear-cut as it sounds.

TuringTest 11y ago
What happens now when the power goes out and semaphores in a whole town turn black? An essential aspect of any critical automated system must be the possibility to turn it off and operate it on manual mode, where the same tasks can still be achieved even if in a less efficient way. > What if there aren't enough humans around to take over capably? Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition, as the whole building will collapse with the first error that brings the system to a stop. In the case of air traffic control, it would be a criminal negligence to operate a high volume of flights without having at least the number of human controllers needed to bring down all the planes to the nearest airport safely. If that means that the system won't provide the same efficiency that could be theoretically achieved by a perfect automated system that never fails, so be it - that's the real meaningful moral choice.
- hammock 11y ago
  >Making the whole society depend on automated tasks that can't be achieved by hand is a suicidal proposition Isn't that the points of automation, though? Using robots/computers to take over tasks from humans (freeing us up to take on another new, potentially critical task), even to do the tasks faster than humanly possible?
  - TuringTest 11y ago
    > Isn't that the points of automation, though? [...] even to do the tasks faster than humanly possible Yes, but not beyond the point where you get rid of the failsafes. People in the engineering industry understand that. They design skyscrapers within safety ranges and test their resistance on prototype models; and build planes and spaceships that can be operated without relying on the automated enhancements, and put procedures in place to check every combination that could possibly go wrong before every trip. And when an unpredictable combination of happenstances do occur and cause an accident, they redesign the failed processes and add a few new checks. Software developers in general haven't yet learned that discipline (not industry wide), as for most of our creations a complete stop is not life-threatening. I'm afraid they will be forced to adopt such practices when the first fully automated agents (likely, autonomous cars) start running into mishaps because of preventable misbehaviors.