5 ms·

This question exists because humans demand a cause for each effect, and a "responsible party" for every tragedy. Sometimes, things are just "accidents." A coll

by akd 17y ago

This question exists because humans demand a cause for each effect, and a "responsible party" for every tragedy. Sometimes, things are just "accidents." A colleague of my dad's tripped and fell while walking down a set of three stairs, hit his head, and died. It was nobody's fault -- it was just bad luck.

gdp 17y ago
But that ignores the idea that somebody designs and builds an autonomous machine. If an airplane autopilot sends a plane full of people into a mountain due to a defect in the software, who is liable?
- sokoloff 17y ago
  The FAA (and common sense) treats an autopilot as just a fancy interface to the actual control surfaces of the aircraft, meaning the pilot-in-command is still responsible. Even on autopilot, I'm supposed to be monitoring the systems, keeping track of where we are, where we're headed, the weather, etc. If my autopilot sends me towards a mountain, it's on me (practically and legally) to fix it.
  - gdp 17y ago
    Which seems to provide a possibly workable model for automated systems of other kinds.
    - gloob 17y ago
      Precisely: we'll hold the pilot of the autonomous drone responsible. Edit: Actually, this raises a question in my mind. Do autonomous military robots have "handlers" or the like?
  - roundsquare 17y ago
    Sure, but if you see some bad weather and decide to turn off autopilot... and the system won't let you, then the company is to blame (legally, though practically you'd obviously do everything you can to make sure you survive). The decision would need to be made case by case in courts to determine who (if anyone) is responsible. At least in court systems like the US has. For very different systems, I'm not sure.
    - sokoloff 17y ago
      The certification process for autopilots, even for light single-engine aircraft, is designed to prevent "autopilot can't be turned off or over-ridden" scenarios. Can it prevent all of them? No, but mine has a mandatory self-test that has to complete before it can be engaged, there is a dedicated breaker with a collar so I can kill power to just the autopilot, failing that I can kill the power to all avionics, failing that kill all electrical power, and failing all that each of the servos has a clutch on it that can easily be over-ridden by even the daintiest pilot. If the autopilot is commanding a left turn that I don't want, I can mechanically override it, or I have 4 electrical ways to shut it off. That takes "autopilot can't be disengaged" way, way down on the list of things to worry about, especially when compared to flying behind a single piston engine, with no anti-icing equipment. If I fly my airplane into a mountain, the NTSB report will say: Probable cause: The pilot in command's failure to maintain clearance from terrain, resulting in in-flight collision with mountainous terrain. They may add: Contributing causes were: <some list of lesser causes such as weather, lighting conditions, failure of navigation equipment, projected spatial disorientation, and if they have evidence to support it, autopilot malfunction or more likely, pilot's incorrect or insufficient reaction to that malfunction>. The likelihood that S-tec [maker of my autopilot] is found liable is precisely 0.00%.
      - roundsquare 17y ago
        Wow... taking the autopilot example too literally. I suppose there is no need for me to point this out, but I'm not a pilot and have never flown anything, with or without autopilot. I was just using this as an example of "something going wrong with the machine" to show that there are times you would blame the pilot and times you would blame the company who built the autopilot. If autopilot is a bad example, replace it with some other autonomous machine... or pretend that we were back in the stone ages of autopilots when things could go wrong. Or, I'll make up a new example where the autopilot all of a sudden inverts the plane and then sets it right before you get a chance to turn it off therefore injuring one of the passengers. Lets not get too caught up in the specifics of autopilots and aviation laws, etc... That said, having a number of failsafe mechanisms (like the ones you described) would be a critical part of any autonomous system, especially in the early stages, which would, of course, affect who we end up blaming.

by akd 17y ago

gdp 17y ago

But that ignores the idea that somebody designs and builds an autonomous machine. If an airplane autopilot sends a plane full of people into a mountain due to a defect in the software, who is liable?

sokoloff 17y ago
The FAA (and common sense) treats an autopilot as just a fancy interface to the actual control surfaces of the aircraft, meaning the pilot-in-command is still responsible. Even on autopilot, I'm supposed to be monitoring the systems, keeping track of where we are, where we're headed, the weather, etc. If my autopilot sends me towards a mountain, it's on me (practically and legally) to fix it.
- gdp 17y ago
  Which seems to provide a possibly workable model for automated systems of other kinds.
  - gloob 17y ago
    Precisely: we'll hold the pilot of the autonomous drone responsible. Edit: Actually, this raises a question in my mind. Do autonomous military robots have "handlers" or the like?
- roundsquare 17y ago
  Sure, but if you see some bad weather and decide to turn off autopilot... and the system won't let you, then the company is to blame (legally, though practically you'd obviously do everything you can to make sure you survive). The decision would need to be made case by case in courts to determine who (if anyone) is responsible. At least in court systems like the US has. For very different systems, I'm not sure.
  - sokoloff 17y ago
    The certification process for autopilots, even for light single-engine aircraft, is designed to prevent "autopilot can't be turned off or over-ridden" scenarios. Can it prevent all of them? No, but mine has a mandatory self-test that has to complete before it can be engaged, there is a dedicated breaker with a collar so I can kill power to just the autopilot, failing that I can kill the power to all avionics, failing that kill all electrical power, and failing all that each of the servos has a clutch on it that can easily be over-ridden by even the daintiest pilot. If the autopilot is commanding a left turn that I don't want, I can mechanically override it, or I have 4 electrical ways to shut it off. That takes "autopilot can't be disengaged" way, way down on the list of things to worry about, especially when compared to flying behind a single piston engine, with no anti-icing equipment. If I fly my airplane into a mountain, the NTSB report will say: Probable cause: The pilot in command's failure to maintain clearance from terrain, resulting in in-flight collision with mountainous terrain. They may add: Contributing causes were: <some list of lesser causes such as weather, lighting conditions, failure of navigation equipment, projected spatial disorientation, and if they have evidence to support it, autopilot malfunction or more likely, pilot's incorrect or insufficient reaction to that malfunction>. The likelihood that S-tec [maker of my autopilot] is found liable is precisely 0.00%.
    - roundsquare 17y ago
      Wow... taking the autopilot example too literally. I suppose there is no need for me to point this out, but I'm not a pilot and have never flown anything, with or without autopilot. I was just using this as an example of "something going wrong with the machine" to show that there are times you would blame the pilot and times you would blame the company who built the autopilot. If autopilot is a bad example, replace it with some other autonomous machine... or pretend that we were back in the stone ages of autopilots when things could go wrong. Or, I'll make up a new example where the autopilot all of a sudden inverts the plane and then sets it right before you get a chance to turn it off therefore injuring one of the passengers. Lets not get too caught up in the specifics of autopilots and aviation laws, etc... That said, having a number of failsafe mechanisms (like the ones you described) would be a critical part of any autonomous system, especially in the early stages, which would, of course, affect who we end up blaming.