5 ms·

If an autonomous machine kills someone, who is responsible?

4 points by muon 17y ago

yannis 17y ago
The autonomous machine should be brought to court and be allowed to defend itself before any judgement is passed. They should be allowed access to legal aid even a pro deo counsel. It is not impossible actually for a legal system - that over time has passed judgement to witches, dogs donkeys, used torture to extract confessions and in some parts of the world demands its high priests to wear funny clothes and wigs - to be modified to suit the new circumstances. In the meantime if you developing software for automating trains, aircraft or heavy construction machinery get plenty of Professional Idemnity Insurance. If you developing software for drones, missiles or nukes don't worry, there will be nobody alive to sue you if something goes wrong!
- roundsquare 17y ago
  I don't think autonomous here is the same as sentient/intelligent/etc... It just means a machine that does something without human intervention. E.g. if a Rumba takes off your toe, you can't take it to court.
anigbrowl 17y ago
I don't know, but expect to see Hofstadter and Searle debate by proxy in a courtroom, because Searle's Chinese Room argument is exactly the sort of thing you could put over on a typical jury.
- stijnm 17y ago
  I had not heard of this before, but very interesting! For those wanting more info: http://en.wikipedia.org/wiki/Chinese_room http://en.wikipedia.org/wiki/Chinese_room
  - anigbrowl 17y ago
    If you found it interesting then you really need to read http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach ...and indeed, pretty much everything else he has written. It seems to me that the long slow debate between these two philophers (add in Daniel Dennett on Hofstadter's side) turns on the fundamental philosophical question of where consciousness begins and ends, which has considerable ramifications beyond the field of artificial intelligence. I reject Searle's argument for reasons too long to go into here, but he is a very interesting philosopher in his own right. While I'm on this topic, I may as well also mention Stanislaw Lem's essay Non Serviam excerpted here (http://themindi.blogspot.com/2007/02/chapter-19-non-serviam.html http://themindi.blogspot.com/2007/02/chapter-19-non-serviam....) from A Perfect Vacuum: Perfect reviews of nonexistent books. Much, perhaps most, of Lem's output is concerned with the nature of consciousness and autonomy explored through the medium of fiction, just as Philip K dick tends to explore the same questions from a psychological standpoint.
stijnm 17y ago
I don't want to forego a stimulating intelectual discourse, but isn't it obvious? Replace 'autonomous machine' with 'child' and it is clear a parent is responsible. For autonomous machines these are the owners. Machines may be autonomous, but if a real person is liable, then I am sure there will be plenty of checks and tests done to prevent any disasters. In fact, the results of the accounting scandals not so long ago may be a good template. Investors were fully dependent on this autonomous, faceless, financial machine to churn out profits. That blew up in their face and they had no-one to blame. One result was the enactment of Sarbanes-Oxley which set, amongst other things, criminal penalties on individuals. That is good motivation: accountability.
- eru 17y ago
  Interesting. However you forget one thing: Parents are not liable for damage their children do. E.g. in Germany as long as parents do not violate the obligatory supervision neither can they nor their children be made liable for any acts of their children.
  - stijnm 17y ago
    Your example implicitly means that you do agree someone needs to be made liable for the child through 'obligatory supervision' - because if you don't do that, you will be accountable. You say 'as long as parents do not violate'. Indeed, as long as you do what is required to ensure nothing goes wrong ("bonus pater familias"), but an accident still happens, then society/German Law accepts no one is liable. Your argument seems to point more to: How do you define someone as liable and how do you enforce it?
    - roundsquare 17y ago
      If we take the case of the robot surgeon, then the person/institution who made the decision to use the robot would presumably be responsible (e.g. in this case, the hospital). The 'obligatory supervision' in this case would mean running regular tests, getting it repaired, etc... I think in most cases we can come up with someone who is responsible for the robot. The person who owns it will generally be the person responsible but there will be exceptions (e.g. if I borrow your robot, break the "don't attack people" software and then use it). If, after doing all the maintenance, the robot surgeon still kills someone, then you would need to investigate. Did something go wrong with the robot? If so, is this a defect in design that the manufacturers should have thought of? If so, blame them. If not, well, no one is to blame (unless it happens with the next model... where they should have learned from the past). I suspect we'd also end up seeing recalls of robots when flaws are detected, as we do with cars occasionally.
gdp 17y ago
Software defects kill people on an infrequent but fairly regular basis. Legal precedent and licensing agreements suggest that this is the software user's fault, not the manufacturer's. So I assume we'll carry on with this preposterous state of affairs and just extend it to software-in-machines killing people autonomously, 'cause a defect in that still wouldn't be the manufacturer's fault, right?
- patio11 17y ago
  We also have fairly well-developed laws and case law on animal attacks. If you keep a chihuahua in a jurisdiction which does not have strict liability, and it bites someone, you are likely not at fault unless you had reason to know it was prone to violence AND you took insufficient steps to restrain it. Your local jurisdiction may have strict liability ("It is the owner's fault, period") or may have designated certain breeds as "aggressive dogs" whose owners are presumptively liable. You could see this happening with the law of machines, too. Roomba crushes someone? Awful but lawful. KillBot9000 mistakes a troop of Girl Scouts for armed revolutionaries? You're probably going to end up liable.
estacado 17y ago
God.
akd 17y ago
This question exists because humans demand a cause for each effect, and a "responsible party" for every tragedy. Sometimes, things are just "accidents." A colleague of my dad's tripped and fell while walking down a set of three stairs, hit his head, and died. It was nobody's fault -- it was just bad luck.
- gdp 17y ago
  But that ignores the idea that somebody designs and builds an autonomous machine. If an airplane autopilot sends a plane full of people into a mountain due to a defect in the software, who is liable?
  - sokoloff 17y ago
    The FAA (and common sense) treats an autopilot as just a fancy interface to the actual control surfaces of the aircraft, meaning the pilot-in-command is still responsible. Even on autopilot, I'm supposed to be monitoring the systems, keeping track of where we are, where we're headed, the weather, etc. If my autopilot sends me towards a mountain, it's on me (practically and legally) to fix it.
    - gdp 17y ago
      Which seems to provide a possibly workable model for automated systems of other kinds.
      - gloob 17y ago
        Precisely: we'll hold the pilot of the autonomous drone responsible. Edit: Actually, this raises a question in my mind. Do autonomous military robots have "handlers" or the like?
    - roundsquare 17y ago
      Sure, but if you see some bad weather and decide to turn off autopilot... and the system won't let you, then the company is to blame (legally, though practically you'd obviously do everything you can to make sure you survive). The decision would need to be made case by case in courts to determine who (if anyone) is responsible. At least in court systems like the US has. For very different systems, I'm not sure.

4 points by muon 17y ago

The autonomous machine should be brought to court and be allowed to defend itself before any judgement is passed. They should be allowed access to legal aid even a pro deo counsel. It is not impossible actually for a legal system - that over time has passed judgement to witches, dogs donkeys, used torture to extract confessions and in some parts of the world demands its high priests to wear funny clothes and wigs - to be modified to suit the new circumstances. In the meantime if you developing software for automating trains, aircraft or heavy construction machinery get plenty of Professional Idemnity Insurance. If you developing software for drones, missiles or nukes don't worry, there will be nobody alive to sue you if something goes wrong!

roundsquare 17y ago
I don't think autonomous here is the same as sentient/intelligent/etc... It just means a machine that does something without human intervention. E.g. if a Rumba takes off your toe, you can't take it to court.

anigbrowl 17y ago

I don't know, but expect to see Hofstadter and Searle debate by proxy in a courtroom, because Searle's Chinese Room argument is exactly the sort of thing you could put over on a typical jury.

stijnm 17y ago
I had not heard of this before, but very interesting! For those wanting more info: http://en.wikipedia.org/wiki/Chinese_room http://en.wikipedia.org/wiki/Chinese_room
- anigbrowl 17y ago
  If you found it interesting then you really need to read http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach http://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach ...and indeed, pretty much everything else he has written. It seems to me that the long slow debate between these two philophers (add in Daniel Dennett on Hofstadter's side) turns on the fundamental philosophical question of where consciousness begins and ends, which has considerable ramifications beyond the field of artificial intelligence. I reject Searle's argument for reasons too long to go into here, but he is a very interesting philosopher in his own right. While I'm on this topic, I may as well also mention Stanislaw Lem's essay Non Serviam excerpted here (http://themindi.blogspot.com/2007/02/chapter-19-non-serviam.html http://themindi.blogspot.com/2007/02/chapter-19-non-serviam....) from A Perfect Vacuum: Perfect reviews of nonexistent books. Much, perhaps most, of Lem's output is concerned with the nature of consciousness and autonomy explored through the medium of fiction, just as Philip K dick tends to explore the same questions from a psychological standpoint.

stijnm 17y ago

I don't want to forego a stimulating intelectual discourse, but isn't it obvious? Replace 'autonomous machine' with 'child' and it is clear a parent is responsible. For autonomous machines these are the owners. Machines may be autonomous, but if a real person is liable, then I am sure there will be plenty of checks and tests done to prevent any disasters. In fact, the results of the accounting scandals not so long ago may be a good template. Investors were fully dependent on this autonomous, faceless, financial machine to churn out profits. That blew up in their face and they had no-one to blame. One result was the enactment of Sarbanes-Oxley which set, amongst other things, criminal penalties on individuals. That is good motivation: accountability.

eru 17y ago
Interesting. However you forget one thing: Parents are not liable for damage their children do. E.g. in Germany as long as parents do not violate the obligatory supervision neither can they nor their children be made liable for any acts of their children.
- stijnm 17y ago
  Your example implicitly means that you do agree someone needs to be made liable for the child through 'obligatory supervision' - because if you don't do that, you will be accountable. You say 'as long as parents do not violate'. Indeed, as long as you do what is required to ensure nothing goes wrong ("bonus pater familias"), but an accident still happens, then society/German Law accepts no one is liable. Your argument seems to point more to: How do you define someone as liable and how do you enforce it?
  - roundsquare 17y ago
    If we take the case of the robot surgeon, then the person/institution who made the decision to use the robot would presumably be responsible (e.g. in this case, the hospital). The 'obligatory supervision' in this case would mean running regular tests, getting it repaired, etc... I think in most cases we can come up with someone who is responsible for the robot. The person who owns it will generally be the person responsible but there will be exceptions (e.g. if I borrow your robot, break the "don't attack people" software and then use it). If, after doing all the maintenance, the robot surgeon still kills someone, then you would need to investigate. Did something go wrong with the robot? If so, is this a defect in design that the manufacturers should have thought of? If so, blame them. If not, well, no one is to blame (unless it happens with the next model... where they should have learned from the past). I suspect we'd also end up seeing recalls of robots when flaws are detected, as we do with cars occasionally.

gdp 17y ago

Software defects kill people on an infrequent but fairly regular basis. Legal precedent and licensing agreements suggest that this is the software user's fault, not the manufacturer's. So I assume we'll carry on with this preposterous state of affairs and just extend it to software-in-machines killing people autonomously, 'cause a defect in that still wouldn't be the manufacturer's fault, right?

patio11 17y ago
We also have fairly well-developed laws and case law on animal attacks. If you keep a chihuahua in a jurisdiction which does not have strict liability, and it bites someone, you are likely not at fault unless you had reason to know it was prone to violence AND you took insufficient steps to restrain it. Your local jurisdiction may have strict liability ("It is the owner's fault, period") or may have designated certain breeds as "aggressive dogs" whose owners are presumptively liable. You could see this happening with the law of machines, too. Roomba crushes someone? Awful but lawful. KillBot9000 mistakes a troop of Girl Scouts for armed revolutionaries? You're probably going to end up liable.

estacado 17y ago

God.

akd 17y ago

This question exists because humans demand a cause for each effect, and a "responsible party" for every tragedy. Sometimes, things are just "accidents." A colleague of my dad's tripped and fell while walking down a set of three stairs, hit his head, and died. It was nobody's fault -- it was just bad luck.

gdp 17y ago
But that ignores the idea that somebody designs and builds an autonomous machine. If an airplane autopilot sends a plane full of people into a mountain due to a defect in the software, who is liable?
- sokoloff 17y ago
  The FAA (and common sense) treats an autopilot as just a fancy interface to the actual control surfaces of the aircraft, meaning the pilot-in-command is still responsible. Even on autopilot, I'm supposed to be monitoring the systems, keeping track of where we are, where we're headed, the weather, etc. If my autopilot sends me towards a mountain, it's on me (practically and legally) to fix it.
  - gdp 17y ago
    Which seems to provide a possibly workable model for automated systems of other kinds.
    - gloob 17y ago
      Precisely: we'll hold the pilot of the autonomous drone responsible. Edit: Actually, this raises a question in my mind. Do autonomous military robots have "handlers" or the like?
  - roundsquare 17y ago
    Sure, but if you see some bad weather and decide to turn off autopilot... and the system won't let you, then the company is to blame (legally, though practically you'd obviously do everything you can to make sure you survive). The decision would need to be made case by case in courts to determine who (if anyone) is responsible. At least in court systems like the US has. For very different systems, I'm not sure.