7 ms·

Killing Robots and Just War Theory

45 points by behindai 5y ago

roenxi 5y ago
A bit more scary is the political consequences. One of the strongest controls on rogue generals is that a fair chunk of their troops will mutiny if told to destroy their home country for said general's benefit, and logistical support would vanish. Armies of killer robots start radically changing the calculations. It isn't obvious how but it probably removes a lot of the interest of more ordinary people and makes it a question of what the elites want. If the programmers are on side and the factories are mostly automatic, then the ability of people to push back on their own military becomes complicated.
- k__ 5y ago
  How is this different from nuclear weapons?
  - roenxi 5y ago
    If a general lives in the penthouse at the top of a skyscraper, and maybe has a few thousand people who support him, nukes are not going to help deal with revolutionaries on the bottom floor. They are too indiscriminate and can't be used against close up targets. Whereas a general with a drone swarm and a few thousand supporters might be able to suppress a city. Fortunately a little far fetched, but what was impossible with nukes is only impractical with drones.
    - throw1234651234 5y ago
      This is well put. There is also the idea that as armies got MI-24s/AH-64s, etc, opposing forces got Stingers/Iglas. Perhaps it will play out in a way where once armies get drones, OpFor gets counters - jamming / hacking / EM / whatever.
  - MisterTea 5y ago
    The land to be ruled by said rogue general will not be a radioactive wasteland.
  - raclage 5y ago
    Sufficiently advanced robots would leave a lot more of value behind than a nuke which makes deploying them much less costly to the person that wants to actually control the territory.
  - behindai 5y ago
    Seeing AI weapons as a weapon of mass destruction is a trending discourse.
  - wcarss 5y ago
    It's a much harder weapon to control via regulation and treaties than nuclear weapons: there is no nuclear material required to hook guns up to computers en masse in factories, let alone improvisationally. Once this cat is really out of the bag, everyone who wants it will have it.
    - est31 5y ago
      Manufacturing chips is an extremely complicated process and requires thousands of specialists from multiple different disciplines to cooperate. Building a new fab costs billions and the buildings are extremely sensitive for damage. I think one of the reasons why the west wants to block mainland China from manufacturing chips is to prevent them from building a robot army.
      - wcarss 5y ago
        It's true that building the chips is hard, but making it hard to buy general purpose small computers, or digital camera sensors suitable for vision, or rotor blades, or batteries, or antennas, is also going to be very difficult. Proliferation has already happened. Maybe cheap LIDAR will be something we can keep a lid on? But regular vision is still likely pretty good for lining up a shot or finding a face.
        cabalamat 5y ago
        > but making it hard to buy general purpose small computers [...] is also going to be very difficult Just ban general purpose computers that allow running arbitrary unsigned code. I'm sure lots of governments and corporations would be happy to get on board that one, and many end-users wouldn't notice.
        kian 5y ago
        A cure worse than the disease. Also wouldn't stop the military from building up a drone army and then someone within it choosing to go rogue and take it with them. It only ensures that no one else can stop them.
  - crznp 5y ago
    The world made it through the cold war without major disaster, but it is easy to imagine history taking a different path, so I don't think that this is a comforting comparison. Also, nuclear weapons can't be used 'a little bit' but killer robots can be -- it is much less clear where we draw the line. It seems likely that their use will be normalized/entrenched before we figure that out. Though I hope that you're right.
- sandworm101 5y ago
  >> that a fair chunk of their troops will mutiny if told to destroy their home country I'd be careful about this. There is a general principle in military communities that troops are not to obey "illegal orders" but time and time again throughout history we have seen that backstop fail. At a base level we all do what we think is best for ourselves. Troops will shoot prisoners. Troops will shoot protestors. They will nuke their own hometowns if they believe that is either necessary or if failure to do so will result in their own demise. And many troops, people, will do what they are told out of pure loyalty to a charismatic leader. The only real safeguard is layer upon layer of checks to ensure no one leader can ever exercise independent control.
  - wcarss 5y ago
    It's a fair point that humans don't magically make this go away, but the thrust of the argument above is that robots are likely worse, because they lack any capacity at all for questioning or for remorse, and they also allow great physical detachment from the horrors they can cause. They don't stop to eat or sleep either, and they'll certainly do things that can cause their own destruction. Humans also may be even likelier to be loyal than moral if a drone with a gun follows them around to keep them on-side.
  - est31 5y ago
    Humans have a natural aversion to killing, and yes, during war mentioned atrocities do happen, but the soldiers are not perfect killing machines. Often they miss intentionally and just shoot in the general vicinity of the opponent. See this reddit thread: https://old.reddit.com/r/history/comments/b6k528/percentage_of_soldier_who_purposely_missed_or/ https://old.reddit.com/r/history/comments/b6k528/percentage_... Troops can also desert which is especially prevalent in civil war situations. That's why during war, you need propagandists coming up with constant stories about how horrible and bad the other side is, so that you get people to shoot. The unit that so brutally killed the tianmen protesters was one of the most brutal ones the Chinese military had. That's why they had to wait for days for it to arrive. Had they deployed different units, they might have fraternized with the protesters and joined them in their protest, making the task even harder for the leadership. Try getting even the most brutal unit to shoot at people in their own uniform. For military parades, you can't afford 5% of people who intentionally don't march in lock step. That's why usually the troops which participate in such parades are isolated and fed propaganda leading up to the parade. The human component is a constant variable that needs to be accounted for in military leadership.
- SerLava 5y ago
  It just has to be a war crime to use these. A domestic army of robots is really just a foreign army on contract. They have zero loyalty to anyone who doesn't directly control them.
- cabalamat 5y ago
  This is true. Killer robots push societies in the direction of being more hierarchial, for the reasons you state.
  - abecedarius 5y ago
    This generalizes to automation of nonlethal means of population control -- crowd control weapons, surveillance and censorship. On the margin it'd hold for partial automation short of AGI -- reducing the needed size of the coalition on top.
- Gravityloss 5y ago
  This works better if a large enough portion of the soldiers are conscripts (possibly rotated in and out) and not career soldiers. The conscripts know they will go back to their normal life, day job and be part of the civilian society again. If it's just career soldiers, then it's in their career interest to enlarge the military's influence on society, increase budgets etc.
- JumpCrisscross 5y ago
  > troops will mutiny if told to destroy their home country for said general's benefit I’d generalise this check to all leaders. No man rules alone. Power only exists in its relationship with people. Robotic armies permit force projection without any people. That changes the political calculus in all sections of society, from military leadership to billionaires to members of a terrorist group.
- shadofx 5y ago
  One good zero-day could give total military and industrial control of multiple superpower nations to a single talented sociopath
leashless 5y ago
"The Second Amendment in Iraq, Combat Robotics, and the Future of Human Liberty" argues that combat robots are unconstitutional for the US to develop, because they render the second amendment useless. http://vinay.howtolivewiki.com/blog/bigdeal/the-second-amendment-in-iraq-combat-robotics-and-the-future-of-human-liberty-820 http://vinay.howtolivewiki.com/blog/bigdeal/the-second-amend...
- merpnderp 5y ago
  That's one heck of a stretch. The more reasonable interpretation would be that the 2nd simply means combat robots should be legal for citizens to own and operate. I mean who wouldn't sleep better at night knowing your XJ9000 Eviscerator brand murderbot is patrolling your home for unexpected guests? /s
  - krapp 5y ago
    The only solution to a bad robot with a gun is a good robot with a gun.
- oytis 5y ago
  It is already pretty useless, because militia is not going to have tanks, bombers, fighter jets and nukes anyway. It was an alright idea when it was introduced, but is mostly obsolete today.
  - Nbox9 5y ago
    Vietnam and Iraq are counter examples to this, they didn’t fight with tanks, bombers, fighter jets or nukes.
    - WFHRenaissance 5y ago
      Ding ding ding! Insurgencies that combine small arms, improvised explosives, clandestine cell social structures, fault-tolerant and encrypted communication methods, and distributed fundraising techniques can and have countered imperial military adventurism. This is not a libertarian fantasy; it happens every day.
thedudeabides5 5y ago
Ok, so if the robot commits war crimes, who is responsible? The designer? The soldier who turned it on? The general who deployed a couple thousand to the urban center before they had the “bug”
- pjc50 5y ago
  Realistically, if you win the war, or even if you're a western power that quietly retreats after losing it, there's nobody responsible. And if you lose the war the winner decides who was responsible for your killer robot army. There's Nuremberg precedent for "both".
  - monkeybutton 5y ago
    If you're the designer on the losing side you might get lucky and the winner will 'look to other way' of your crimes in exchange for your future services. Also a precedent from WW2.
    - stared 5y ago
      Engineers and scientists are generally considered a "resource". The winning power takes them, if needed. Scientists are generally not considered a threat to power (unlike high-ranked officials). For example, Piotr Kapitsa had his arguments with Lavrentiy Beria and Stalin. Instead of death (delivered for much smaller insubordination) he was gifted a rifle: "Beria summoned Kapitsa who amazingly refused him: “If you want to speak to me, then come to the Institute.” Beria ate humble pie and took a hunting rifle as a peace offering. But Kapitsa refused to help anymore."
- x86_64Ubuntu 5y ago
  We already have systems like that, whereby the accountability is intentionally so diffuse that no one can realistically be held accountable. Was it the programmers? The people who provided the programmers the specs? The military who set the kill parameters?
  - 5y ago
jxcole 5y ago
>> could not be understood by people and thus programmers are detached from the decisions making by machines. This isn't really true. A programmer may not understand the exact mechanism of computation but they will have trained it in a particular way using particular training data to do a particular purpose. That purpose is known to the programmer so this fact actually doesn't change anything. It is possible the robot will occasionally act in an unexpected way, but the majority of the time it will follow it's programmed objective, even if following that objective unpredictably.

45 points by behindai 5y ago

A bit more scary is the political consequences. One of the strongest controls on rogue generals is that a fair chunk of their troops will mutiny if told to destroy their home country for said general's benefit, and logistical support would vanish. Armies of killer robots start radically changing the calculations. It isn't obvious how but it probably removes a lot of the interest of more ordinary people and makes it a question of what the elites want. If the programmers are on side and the factories are mostly automatic, then the ability of people to push back on their own military becomes complicated.

k__ 5y ago
How is this different from nuclear weapons?
- roenxi 5y ago
  If a general lives in the penthouse at the top of a skyscraper, and maybe has a few thousand people who support him, nukes are not going to help deal with revolutionaries on the bottom floor. They are too indiscriminate and can't be used against close up targets. Whereas a general with a drone swarm and a few thousand supporters might be able to suppress a city. Fortunately a little far fetched, but what was impossible with nukes is only impractical with drones.
  - throw1234651234 5y ago
    This is well put. There is also the idea that as armies got MI-24s/AH-64s, etc, opposing forces got Stingers/Iglas. Perhaps it will play out in a way where once armies get drones, OpFor gets counters - jamming / hacking / EM / whatever.
- MisterTea 5y ago
  The land to be ruled by said rogue general will not be a radioactive wasteland.
- raclage 5y ago
  Sufficiently advanced robots would leave a lot more of value behind than a nuke which makes deploying them much less costly to the person that wants to actually control the territory.
- behindai 5y ago
  Seeing AI weapons as a weapon of mass destruction is a trending discourse.
- wcarss 5y ago
  It's a much harder weapon to control via regulation and treaties than nuclear weapons: there is no nuclear material required to hook guns up to computers en masse in factories, let alone improvisationally. Once this cat is really out of the bag, everyone who wants it will have it.
  - est31 5y ago
    Manufacturing chips is an extremely complicated process and requires thousands of specialists from multiple different disciplines to cooperate. Building a new fab costs billions and the buildings are extremely sensitive for damage. I think one of the reasons why the west wants to block mainland China from manufacturing chips is to prevent them from building a robot army.
    - wcarss 5y ago
      It's true that building the chips is hard, but making it hard to buy general purpose small computers, or digital camera sensors suitable for vision, or rotor blades, or batteries, or antennas, is also going to be very difficult. Proliferation has already happened. Maybe cheap LIDAR will be something we can keep a lid on? But regular vision is still likely pretty good for lining up a shot or finding a face.
      - cabalamat 5y ago
        > but making it hard to buy general purpose small computers [...] is also going to be very difficult Just ban general purpose computers that allow running arbitrary unsigned code. I'm sure lots of governments and corporations would be happy to get on board that one, and many end-users wouldn't notice.
        kian 5y ago
        A cure worse than the disease. Also wouldn't stop the military from building up a drone army and then someone within it choosing to go rogue and take it with them. It only ensures that no one else can stop them.
- crznp 5y ago
  The world made it through the cold war without major disaster, but it is easy to imagine history taking a different path, so I don't think that this is a comforting comparison. Also, nuclear weapons can't be used 'a little bit' but killer robots can be -- it is much less clear where we draw the line. It seems likely that their use will be normalized/entrenched before we figure that out. Though I hope that you're right.
sandworm101 5y ago
>> that a fair chunk of their troops will mutiny if told to destroy their home country I'd be careful about this. There is a general principle in military communities that troops are not to obey "illegal orders" but time and time again throughout history we have seen that backstop fail. At a base level we all do what we think is best for ourselves. Troops will shoot prisoners. Troops will shoot protestors. They will nuke their own hometowns if they believe that is either necessary or if failure to do so will result in their own demise. And many troops, people, will do what they are told out of pure loyalty to a charismatic leader. The only real safeguard is layer upon layer of checks to ensure no one leader can ever exercise independent control.
- wcarss 5y ago
  It's a fair point that humans don't magically make this go away, but the thrust of the argument above is that robots are likely worse, because they lack any capacity at all for questioning or for remorse, and they also allow great physical detachment from the horrors they can cause. They don't stop to eat or sleep either, and they'll certainly do things that can cause their own destruction. Humans also may be even likelier to be loyal than moral if a drone with a gun follows them around to keep them on-side.
- est31 5y ago
  Humans have a natural aversion to killing, and yes, during war mentioned atrocities do happen, but the soldiers are not perfect killing machines. Often they miss intentionally and just shoot in the general vicinity of the opponent. See this reddit thread: https://old.reddit.com/r/history/comments/b6k528/percentage_of_soldier_who_purposely_missed_or/ https://old.reddit.com/r/history/comments/b6k528/percentage_... Troops can also desert which is especially prevalent in civil war situations. That's why during war, you need propagandists coming up with constant stories about how horrible and bad the other side is, so that you get people to shoot. The unit that so brutally killed the tianmen protesters was one of the most brutal ones the Chinese military had. That's why they had to wait for days for it to arrive. Had they deployed different units, they might have fraternized with the protesters and joined them in their protest, making the task even harder for the leadership. Try getting even the most brutal unit to shoot at people in their own uniform. For military parades, you can't afford 5% of people who intentionally don't march in lock step. That's why usually the troops which participate in such parades are isolated and fed propaganda leading up to the parade. The human component is a constant variable that needs to be accounted for in military leadership.
SerLava 5y ago
It just has to be a war crime to use these. A domestic army of robots is really just a foreign army on contract. They have zero loyalty to anyone who doesn't directly control them.
cabalamat 5y ago
This is true. Killer robots push societies in the direction of being more hierarchial, for the reasons you state.
- abecedarius 5y ago
  This generalizes to automation of nonlethal means of population control -- crowd control weapons, surveillance and censorship. On the margin it'd hold for partial automation short of AGI -- reducing the needed size of the coalition on top.
Gravityloss 5y ago
This works better if a large enough portion of the soldiers are conscripts (possibly rotated in and out) and not career soldiers. The conscripts know they will go back to their normal life, day job and be part of the civilian society again. If it's just career soldiers, then it's in their career interest to enlarge the military's influence on society, increase budgets etc.
JumpCrisscross 5y ago
> troops will mutiny if told to destroy their home country for said general's benefit I’d generalise this check to all leaders. No man rules alone. Power only exists in its relationship with people. Robotic armies permit force projection without any people. That changes the political calculus in all sections of society, from military leadership to billionaires to members of a terrorist group.
shadofx 5y ago
One good zero-day could give total military and industrial control of multiple superpower nations to a single talented sociopath

leashless 5y ago

"The Second Amendment in Iraq, Combat Robotics, and the Future of Human Liberty" argues that combat robots are unconstitutional for the US to develop, because they render the second amendment useless. http://vinay.howtolivewiki.com/blog/bigdeal/the-second-amendment-in-iraq-combat-robotics-and-the-future-of-human-liberty-820 http://vinay.howtolivewiki.com/blog/bigdeal/the-second-amend...

merpnderp 5y ago
That's one heck of a stretch. The more reasonable interpretation would be that the 2nd simply means combat robots should be legal for citizens to own and operate. I mean who wouldn't sleep better at night knowing your XJ9000 Eviscerator brand murderbot is patrolling your home for unexpected guests? /s
- krapp 5y ago
  The only solution to a bad robot with a gun is a good robot with a gun.
oytis 5y ago
It is already pretty useless, because militia is not going to have tanks, bombers, fighter jets and nukes anyway. It was an alright idea when it was introduced, but is mostly obsolete today.
- Nbox9 5y ago
  Vietnam and Iraq are counter examples to this, they didn’t fight with tanks, bombers, fighter jets or nukes.
  - WFHRenaissance 5y ago
    Ding ding ding! Insurgencies that combine small arms, improvised explosives, clandestine cell social structures, fault-tolerant and encrypted communication methods, and distributed fundraising techniques can and have countered imperial military adventurism. This is not a libertarian fantasy; it happens every day.

thedudeabides5 5y ago

Ok, so if the robot commits war crimes, who is responsible? The designer? The soldier who turned it on? The general who deployed a couple thousand to the urban center before they had the “bug”

pjc50 5y ago
Realistically, if you win the war, or even if you're a western power that quietly retreats after losing it, there's nobody responsible. And if you lose the war the winner decides who was responsible for your killer robot army. There's Nuremberg precedent for "both".
- monkeybutton 5y ago
  If you're the designer on the losing side you might get lucky and the winner will 'look to other way' of your crimes in exchange for your future services. Also a precedent from WW2.
  - stared 5y ago
    Engineers and scientists are generally considered a "resource". The winning power takes them, if needed. Scientists are generally not considered a threat to power (unlike high-ranked officials). For example, Piotr Kapitsa had his arguments with Lavrentiy Beria and Stalin. Instead of death (delivered for much smaller insubordination) he was gifted a rifle: "Beria summoned Kapitsa who amazingly refused him: “If you want to speak to me, then come to the Institute.” Beria ate humble pie and took a hunting rifle as a peace offering. But Kapitsa refused to help anymore."
x86_64Ubuntu 5y ago
We already have systems like that, whereby the accountability is intentionally so diffuse that no one can realistically be held accountable. Was it the programmers? The people who provided the programmers the specs? The military who set the kill parameters?
- 5y ago

jxcole 5y ago

>> could not be understood by people and thus programmers are detached from the decisions making by machines. This isn't really true. A programmer may not understand the exact mechanism of computation but they will have trained it in a particular way using particular training data to do a particular purpose. That purpose is known to the programmer so this fact actually doesn't change anything. It is possible the robot will occasionally act in an unexpected way, but the majority of the time it will follow it's programmed objective, even if following that objective unpredictably.