6 ms·

Autonomous Weapons: The False Promise of Civilian Protection (2022)

wolverine876 3y ago
> Advanced militaries have clearly indicated their interest in such systems. Reasons range from speed of response and enhanced situational awareness to the ability to overwhelm the defence systems of adversaries. My understanding of experts I've read over years: Western militaries, at least, have for over a decade worked on keeping humans 'in the loop', for the obvious moral reasons and also for effectiveness - human-machine teaming is considered much more effective than either humans or machines alone. Yet they are developing autonomous machines because they don't believe they can rely on communications. The measure-countermeasure contest between communication and jamming leans too much toward the latter and it becomes a serious vulnerability: The machine army could be entirely shut down by simply jamming communications.
- lettergram 3y ago
  Imo there’s a lot of rationalization. I worked on an autonomous quad copter learning to fly in a simulated environment back in 2013-2014. It would then target and track a target following it. We then moved the AI from a simulation to reality and it worked very well. I was an undergraduate at university at the time. Based on the training data we had (what looked a lot like predictor drone video), I imagine the tech is fully capable and ready to go as needed. Everyone knew what we were building, but rationalized it as “oh this will be used for speeding tickets or tracking drug deals”… like sure, but I can see the AKs in the video lol
  - m463 3y ago
    > speeding tickets > drug deals I can't help but think of how the world changes. Think of speeding tickets, until the 70's, vast portions of the US did not have a speed limit. This changed during the middle east oil embargo with the introduction of the national 55mph speed limit (under the guise of saving energy). When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them. I wonder if the tolerance value for going after a speeding car might become smaller as self-driving cards minimize offenses and ticket revenue drops. In the same way, "drug deals" have led to civil asset forfeiture, causing lots of out-of-town travelers to give up their property, to the benefit of police departments. so... today's "speeding tickets" and "drug deals" might become tomorrow's problem for a wider cross-section of the population.
    - voxelghost 3y ago
      >Think of speeding tickets, until the 70's, vast portions of the US did not have a speed limit. This changed during the middle east oil embargo with the introduction of the national 55mph speed limit (under the guise of saving energy). They did not have a federal speed limit. State or local speed limits were imposed everywhere. Although pre-1973 limits were often a lot higher than the federal limit. >When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them. The 1973 law was repealed in 1995. And the mandate of setting speed-limits were returned to state-level
      - timschmidt 3y ago
        Nevada and Montana didn't have posted speed limits on many or any highways, respectively, until 1999.
        voxelghost 3y ago
        Out of my knowledge area, but according to ChatGPT; >[Well..], both states did have speed limits, but they had sections of rural highways with "reasonable and prudent" speed limits, which were often unposted. >But they were not without limitations. Drivers were still expected to drive at a safe and appropriate speed for the conditions, and law enforcement could *enforce speed limits based on what they deemed reasonable and prudent.* So to me than seems even more open to abuse than the current situation.
    - ryandrake 3y ago
      > I wonder if the tolerance value for going after a speeding car might become smaller as self-driving cards minimize offenses and ticket revenue drops. In a future world where every car is self-driving and obeys the posted speed limit, I can see states and municipalities resorting to trickery and “hacks” like declaring a speed limit that changes randomly and unpredictably, so self-driving cars have trouble adapting and can get cited. It’s not like they are just going to allow that revenue stream to disappear.
  - chasd00 3y ago
    > It would then target and track a target following it Isn’t that just the standard “follow me” feature?
  - YeGoblynQueenne 3y ago
    >> We then moved the AI from a simulation to reality and it worked very well. I was an undergraduate at university at the time. There's tons of work like that (no offense meant) and it never amounts to anything much. If you used Deep RL or imitation learning, or really any of the Big Data, statistical machine learning techniques that are well known, then it wouldn't have worked in practice. Those approaches are extremely unreliable and fail to generalise to new environments robustly enough to be of use in the real world. The reason why the article above can still ask the question "Have we seen active deployment of killer robots?" is that the question doesn't have a clear answer. If what you did as an undergraduate could really work as well as some people would like, there would be no question left to anyone's mind. Again, that's not to bash your work. NVIDIA recently announced a new "moonshot" to develop humanoid robots for domestic use. That's because there's no such thing yet. Just a quick glance at the robotics and AI literature should be enough to convince anyone that true artificial autonomy is not yet possible.
    - lettergram 3y ago
      Eh I know what we worked on ended up being deployed in production systems. most of the team went on (not me) to found a company too: https://reconstructinc.com/ https://reconstructinc.com/
      - YeGoblynQueenne 3y ago
        That a system is deployed doesn't mean it does what it is said to do. Specifically if we're talking about self-guided missiles or loitering munitions, those are designed to be destroyed on impact and so the requirements for autonomy are much smaller. If you're talking about a gun drone that autonomously hunts down people and kills them, then that's not a technological capability currently available, at least not if you're interested in accuracy. Although, I guess that's the point, accuracy is often not a priority. Like early gun powder weapons part of the value of autonomous weapons is their potential for terror.
- nradov 3y ago
  Autonomous weapons systems are nothing new. The US Navy put the fully autonomous Mark 60 CAPTOR mine into service in 1979. It can be placed in a fixed location and wait months for a target based on a passive sonar signature, then launch a homing torpedo to attack. Zero humans in the loop.
  - wolverine876 3y ago
    I agree that the line between that and automomous weapons is hard to draw. A simpler example is any precision targeted missile or munition. At some point every munition leaves human control, including artillery and bullets. But is my 1979 calculator artificially intelligent because it can duplicate some cognitive things that I can do? There's a major qualitative difference where we find all the significance in the meaning of AI. Still, some of the problems with mines, which are illegal or highly discouraged in many situations, are similar to autonomous robotic weapon systems: A fundamental law of war is discrimination between military and civilian targets. Mines are literally indiscriminate, by that legal definition; the worry is that autonomous robotic weapons may be the same. Arguably, they could be more discriminate. I think one difference is in the 'robotic' aspect of autonomous robotic weapon systems (my terminology): Robots don't need to be mobile, of course, but those with significant mobility seem far riskier and deadlier than mines or artillery. Imagine one that, even from some malfunction, ends up in a city or a refugee camp.
  - GartzenDeHaes 3y ago
    As I recall, this type of mine (or a similar one) could target specific types of ships, or even a specific individual ship if enough data was available.
wolverine876 3y ago
There's another major issue with civilians and autonomous warfare machines of all sorts: Civilian control. For all of history, military power depended most of all on population size (to provide soldiers) and economic resources (to equip them). Mad wizards were limited by population. However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army. Also, note that these machines are often built for the US military by people who openly denegrate democracy and seem megalomaniacal; they have the technology and a lot of wealth. Could they build a factory somewhere and hide in their bunker until it's over?
- deleted 3y ago
  [deleted]
- nonrandomstring 3y ago
  > megalomaniacal persons with lust for power and disregard of others I think the problem with these sorts of people is that they are not actually that smart. Cognitively, as all-rounders, they lack crucial faculties of imagination, which not ironically are rooted in compassion, that lead to gross strategic errors. Landmines and impact grenades tied to a balloon are also "autonomous weapons". They are indiscriminate. The former are persistent and the later are ephemeral. The former are also reusable. Within a weapons taxonomy the most dangerous (for yourself) to manufacture are persistent, autonomous and reusable. In other words, like a knife in a fight, there's a very good likelihood your opponent will take it off you and use it against you. Most IEDs used against British and US soldiers are made from UXO. Weapons like drones are a gift to terrorists. For everything you make, one must imagine it being used on a busy city shopping street where your own family are. It's because the megalomaniacs are not actually that smart that they don't realise taking humans out of the loop is the same as relinquishing control, which is one step removed from handing control to the enemy.
  - MichaelZuo 3y ago
    This seems a bit redundant to be honest. Any maniac who is actually super competent would be able to fool you into thinking otherwise. Because they can outwit you by definition. And the inverse also is tautological, of course you can spot those not that competent.
deleted 3y ago
[deleted]
ChrisMarshallNY 3y ago
Whenever I hear "autonomous weapons," I think of this scene: https://www.youtube.com/watch?v=TYsulVXpgYg https://www.youtube.com/watch?v=TYsulVXpgYg
aaron695 3y ago
[dead]
tomohawk 3y ago
If you're interested in this sort of thing, Ryan Mcbeth has been talking about this recently. https://www.youtube.com/watch?v=0AUqYmCeZN4 https://www.youtube.com/watch?v=0AUqYmCeZN4
pvillano 3y ago
A landmine is an autonomous weapon. Landmines also kill a lot of civilians.
- andsoitis 3y ago
  > A landmine is an autonomous weapon. Landmines also kill a lot of civilians. "Anti-personnel landmines are prohibited under the Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction (or Mine Ban Convention), adopted in 1997. More than 150 countries have joined this treaty." - https://disarmament.unoda.org/convarms/landmines/ https://disarmament.unoda.org/convarms/landmines/
  - slaymaker1907 3y ago
    150 countries, but missing a lot of key ones like the US, Israel, Russia, and China. (They absolutely should sign this, though). I’m suspicious that all it will take is one major shock for the house of cards that is international law to come tumbling down just like it did before WW2.
    - cpgxiii 3y ago
      Definitely. Prohibitions on land mines and cluster weapons are pure political theater. Mines and cluster weapons don't exist to be cruel, they exist because they work, and the collateral damage caused by their persistence after a conflict is over is an unfortunate consequence. Essentially all of the countries that signed onto these bans either (1) did so in a moment of blinding post-cold war naivety, where they could spin downsizing their armed forces into a good thing for the world, or (2) are too small and/or uninvolved to matter. Anyone else who actually expected to need to fight a war again didn't sign, because they knew exactly how useful these weapons are. The US and SK will never accept a land mine prohibition that includes the minefields of the DMZ, since those minefields are an important speedbump against any sort of ground invasion. Likewise, a cluster munition ban is practically DoA because cluster munitions are the best weapons to attack columns of lightly/unarmored equipment and dispersed aircraft/helicopters. But until February 2022 in Ukraine, a whole bunch of countries (primarily European) had convinced themselves that the possibility of a ground invasion was a thing of the past.
beloch 3y ago
[flagged]
- adolph 3y ago
  It is interesting/odd that this conflict of all going on at the moment brings forth the examples of AI utilization. Why not Russia-Ukraine, or Myanmar, or any other conflict? What is the specific example of "AI" as opposed to "statistics" that is being used? It doesn't seem very AI to perform analysis of social graph to ID someone as a target and then tell your autonomous surveillance platform to watch that person's addresses. https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflicts https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflict...
  - beloch 3y ago
    First, Gaza really is an ideal testing ground for this sort of AI targeting system. It's tiny. Pretty much all of it is under continual surveillance. Israel has completely modern precision bombing capabilities. By comparison, just the front in the Ukraine-Russia war is massive. Many of the targets are nowhere near the front. The surveillance and precision air strike capabilities of both sides are inferior to those of the IDF. Yet, we do see a lot of really savvy strikes being made on the basis of cell-phone data, etc.. This kind of intelligence could have been gathered by humans, but it's not impossible that either or both sides are already using AI targeting systems similar to Lavender. They're likely just not as effective for the reasons mentioned above, and others. Militaries are likely not going to talk about this stuff much. Heck, this capability is in the same ballpark as things that past leaders may have sacrificed cities in order to hide[1]. Even the article on Lavender uses a lot of anonymous sources, so who really knows if it's true? Second, it may not seem "very AI" to identify targets and determine when they're home. Yes, humans could do this. It would take a significant amount of time though. If the reports are true, Lavender is able to do this so quickly that Israel can bomb the homes not just of leaders and generals, but of raw recruits who've probably received the sum total preparation of a rifle and a one-time pep-talk. Quantity has a quality all it's own. https://en.wikipedia.org/wiki/Coventry_Blitz#Coventry_and_Ultra https://en.wikipedia.org/wiki/Coventry_Blitz#Coventry_and_Ul...
- 3y ago
deleted 3y ago
[deleted]

> Advanced militaries have clearly indicated their interest in such systems. Reasons range from speed of response and enhanced situational awareness to the ability to overwhelm the defence systems of adversaries. My understanding of experts I've read over years: Western militaries, at least, have for over a decade worked on keeping humans 'in the loop', for the obvious moral reasons and also for effectiveness - human-machine teaming is considered much more effective than either humans or machines alone. Yet they are developing autonomous machines because they don't believe they can rely on communications. The measure-countermeasure contest between communication and jamming leans too much toward the latter and it becomes a serious vulnerability: The machine army could be entirely shut down by simply jamming communications.

lettergram 3y ago
Imo there’s a lot of rationalization. I worked on an autonomous quad copter learning to fly in a simulated environment back in 2013-2014. It would then target and track a target following it. We then moved the AI from a simulation to reality and it worked very well. I was an undergraduate at university at the time. Based on the training data we had (what looked a lot like predictor drone video), I imagine the tech is fully capable and ready to go as needed. Everyone knew what we were building, but rationalized it as “oh this will be used for speeding tickets or tracking drug deals”… like sure, but I can see the AKs in the video lol
- m463 3y ago
  > speeding tickets > drug deals I can't help but think of how the world changes. Think of speeding tickets, until the 70's, vast portions of the US did not have a speed limit. This changed during the middle east oil embargo with the introduction of the national 55mph speed limit (under the guise of saving energy). When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them. I wonder if the tolerance value for going after a speeding car might become smaller as self-driving cards minimize offenses and ticket revenue drops. In the same way, "drug deals" have led to civil asset forfeiture, causing lots of out-of-town travelers to give up their property, to the benefit of police departments. so... today's "speeding tickets" and "drug deals" might become tomorrow's problem for a wider cross-section of the population.
  - voxelghost 3y ago
    >Think of speeding tickets, until the 70's, vast portions of the US did not have a speed limit. This changed during the middle east oil embargo with the introduction of the national 55mph speed limit (under the guise of saving energy). They did not have a federal speed limit. State or local speed limits were imposed everywhere. Although pre-1973 limits were often a lot higher than the federal limit. >When oil supply stabilized and domestic and foreign oil became plentiful, it was not repealed. Now speeding tickets are common. A huge majority of people have gotten them. The 1973 law was repealed in 1995. And the mandate of setting speed-limits were returned to state-level
    - timschmidt 3y ago
      Nevada and Montana didn't have posted speed limits on many or any highways, respectively, until 1999.
      - voxelghost 3y ago
        Out of my knowledge area, but according to ChatGPT; >[Well..], both states did have speed limits, but they had sections of rural highways with "reasonable and prudent" speed limits, which were often unposted. >But they were not without limitations. Drivers were still expected to drive at a safe and appropriate speed for the conditions, and law enforcement could *enforce speed limits based on what they deemed reasonable and prudent.* So to me than seems even more open to abuse than the current situation.
  - ryandrake 3y ago
    > I wonder if the tolerance value for going after a speeding car might become smaller as self-driving cards minimize offenses and ticket revenue drops. In a future world where every car is self-driving and obeys the posted speed limit, I can see states and municipalities resorting to trickery and “hacks” like declaring a speed limit that changes randomly and unpredictably, so self-driving cars have trouble adapting and can get cited. It’s not like they are just going to allow that revenue stream to disappear.
- chasd00 3y ago
  > It would then target and track a target following it Isn’t that just the standard “follow me” feature?
- YeGoblynQueenne 3y ago
  >> We then moved the AI from a simulation to reality and it worked very well. I was an undergraduate at university at the time. There's tons of work like that (no offense meant) and it never amounts to anything much. If you used Deep RL or imitation learning, or really any of the Big Data, statistical machine learning techniques that are well known, then it wouldn't have worked in practice. Those approaches are extremely unreliable and fail to generalise to new environments robustly enough to be of use in the real world. The reason why the article above can still ask the question "Have we seen active deployment of killer robots?" is that the question doesn't have a clear answer. If what you did as an undergraduate could really work as well as some people would like, there would be no question left to anyone's mind. Again, that's not to bash your work. NVIDIA recently announced a new "moonshot" to develop humanoid robots for domestic use. That's because there's no such thing yet. Just a quick glance at the robotics and AI literature should be enough to convince anyone that true artificial autonomy is not yet possible.
  - lettergram 3y ago
    Eh I know what we worked on ended up being deployed in production systems. most of the team went on (not me) to found a company too: https://reconstructinc.com/ https://reconstructinc.com/
    - YeGoblynQueenne 3y ago
      That a system is deployed doesn't mean it does what it is said to do. Specifically if we're talking about self-guided missiles or loitering munitions, those are designed to be destroyed on impact and so the requirements for autonomy are much smaller. If you're talking about a gun drone that autonomously hunts down people and kills them, then that's not a technological capability currently available, at least not if you're interested in accuracy. Although, I guess that's the point, accuracy is often not a priority. Like early gun powder weapons part of the value of autonomous weapons is their potential for terror.
nradov 3y ago
Autonomous weapons systems are nothing new. The US Navy put the fully autonomous Mark 60 CAPTOR mine into service in 1979. It can be placed in a fixed location and wait months for a target based on a passive sonar signature, then launch a homing torpedo to attack. Zero humans in the loop.
- wolverine876 3y ago
  I agree that the line between that and automomous weapons is hard to draw. A simpler example is any precision targeted missile or munition. At some point every munition leaves human control, including artillery and bullets. But is my 1979 calculator artificially intelligent because it can duplicate some cognitive things that I can do? There's a major qualitative difference where we find all the significance in the meaning of AI. Still, some of the problems with mines, which are illegal or highly discouraged in many situations, are similar to autonomous robotic weapon systems: A fundamental law of war is discrimination between military and civilian targets. Mines are literally indiscriminate, by that legal definition; the worry is that autonomous robotic weapons may be the same. Arguably, they could be more discriminate. I think one difference is in the 'robotic' aspect of autonomous robotic weapon systems (my terminology): Robots don't need to be mobile, of course, but those with significant mobility seem far riskier and deadlier than mines or artillery. Imagine one that, even from some malfunction, ends up in a city or a refugee camp.
- GartzenDeHaes 3y ago
  As I recall, this type of mine (or a similar one) could target specific types of ships, or even a specific individual ship if enough data was available.

wolverine876 3y ago

There's another major issue with civilians and autonomous warfare machines of all sorts: Civilian control. For all of history, military power depended most of all on population size (to provide soldiers) and economic resources (to equip them). Mad wizards were limited by population. However, mad wizards - megalomaniacal persons with lust for power and disregard of others - with enough resources might manufacture a machine army. Also, note that these machines are often built for the US military by people who openly denegrate democracy and seem megalomaniacal; they have the technology and a lot of wealth. Could they build a factory somewhere and hide in their bunker until it's over?

deleted 3y ago
[deleted]
nonrandomstring 3y ago
> megalomaniacal persons with lust for power and disregard of others I think the problem with these sorts of people is that they are not actually that smart. Cognitively, as all-rounders, they lack crucial faculties of imagination, which not ironically are rooted in compassion, that lead to gross strategic errors. Landmines and impact grenades tied to a balloon are also "autonomous weapons". They are indiscriminate. The former are persistent and the later are ephemeral. The former are also reusable. Within a weapons taxonomy the most dangerous (for yourself) to manufacture are persistent, autonomous and reusable. In other words, like a knife in a fight, there's a very good likelihood your opponent will take it off you and use it against you. Most IEDs used against British and US soldiers are made from UXO. Weapons like drones are a gift to terrorists. For everything you make, one must imagine it being used on a busy city shopping street where your own family are. It's because the megalomaniacs are not actually that smart that they don't realise taking humans out of the loop is the same as relinquishing control, which is one step removed from handing control to the enemy.
- MichaelZuo 3y ago
  This seems a bit redundant to be honest. Any maniac who is actually super competent would be able to fool you into thinking otherwise. Because they can outwit you by definition. And the inverse also is tautological, of course you can spot those not that competent.

deleted 3y ago

[deleted]

ChrisMarshallNY 3y ago

Whenever I hear "autonomous weapons," I think of this scene: https://www.youtube.com/watch?v=TYsulVXpgYg https://www.youtube.com/watch?v=TYsulVXpgYg

aaron695 3y ago

[dead]

tomohawk 3y ago

If you're interested in this sort of thing, Ryan Mcbeth has been talking about this recently. https://www.youtube.com/watch?v=0AUqYmCeZN4 https://www.youtube.com/watch?v=0AUqYmCeZN4

pvillano 3y ago

A landmine is an autonomous weapon. Landmines also kill a lot of civilians.

andsoitis 3y ago
> A landmine is an autonomous weapon. Landmines also kill a lot of civilians. "Anti-personnel landmines are prohibited under the Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction (or Mine Ban Convention), adopted in 1997. More than 150 countries have joined this treaty." - https://disarmament.unoda.org/convarms/landmines/ https://disarmament.unoda.org/convarms/landmines/
- slaymaker1907 3y ago
  150 countries, but missing a lot of key ones like the US, Israel, Russia, and China. (They absolutely should sign this, though). I’m suspicious that all it will take is one major shock for the house of cards that is international law to come tumbling down just like it did before WW2.
  - cpgxiii 3y ago
    Definitely. Prohibitions on land mines and cluster weapons are pure political theater. Mines and cluster weapons don't exist to be cruel, they exist because they work, and the collateral damage caused by their persistence after a conflict is over is an unfortunate consequence. Essentially all of the countries that signed onto these bans either (1) did so in a moment of blinding post-cold war naivety, where they could spin downsizing their armed forces into a good thing for the world, or (2) are too small and/or uninvolved to matter. Anyone else who actually expected to need to fight a war again didn't sign, because they knew exactly how useful these weapons are. The US and SK will never accept a land mine prohibition that includes the minefields of the DMZ, since those minefields are an important speedbump against any sort of ground invasion. Likewise, a cluster munition ban is practically DoA because cluster munitions are the best weapons to attack columns of lightly/unarmored equipment and dispersed aircraft/helicopters. But until February 2022 in Ukraine, a whole bunch of countries (primarily European) had convinced themselves that the possibility of a ground invasion was a thing of the past.

beloch 3y ago

[flagged]

adolph 3y ago
It is interesting/odd that this conflict of all going on at the moment brings forth the examples of AI utilization. Why not Russia-Ukraine, or Myanmar, or any other conflict? What is the specific example of "AI" as opposed to "statistics" that is being used? It doesn't seem very AI to perform analysis of social graph to ID someone as a target and then tell your autonomous surveillance platform to watch that person's addresses. https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflicts https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflict...
- beloch 3y ago
  First, Gaza really is an ideal testing ground for this sort of AI targeting system. It's tiny. Pretty much all of it is under continual surveillance. Israel has completely modern precision bombing capabilities. By comparison, just the front in the Ukraine-Russia war is massive. Many of the targets are nowhere near the front. The surveillance and precision air strike capabilities of both sides are inferior to those of the IDF. Yet, we do see a lot of really savvy strikes being made on the basis of cell-phone data, etc.. This kind of intelligence could have been gathered by humans, but it's not impossible that either or both sides are already using AI targeting systems similar to Lavender. They're likely just not as effective for the reasons mentioned above, and others. Militaries are likely not going to talk about this stuff much. Heck, this capability is in the same ballpark as things that past leaders may have sacrificed cities in order to hide[1]. Even the article on Lavender uses a lot of anonymous sources, so who really knows if it's true? Second, it may not seem "very AI" to identify targets and determine when they're home. Yes, humans could do this. It would take a significant amount of time though. If the reports are true, Lavender is able to do this so quickly that Israel can bomb the homes not just of leaders and generals, but of raw recruits who've probably received the sum total preparation of a rifle and a one-time pep-talk. Quantity has a quality all it's own. https://en.wikipedia.org/wiki/Coventry_Blitz#Coventry_and_Ultra https://en.wikipedia.org/wiki/Coventry_Blitz#Coventry_and_Ul...
3y ago

deleted 3y ago

[deleted]