8 ms·

So, I realize that lots of apocalyptic projections get made about AI that have a habit of not coming true, but this seems like an awful idea. Also, an awfully

So, I realize that lots of apocalyptic projections get made about AI that have a habit of not coming true, but this seems like an awful idea. Also, an awfully hard one to resist. If your opponent has fast-as-a-computer-can-pull-the-trigger firepower, it's going to be pretty hard to resist the urge to get your own. Once you have two sides with automated offensive capability, we are one bug away from the kind of mistakes that, during the Cold War, were intercepted by humans in the loop. I'm not sure how this can NOT end badly.

Waterluvian 6y ago
I have a feeling that humans will remain overwhelmingly responsible for most of the wartime mistakes. I think this will further reduce conflict between advanced nations (robots killing robots just becomes a stalemate) and will intensely escalate any asymmetric conflicts where one advanced nation can crush whoever without having to pay the price of deploying humans (think CIA drone strikes).
- gizmo 6y ago
  "wartime mistakes" is a nice euphemism. And I'm sure that people will continue be held to account. For instance when somebody whoopsies and dronestrikes a wedding party in Jemen.
- wizzwizz4 6y ago
  > I have a feeling that humans will remain overwhelmingly responsible for most of the wartime mistakes. Of course we will. It's not as though they're programming themselves. Somebody has to tell the computer what to do.
  - Waterluvian 6y ago
    Hah. Good point. But I meant like self driving: we suuuuuuuck. Let's not fool ourselves acting like it's critical that there's a human touch to things.
    - gmfawcett 6y ago
      Nor should we fool ourselves that it's important for a machine to take over. We don't "suuuck" at driving -- or at killing other people, for that matter: as a species, on average, we're fairly competent at both tasks. The fact that a machine might have a lower failure rate isn't a singular reason to embrace the tech.
      - Waterluvian 6y ago
        I agree with that. I was also referring to how much we suck at not killing the wrong people. I imagine systemic racism would still exist, but at least the robots are making decisions consistently.
        heavyset_go 6y ago
        > I imagine systemic racism would still exist, but at least the robots are making decisions consistently. We program our robots with our prejudices[1]. If you don't control the machines, you can't change their programming. It's entirely possible that such prejudices are cemented through machines that cannot be influenced or changed except through actions of their owners. What happens when the owners of these machines are indifferent towards, or outright supportive of, prejudice? History is rife with owners of companies, industries and governments exploiting and furthering prejudice for their own benefit. [1] https://www.americanbar.org/groups/judicial/publications/judges_journal/2020/winter/ai-and-judges-ethical-obligations/ https://www.americanbar.org/groups/judicial/publications/jud...
        penetraitor69 6y ago
        Exactly. If your fire-control system is trained on a dataset that (through carelessness or malice) has some bias, you could end up with a racist machine gun turret.
        marshmallow_12 6y ago
        the problem with this is, our prejudices can change, sometimes very quickly. A person who loves dogs, for example, might hate them after being mauled by one. Simply put, i don't think we can simply write down our emotional processes easily enough to make advanced ai relevant.
- baybal2 6y ago
  I'd say more countries will choose a nuclear route over being droned to death at near zero cost to attacker. Atomic weapons are forties technology, and even nations as locked down, and downtrodden like North Korea, or Israel (as it was in sixties) can not only make one, but seemingly genuinely capable of keeping it hidden against modern spy, and nuclear detection tech.
  - vladTheInhaler 6y ago
    Nuclear weapons still aren't easy to make by any means. It's not so much the nuclear physics that's the challenge as it is all of the engineering best-practices. That, and the difficulty of obtaining raw materials. And nuclear weapons aren't a binary either. North Korea has demonstrated the actual bomb, but their delivery systems are still far from reliable. Their rockets might technically have the range to threaten the US, but what are the chances they could actually hit their target? And how many launch vehicles do they realistically have? Not likely enough to seriously threated any other nuclear powers. And this after making decades of enormous political and economic sacrifices to pursue nuclear weapons. I don't think most countries will feel tempted to follow their lead.
    - baybal2 6y ago
      You don't need 100% reliance on ICBMs if your only aim just to hit enemy's mainland for the sake of it. Cheaper means are on the table. Much more relevant theatre range weapons are things even NK can manufacture. I was very surprised to learn that North Korea not only designed a working cruise missile with their economy, but had a working terrain hugging flight on it.
      - Waterluvian 6y ago
        This got me thinking about the common misperception of how empty the Earth is: If you dropped a bomb of a certain type and strength on a random coordinate in <country> how many people do you kill on average?
        6y ago
altcognito 6y ago
It's ridiculous to think that all the advanced countries aren't working on exactly this. To some extent, it totally makes sense. Defining an area as a kill zone and a bomb that is dropped on a particular area aren't all that different.
- deleted 6y ago
  [deleted]
- zamadatix 6y ago
  In intent no, in unintended risk yes. And not in a "collateral damage" kind of way, more in a "it's hard to use a bomb in a way the damage zone becomes 100x bigger than intended".
- thesuperbigfrog 6y ago
  >> It's ridiculous to think that all the advanced countries aren't working on exactly this. They are. And a centralized AI will be needed to coordinate all of the autonomous weapon systems and react faster than the bad guys--SkyNet is coming too.
  - JumpCrisscross 6y ago
    > a centralized AI will be needed to coordinate all of the autonomous weapon systems Why? At a tactical level, an AI to coordinate a battlefield makes sense. (Speed of light means local computing has an advantage, particularly with directed energy weapons.) And an AI to advise on strategy, and coördinate those battlefield AIs, makes sense. But while battles are mechanical, war is political. At the point that the human becomes replaceable we’d have an AI substitute for government, which is a long way away. For grand strategic concerns, we have nukes, and those still fly slowly and detectably enough that an AI commanding counterstrike is unnecessary. (Unless submarine platforms become detectable and thus destroyable in a first strike.)
  - freeone3000 6y ago
    Why? Each individual system is more than capable of simply fulfilling tactics on their own. We're not short computing power! Central coordination is a risk to be avouded -- each individual drone can simply be Skynet all in their own.
fossuser 6y ago
Reminds me in a weird way of this episode: https://www.imdb.com/title/tt0708414/ https://www.imdb.com/title/tt0708414/ A culture has computer simulated war and “casualties” must report to euthanasia chambers. This was the end result agreed to to stop actual bombings. Of course, the officers disrupt it in the end. But it’d be interesting if they returned and nothing was left but craters. More on topic, iron dome is an automated missile interception system and works well to protect Israelis from missile attacks. Just because a system is automated doesn’t necessarily mean it’s a bad idea, but obviously there’s high risk. https://en.wikipedia.org/wiki/Iron_Dome https://en.wikipedia.org/wiki/Iron_Dome
- jbob2000 6y ago
  You could argue that the Iron Dome has allowed the Israelis to perpetuate the conflict. If Israel citizens were faced with actually dying, they might push for a peaceful resolution faster.
  - philwelch 6y ago
    A faster resolution, yes, but not a peaceful one.
  - fossuser 6y ago
    On the subject of ethics - proportionate response, narrow targeting, and trying not to kill civilians are good things. This topic is too political to have a nuanced internet discussion and quickly dissolved into weird anti-semitism most of the time. https://samharris.org/podcasts/why-dont-i-criticize-israel/ https://samharris.org/podcasts/why-dont-i-criticize-israel/
    - tomjen3 6y ago
      >proportionate response, narrow targeting, and trying not to kill civilians are good things. How is it better to only respond with proportionate responses? If you were known as generally a good country, but could go mad at the drop of a hat, other countries would take even more steps to avoid harming you, wouldn't they? At the same time, if you were known of flattening neighborhods or even entire cities, your enemies wouldn't be able to hide among civilians. In either case you trade of more deaths now, for fewer dead in the future.
QuesnayJr 6y ago
When I watch the Star Wars movies, I mostly think of them as pure fantasy. Except for the killer drone in "Attack of the Clones" -- that was the one time I thought "That's definitely going to happen. They're going to make that."
JumpCrisscross 6y ago
> Once you have two sides with automated offensive capability, we are one bug away from the kind of mistakes that, during the Cold War, were intercepted by humans in the loop There is a massive difference between guns on automated fire control and an AI aiming missiles. One could argue that, by making the fire control guaranteed, the border might stop being challenged and deaths thus go down. (Nobody to taunt, either.)
- m463 6y ago
  > One could argue that, by making the fire control guaranteed, Landmines work by the same principle. Many killed by landmines are children. > 87% of the casualties were civilians and 47% were children (less than 18 years old). https://en.wikipedia.org/wiki/Land_mine#Casualties https://en.wikipedia.org/wiki/Land_mine#Casualties
  - JumpCrisscross 6y ago
    > Landmines work by the same principle Not really. Mine fields are not certain death. They don’t always have the opportunity to announce themselves. Guns on automated fire control have the option of announcing themselves prior to engaging. They can also be turned off, something not easy to do with mines. It’s painful but necessary to note that Hamas have a track record of weaponising children [1]. A child approaching the border is thus not guaranteed safe, and could reasonably be fired upon by a human or a gun on automated fire control. (Not arguing for killing kids. Just that “kids could get killed” isn’t an effective argument against this weapons platform.) War is terrible. I don’t think automatic guns make it significantly more terrible. (Landmines do.) [1] https://en.m.wikipedia.org/wiki/Use_of_child_suicide_bombers_by_Palestinian_militant_groups https://en.m.wikipedia.org/wiki/Use_of_child_suicide_bombers...
  - imtringued 6y ago
    >Landmines work by the same principle. Many killed by landmines are children. That makes no sense to me. Children being killed by landmines is an example of a lack of fire control. If the land mine had fire control it could detect enemy soldiers then children wouldn't have to die.
kumarvvr 6y ago
The next evolution of such killing tech will be AI capable but remotely controlled systems. Its time governments around the world invest in quality gamers to help them fight their battles. I can't help but think, in a war between two nations fighting exclusively with such tech (not too far fetched for my generation or my kids generation), war itself becomes comical and essentially a measure of economic strength. How quickly can you build these robots and how quickly can you deploy them. Imagine a general shouting orders over a bunch of acne-faced teenagers trying to win a war for his country.
betwixthewires 6y ago
I don't understand the disconnect on this topic with technically proficient and interested people. We can talk all day about how autonomous driving is safer than manual driving, but as soon as weapons are mentioned we rely on this idea that not having a human in the decision making process makes error more likely. How can both be true?
- captainmuon 6y ago
  With a self-driving car, the desired outcome is to save lives. With an autonomous weapon, the desired outcome is to kill. Having a human inefficiency in the loop means you can have mercy and refrain from killing. I don't think the problem is with AIs acurrately identifying friends and enemies. You can't solve that 100% anyway, but if it can aid humans and prevent mistakes, all the better. The problem is when you start to build absolutely relentless border drones that can one-shot the enemy, without any possibility for mercy or negotiation.
  - betwixthewires 6y ago
    I don't think there's a single militarized border on earth where the sentry holding the gun has any say on whether he must shoot someone who violates the perimeter. Mercy and negotiation on these issues happen with diplomacy, not along the fenceline. With weapons, the desired outcome is not to kill, not in most circumstances anyway. In circumstances where it is, it is almost always the desire of an individual wielding a weapon. Normally, the desired outcome is deterrence or defense, and when used proactively, to lessen the enemy's ability to continue to wage the conflict. With deterrence, the threat of death is the deterrent, with defense, killing is the machanism by which the threat is neutralized, and with proactive aggression the goal may or may not include killing, and is more likely to include killing if the enemy's ability to wage conflict is dependent on manpower. In all three scenarios, the desired outcome of developing autonomous military capability is to save lives.
    - bigfudge 6y ago
      I doubt that is true. In Iraq and Afghanistan there was wide variation in the number of civilians killed by different units. Some of that may be context-effects, but I doubt all of it. Also, having people in the loop reminds at least someone that this is 'real' and that we need to carry on thinking about it. If you can obliterate enemies' bodies and the only duty on people is to restock the ammo I think that reduces the moral burden of conflict in an unacceptable way.

Waterluvian 6y ago

I have a feeling that humans will remain overwhelmingly responsible for most of the wartime mistakes. I think this will further reduce conflict between advanced nations (robots killing robots just becomes a stalemate) and will intensely escalate any asymmetric conflicts where one advanced nation can crush whoever without having to pay the price of deploying humans (think CIA drone strikes).

gizmo 6y ago
"wartime mistakes" is a nice euphemism. And I'm sure that people will continue be held to account. For instance when somebody whoopsies and dronestrikes a wedding party in Jemen.
wizzwizz4 6y ago
> I have a feeling that humans will remain overwhelmingly responsible for most of the wartime mistakes. Of course we will. It's not as though they're programming themselves. Somebody has to tell the computer what to do.
- Waterluvian 6y ago
  Hah. Good point. But I meant like self driving: we suuuuuuuck. Let's not fool ourselves acting like it's critical that there's a human touch to things.
  - gmfawcett 6y ago
    Nor should we fool ourselves that it's important for a machine to take over. We don't "suuuck" at driving -- or at killing other people, for that matter: as a species, on average, we're fairly competent at both tasks. The fact that a machine might have a lower failure rate isn't a singular reason to embrace the tech.
    - Waterluvian 6y ago
      I agree with that. I was also referring to how much we suck at not killing the wrong people. I imagine systemic racism would still exist, but at least the robots are making decisions consistently.
      - heavyset_go 6y ago
        > I imagine systemic racism would still exist, but at least the robots are making decisions consistently. We program our robots with our prejudices[1]. If you don't control the machines, you can't change their programming. It's entirely possible that such prejudices are cemented through machines that cannot be influenced or changed except through actions of their owners. What happens when the owners of these machines are indifferent towards, or outright supportive of, prejudice? History is rife with owners of companies, industries and governments exploiting and furthering prejudice for their own benefit. [1] https://www.americanbar.org/groups/judicial/publications/judges_journal/2020/winter/ai-and-judges-ethical-obligations/ https://www.americanbar.org/groups/judicial/publications/jud...
        penetraitor69 6y ago
        Exactly. If your fire-control system is trained on a dataset that (through carelessness or malice) has some bias, you could end up with a racist machine gun turret.
        marshmallow_12 6y ago
        the problem with this is, our prejudices can change, sometimes very quickly. A person who loves dogs, for example, might hate them after being mauled by one. Simply put, i don't think we can simply write down our emotional processes easily enough to make advanced ai relevant.
baybal2 6y ago
I'd say more countries will choose a nuclear route over being droned to death at near zero cost to attacker. Atomic weapons are forties technology, and even nations as locked down, and downtrodden like North Korea, or Israel (as it was in sixties) can not only make one, but seemingly genuinely capable of keeping it hidden against modern spy, and nuclear detection tech.
- vladTheInhaler 6y ago
  Nuclear weapons still aren't easy to make by any means. It's not so much the nuclear physics that's the challenge as it is all of the engineering best-practices. That, and the difficulty of obtaining raw materials. And nuclear weapons aren't a binary either. North Korea has demonstrated the actual bomb, but their delivery systems are still far from reliable. Their rockets might technically have the range to threaten the US, but what are the chances they could actually hit their target? And how many launch vehicles do they realistically have? Not likely enough to seriously threated any other nuclear powers. And this after making decades of enormous political and economic sacrifices to pursue nuclear weapons. I don't think most countries will feel tempted to follow their lead.
  - baybal2 6y ago
    You don't need 100% reliance on ICBMs if your only aim just to hit enemy's mainland for the sake of it. Cheaper means are on the table. Much more relevant theatre range weapons are things even NK can manufacture. I was very surprised to learn that North Korea not only designed a working cruise missile with their economy, but had a working terrain hugging flight on it.
    - Waterluvian 6y ago
      This got me thinking about the common misperception of how empty the Earth is: If you dropped a bomb of a certain type and strength on a random coordinate in <country> how many people do you kill on average?
      - 6y ago

altcognito 6y ago

It's ridiculous to think that all the advanced countries aren't working on exactly this. To some extent, it totally makes sense. Defining an area as a kill zone and a bomb that is dropped on a particular area aren't all that different.

deleted 6y ago
[deleted]
zamadatix 6y ago
In intent no, in unintended risk yes. And not in a "collateral damage" kind of way, more in a "it's hard to use a bomb in a way the damage zone becomes 100x bigger than intended".
thesuperbigfrog 6y ago
>> It's ridiculous to think that all the advanced countries aren't working on exactly this. They are. And a centralized AI will be needed to coordinate all of the autonomous weapon systems and react faster than the bad guys--SkyNet is coming too.
- JumpCrisscross 6y ago
  > a centralized AI will be needed to coordinate all of the autonomous weapon systems Why? At a tactical level, an AI to coordinate a battlefield makes sense. (Speed of light means local computing has an advantage, particularly with directed energy weapons.) And an AI to advise on strategy, and coördinate those battlefield AIs, makes sense. But while battles are mechanical, war is political. At the point that the human becomes replaceable we’d have an AI substitute for government, which is a long way away. For grand strategic concerns, we have nukes, and those still fly slowly and detectably enough that an AI commanding counterstrike is unnecessary. (Unless submarine platforms become detectable and thus destroyable in a first strike.)
- freeone3000 6y ago
  Why? Each individual system is more than capable of simply fulfilling tactics on their own. We're not short computing power! Central coordination is a risk to be avouded -- each individual drone can simply be Skynet all in their own.

fossuser 6y ago

Reminds me in a weird way of this episode: https://www.imdb.com/title/tt0708414/ https://www.imdb.com/title/tt0708414/ A culture has computer simulated war and “casualties” must report to euthanasia chambers. This was the end result agreed to to stop actual bombings. Of course, the officers disrupt it in the end. But it’d be interesting if they returned and nothing was left but craters. More on topic, iron dome is an automated missile interception system and works well to protect Israelis from missile attacks. Just because a system is automated doesn’t necessarily mean it’s a bad idea, but obviously there’s high risk. https://en.wikipedia.org/wiki/Iron_Dome https://en.wikipedia.org/wiki/Iron_Dome

jbob2000 6y ago
You could argue that the Iron Dome has allowed the Israelis to perpetuate the conflict. If Israel citizens were faced with actually dying, they might push for a peaceful resolution faster.
- philwelch 6y ago
  A faster resolution, yes, but not a peaceful one.
- fossuser 6y ago
  On the subject of ethics - proportionate response, narrow targeting, and trying not to kill civilians are good things. This topic is too political to have a nuanced internet discussion and quickly dissolved into weird anti-semitism most of the time. https://samharris.org/podcasts/why-dont-i-criticize-israel/ https://samharris.org/podcasts/why-dont-i-criticize-israel/
  - tomjen3 6y ago
    >proportionate response, narrow targeting, and trying not to kill civilians are good things. How is it better to only respond with proportionate responses? If you were known as generally a good country, but could go mad at the drop of a hat, other countries would take even more steps to avoid harming you, wouldn't they? At the same time, if you were known of flattening neighborhods or even entire cities, your enemies wouldn't be able to hide among civilians. In either case you trade of more deaths now, for fewer dead in the future.

QuesnayJr 6y ago

When I watch the Star Wars movies, I mostly think of them as pure fantasy. Except for the killer drone in "Attack of the Clones" -- that was the one time I thought "That's definitely going to happen. They're going to make that."

JumpCrisscross 6y ago

> Once you have two sides with automated offensive capability, we are one bug away from the kind of mistakes that, during the Cold War, were intercepted by humans in the loop There is a massive difference between guns on automated fire control and an AI aiming missiles. One could argue that, by making the fire control guaranteed, the border might stop being challenged and deaths thus go down. (Nobody to taunt, either.)

m463 6y ago
> One could argue that, by making the fire control guaranteed, Landmines work by the same principle. Many killed by landmines are children. > 87% of the casualties were civilians and 47% were children (less than 18 years old). https://en.wikipedia.org/wiki/Land_mine#Casualties https://en.wikipedia.org/wiki/Land_mine#Casualties
- JumpCrisscross 6y ago
  > Landmines work by the same principle Not really. Mine fields are not certain death. They don’t always have the opportunity to announce themselves. Guns on automated fire control have the option of announcing themselves prior to engaging. They can also be turned off, something not easy to do with mines. It’s painful but necessary to note that Hamas have a track record of weaponising children [1]. A child approaching the border is thus not guaranteed safe, and could reasonably be fired upon by a human or a gun on automated fire control. (Not arguing for killing kids. Just that “kids could get killed” isn’t an effective argument against this weapons platform.) War is terrible. I don’t think automatic guns make it significantly more terrible. (Landmines do.) [1] https://en.m.wikipedia.org/wiki/Use_of_child_suicide_bombers_by_Palestinian_militant_groups https://en.m.wikipedia.org/wiki/Use_of_child_suicide_bombers...
- imtringued 6y ago
  >Landmines work by the same principle. Many killed by landmines are children. That makes no sense to me. Children being killed by landmines is an example of a lack of fire control. If the land mine had fire control it could detect enemy soldiers then children wouldn't have to die.

kumarvvr 6y ago

The next evolution of such killing tech will be AI capable but remotely controlled systems. Its time governments around the world invest in quality gamers to help them fight their battles. I can't help but think, in a war between two nations fighting exclusively with such tech (not too far fetched for my generation or my kids generation), war itself becomes comical and essentially a measure of economic strength. How quickly can you build these robots and how quickly can you deploy them. Imagine a general shouting orders over a bunch of acne-faced teenagers trying to win a war for his country.

betwixthewires 6y ago

I don't understand the disconnect on this topic with technically proficient and interested people. We can talk all day about how autonomous driving is safer than manual driving, but as soon as weapons are mentioned we rely on this idea that not having a human in the decision making process makes error more likely. How can both be true?

captainmuon 6y ago
With a self-driving car, the desired outcome is to save lives. With an autonomous weapon, the desired outcome is to kill. Having a human inefficiency in the loop means you can have mercy and refrain from killing. I don't think the problem is with AIs acurrately identifying friends and enemies. You can't solve that 100% anyway, but if it can aid humans and prevent mistakes, all the better. The problem is when you start to build absolutely relentless border drones that can one-shot the enemy, without any possibility for mercy or negotiation.
- betwixthewires 6y ago
  I don't think there's a single militarized border on earth where the sentry holding the gun has any say on whether he must shoot someone who violates the perimeter. Mercy and negotiation on these issues happen with diplomacy, not along the fenceline. With weapons, the desired outcome is not to kill, not in most circumstances anyway. In circumstances where it is, it is almost always the desire of an individual wielding a weapon. Normally, the desired outcome is deterrence or defense, and when used proactively, to lessen the enemy's ability to continue to wage the conflict. With deterrence, the threat of death is the deterrent, with defense, killing is the machanism by which the threat is neutralized, and with proactive aggression the goal may or may not include killing, and is more likely to include killing if the enemy's ability to wage conflict is dependent on manpower. In all three scenarios, the desired outcome of developing autonomous military capability is to save lives.
  - bigfudge 6y ago
    I doubt that is true. In Iraq and Afghanistan there was wide variation in the number of civilians killed by different units. Some of that may be context-effects, but I doubt all of it. Also, having people in the loop reminds at least someone that this is 'real' and that we need to carry on thinking about it. If you can obliterate enemies' bodies and the only duty on people is to restock the ammo I think that reduces the moral burden of conflict in an unacceptable way.