8 ms·

While there are obvious perils here, I fear it would be tremendously irresponsible to let our adversaries own this space. AI weapons must be developed if for n

While there are obvious perils here, I fear it would be tremendously irresponsible to let our adversaries own this space. AI weapons must be developed if for no other reason than to battle AI weapons.

swiley 5y ago
Now I understand how the anti-nuclear people in the 20th century felt.
- ethbr0 5y ago
  Nuclear MAD seems to have worked out well, so far.
  - anigbrowl 5y ago
    The problem is that MAD is only reliable in the absence of proliferation, but we go to such lengths to prevent proliferation that it causes other problems. When you get down to it a nuke is not that that hard to build with the resources available to almost any state, so the fragility of deterrence has a distorting effect on international relations.
  - yboris 5y ago
    "so far" included very many near misses - where individuals with explicit authority to launch counter attacks were in position to decide whether to launch NUCLEAR MISSILES AT A COUNTRY -- because some sensors malfunctioned. See Stanislav Petrov - the man who prevented an apocalypse https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov
    - ethbr0 5y ago
      Yes. I'll take near misses with nuclear Armageddon over total war. The anti-nuclear weapons crowd conveniently forgets that the alternative to MAD isn't reliably peace.
      - yboris 5y ago
        What % chance of nuclear holocaust per year are you wiling to take over the "non reliable peace"? We're talking about potential existential risk - a wiping out of all of humanity (through severe ecological poisoning after a barrage of cross-globe nuclear strikes).
        ethbr0 5y ago
        0.01%? If my math works out, that's a 10% chance of an occurrence every 1,000 years.
        yboris 5y ago
        Thank you for providing a number. I have seen estimates of what the risk was - as high as 1% per year for much of the duration of the Cold War.
        ethbr0 5y ago
        So if we said '55 - '90 (35 years), that'd be a 30% chance of nukes being launched @ 1% yearly risk. That's more on the fence for me. On one hand, it's high (and the impact would have been catastrophic!). On the other hand, it prevented NATO / Warsaw Pact mechanized conflict across Europe (which also would have been catastrophic). Tough call. Especially with the uncertainty that, were you to give up your nuclear weapons, there's no guarantee your counterparties would actually do the same. In the 60s foundational position papers, there's definitely a strong thread of "We will commit to MAD not because it's the best approach, but because it's the best zero trust approach."
        yboris 5y ago
        > That's more on the fence for me. What? You claimed you are comfortable with 0.01% chance, but now you're "on the fence" with a 1% chance? An analogy: some chance of your child being occasionally bullied in school, or a 0% chance of bullying but a 30% chance they'll get murdered. You say "tough call" -- and I'm baffled.
  - titzer 5y ago
    ...until it doesn't. I would highly recommend a visit to Hiroshima Peace Memorial Museum if you'd like a different perspective on nuclear weapons, rather than a American/western one. And...man, I wish we had that $5.5 trillion back [https://www.brookings.edu/the-hidden-costs-of-our-nuclear-arsenal-overview-of-project-findings/ https://www.brookings.edu/the-hidden-costs-of-our-nuclear-ar...].
    - ethbr0 5y ago
      See also: Guernica, Warsaw, Dresden, London, Wesel, Tokyo, Hargeisa. Nuclear weapons are bigger explosives. Not different explosives.
      - titzer 5y ago
        Every one of those cities was bombed for weeks, sometimes months. Hiroshima and Nagasaki were single bombs. They cause massive radiation exposure and fallout. Tens of thousands of people were poisoned to death in the aftermath of the Japanese bombings. You haven't seen the suffering of those people and cannot even imagine the hell they were subjected to. And today, we have weapons hundreds of times as powerful. City-ending megaweapons by the thousands. And yes, nuclear winter is a real thing too; we could very seriously impact the global climate and cause mass extinctions in a couple of hours. A nuclear war is a planet-scale atrocity that will ends billions of human lives. I've been to the Hiroshima museum. Like I said, it will change your perspective. I'm not really all that inclined to continue this interchange given your flippance.
        ethbr0 5y ago
        If you want to dismiss 100,000 Tokyo citizens burning to death in two nights because it doesn't fit the narrative of "nuclear is the worst," then that's your decision. But personally, I consider them equally horrific. And as stated previously, if MAD prevents either from happening, the risk seems worth it.
        deleted 5y ago
        [deleted]
titzer 5y ago
We cannot allow a missile gap!
- daveslash 5y ago
  Mr. President, we cannot allow a mineshaft gap!
ljd 5y ago
Famous last words. As an AI practitioner, I want us invest in research to manipulate other AI military systems but not use AI ourselves. Take a page from the HFT book.
- tolbish 5y ago
  I am curious where you would draw the line between "AI military systems" and existing systems in use such as Iron Dome and other algorithm-assisted weapons.
  - eternalban 5y ago
    Two obvious dimensions are: - agency: the machine makes decisions on its own. - defensive vs offensive action
    - Sebb767 5y ago
      > - defensive vs offensive action So if Iron dome launches a rocket in defense it's okay, but that rocket mustn't use AI to find its target? I see your general point, but that's going to be a line that's very hard to draw.
    - tolbish 5y ago
      Perhaps I'm missing something, but that first dimension seems far from obvious. To continue my Iron Dome example, isn't the system making decisions on its own when intercepting anything it classifies as an incoming ballistic threats?
      - eternalban 5y ago
        These are obvious dimensions that need to be considered. Of course there can be discussion as to boundaries drawn in that (here 2-d) space as to 'acceptable' and 'unacceptable'. There may be other dimensions as well, such as 'decision making input sources', etc. As to your specific example, while system x may have 'full agency (autonomous, willful)', if it is purely 'defensive' in nature, it may indeed fall within acceptable zone.
sillysaurusx 5y ago
I completely agree, and as an ML researcher, having that opinion feels heretical. So many people are of a “do no harm” mindset that I worry it means “or I won’t work with you.” And I quite like working with various researchers. I mentioned it publicly once, and I’m bracing for the day that some established researcher quote tweets it and says “we do not need this kind of thinking in our community, and we have a duty to exclude it” or some such. I don’t care much though. You seem right, and that’s good enough for me.
- elefanten 5y ago
  We need researchers who will follow their moral instincts far more than we need researchers who care about conforming to the sociocultural norm. So, thank you. I guess I shouldn’t just say “researchers” —- it’s what we need in our citizens of humanity.
  - mistrial9 5y ago
    you are directly contradicting the code of military command, prepare for social credit demotion! </snark>
qwertox 5y ago
There is a science fiction novel by Stanisław Lem titled Peace on Earth which touches this topic. https://en.wikipedia.org/wiki/Peace_on_Earth_(novel) https://en.wikipedia.org/wiki/Peace_on_Earth_(novel) > The evolution of artificial intelligence has allowed major world powers to sign a rather curious treaty: the Moon is divided into national zones (proportional to each nation's Earth real estate) and all weapons development and production must be moved there to be handled by factories. This is supposed to completely demilitarize Earth, achieving the long-sought dream of world peace. A MAD stabilizing factor is apparently preserved by the ability of countries, in case of war, to quickly ship weapons down from the Moon.
proggy 5y ago
I disagree. While I concede that, if there is credible information that an adversary is developing AI weapons that need to be counterbalanced to reach some sort of Nash equilibrium, simply putting time and effort into the development of any adversarial technology will turn up the temperature in the space, so to speak. More work in this area means more capability, more complexity, more risk of misuse, and a heightened risk of the possibility of additional actors. Look no further than the theory of deterrence, and how it shifted from a tolerable short term strategy during the Cold War (one with few actors), to a high risk strategy as the trove of technology was enlarged, the likelihood of it spreading increased, and the number of active and potential actors increased dramatically also. In my view AI weapons development demands a judicious, steady, paced, and thoroughly informed approach to keep the whole space from burning up —- figuratively and literally.
- elefanten 5y ago
  High risk strategy? Deterrence via MAD is overwhelmingly what keeps the relative peace between the major nuclear powers. It’s not a great situation overall, but you couldn’t uninvent nukes at that point, so it has proven remarkably effective thus far. Same dynamic with AI/ML —- can’t uninvent. If one aggression-minded power has it, balance will be required for stability.
yboris 5y ago
100% disagree. We have treaties that prevent use of gas weapons. We need governments to coordinate and cooperate rather than race each other to murderous death. https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/
- srcmap 5y ago
  How do we enforce any treaties the prevent AI usage?
  - mLuby 5y ago
    With an AI, specifically an AI that kills younger AIs (called the Great Old 1).
- ethbr0 5y ago
  A cynic would say that we only banned chemical weapons because they weren't particularly useful in major power combat. See: cluster munition and mine bans
  - munk-a 5y ago
    That cynic would be wrong since chemical weapons are extremely effective in combat. When the scales aren't even (i.e. something like the vietnam war) the stronger side can afford equipment that allows their soldiers to operate effectively while the other side is unable to respond to aggression greatly lowering expected casualties. With powers on par you still have the factor of well coordinated chemical attacks allowing breakthroughs to be executed much easier. Chemical weapons being banned is an example of a mostly inexplicably altruistic action on the part of major powers. Well, inexplicable if you're a cynic.
    - dmitriid 5y ago
      > That cynic would be wrong since chemical weapons are extremely effective in combat. They are not. It's the most ineffective weapon outside swords and bows. A sibling comment provided the link discussing it: https://acoup.blog/2020/03/20/collections-why-dont-we-use-chemical-weapons-anymore/ https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...
jldugger 5y ago
> it would be tremendously irresponsible to let our adversaries own this space Can't wait for your finished script of Dr Strangelove 2.
- mLuby 5y ago
  Wonder when we'll get to the MAINPART: Militarized Artificial Intelligence Non-Proliferation And Reduction Treaty.
- tablespoon 5y ago
  >> it would be tremendously irresponsible to let our adversaries own this space > Can't wait for your finished script of Dr Strangelove 2. It's easy to mock and parody MAD, arms races, etc., but certain conditions make them necessary and rational, and those conditions can't easily be changed.
sitkack 5y ago
I have only a single downvote, so I'll leave a comment instead. This will directly lead to an AI arms race. We need to put a pin in this just like chemical weapons. Our civilization might not survive a WW1 level experiment. If we go down this path, and we survive, earth will be the best approximation of hell in heaven.
alexfromapex 5y ago
We can use AI to control things but having an AI decide to attack or not without human sign off is a terrible terrible idea
gus_massa 5y ago
Do you realize it's the same reason used to start the Manhattan project? Do you realize that only one country nuked civilian cities?

swiley 5y ago

Now I understand how the anti-nuclear people in the 20th century felt.

ethbr0 5y ago
Nuclear MAD seems to have worked out well, so far.
- anigbrowl 5y ago
  The problem is that MAD is only reliable in the absence of proliferation, but we go to such lengths to prevent proliferation that it causes other problems. When you get down to it a nuke is not that that hard to build with the resources available to almost any state, so the fragility of deterrence has a distorting effect on international relations.
- yboris 5y ago
  "so far" included very many near misses - where individuals with explicit authority to launch counter attacks were in position to decide whether to launch NUCLEAR MISSILES AT A COUNTRY -- because some sensors malfunctioned. See Stanislav Petrov - the man who prevented an apocalypse https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov
  - ethbr0 5y ago
    Yes. I'll take near misses with nuclear Armageddon over total war. The anti-nuclear weapons crowd conveniently forgets that the alternative to MAD isn't reliably peace.
    - yboris 5y ago
      What % chance of nuclear holocaust per year are you wiling to take over the "non reliable peace"? We're talking about potential existential risk - a wiping out of all of humanity (through severe ecological poisoning after a barrage of cross-globe nuclear strikes).
      - ethbr0 5y ago
        0.01%? If my math works out, that's a 10% chance of an occurrence every 1,000 years.
        yboris 5y ago
        Thank you for providing a number. I have seen estimates of what the risk was - as high as 1% per year for much of the duration of the Cold War.
        ethbr0 5y ago
        So if we said '55 - '90 (35 years), that'd be a 30% chance of nukes being launched @ 1% yearly risk. That's more on the fence for me. On one hand, it's high (and the impact would have been catastrophic!). On the other hand, it prevented NATO / Warsaw Pact mechanized conflict across Europe (which also would have been catastrophic). Tough call. Especially with the uncertainty that, were you to give up your nuclear weapons, there's no guarantee your counterparties would actually do the same. In the 60s foundational position papers, there's definitely a strong thread of "We will commit to MAD not because it's the best approach, but because it's the best zero trust approach."
        yboris 5y ago
        > That's more on the fence for me. What? You claimed you are comfortable with 0.01% chance, but now you're "on the fence" with a 1% chance? An analogy: some chance of your child being occasionally bullied in school, or a 0% chance of bullying but a 30% chance they'll get murdered. You say "tough call" -- and I'm baffled.
- titzer 5y ago
  ...until it doesn't. I would highly recommend a visit to Hiroshima Peace Memorial Museum if you'd like a different perspective on nuclear weapons, rather than a American/western one. And...man, I wish we had that $5.5 trillion back [https://www.brookings.edu/the-hidden-costs-of-our-nuclear-arsenal-overview-of-project-findings/ https://www.brookings.edu/the-hidden-costs-of-our-nuclear-ar...].
  - ethbr0 5y ago
    See also: Guernica, Warsaw, Dresden, London, Wesel, Tokyo, Hargeisa. Nuclear weapons are bigger explosives. Not different explosives.
    - titzer 5y ago
      Every one of those cities was bombed for weeks, sometimes months. Hiroshima and Nagasaki were single bombs. They cause massive radiation exposure and fallout. Tens of thousands of people were poisoned to death in the aftermath of the Japanese bombings. You haven't seen the suffering of those people and cannot even imagine the hell they were subjected to. And today, we have weapons hundreds of times as powerful. City-ending megaweapons by the thousands. And yes, nuclear winter is a real thing too; we could very seriously impact the global climate and cause mass extinctions in a couple of hours. A nuclear war is a planet-scale atrocity that will ends billions of human lives. I've been to the Hiroshima museum. Like I said, it will change your perspective. I'm not really all that inclined to continue this interchange given your flippance.
      - ethbr0 5y ago
        If you want to dismiss 100,000 Tokyo citizens burning to death in two nights because it doesn't fit the narrative of "nuclear is the worst," then that's your decision. But personally, I consider them equally horrific. And as stated previously, if MAD prevents either from happening, the risk seems worth it.
      - deleted 5y ago
        [deleted]

titzer 5y ago

We cannot allow a missile gap!

daveslash 5y ago
Mr. President, we cannot allow a mineshaft gap!

ljd 5y ago

Famous last words. As an AI practitioner, I want us invest in research to manipulate other AI military systems but not use AI ourselves. Take a page from the HFT book.

tolbish 5y ago
I am curious where you would draw the line between "AI military systems" and existing systems in use such as Iron Dome and other algorithm-assisted weapons.
- eternalban 5y ago
  Two obvious dimensions are: - agency: the machine makes decisions on its own. - defensive vs offensive action
  - Sebb767 5y ago
    > - defensive vs offensive action So if Iron dome launches a rocket in defense it's okay, but that rocket mustn't use AI to find its target? I see your general point, but that's going to be a line that's very hard to draw.
  - tolbish 5y ago
    Perhaps I'm missing something, but that first dimension seems far from obvious. To continue my Iron Dome example, isn't the system making decisions on its own when intercepting anything it classifies as an incoming ballistic threats?
    - eternalban 5y ago
      These are obvious dimensions that need to be considered. Of course there can be discussion as to boundaries drawn in that (here 2-d) space as to 'acceptable' and 'unacceptable'. There may be other dimensions as well, such as 'decision making input sources', etc. As to your specific example, while system x may have 'full agency (autonomous, willful)', if it is purely 'defensive' in nature, it may indeed fall within acceptable zone.

sillysaurusx 5y ago

I completely agree, and as an ML researcher, having that opinion feels heretical. So many people are of a “do no harm” mindset that I worry it means “or I won’t work with you.” And I quite like working with various researchers. I mentioned it publicly once, and I’m bracing for the day that some established researcher quote tweets it and says “we do not need this kind of thinking in our community, and we have a duty to exclude it” or some such. I don’t care much though. You seem right, and that’s good enough for me.

elefanten 5y ago
We need researchers who will follow their moral instincts far more than we need researchers who care about conforming to the sociocultural norm. So, thank you. I guess I shouldn’t just say “researchers” —- it’s what we need in our citizens of humanity.
- mistrial9 5y ago
  you are directly contradicting the code of military command, prepare for social credit demotion! </snark>

qwertox 5y ago

There is a science fiction novel by Stanisław Lem titled Peace on Earth which touches this topic. https://en.wikipedia.org/wiki/Peace_on_Earth_(novel) https://en.wikipedia.org/wiki/Peace_on_Earth_(novel) > The evolution of artificial intelligence has allowed major world powers to sign a rather curious treaty: the Moon is divided into national zones (proportional to each nation's Earth real estate) and all weapons development and production must be moved there to be handled by factories. This is supposed to completely demilitarize Earth, achieving the long-sought dream of world peace. A MAD stabilizing factor is apparently preserved by the ability of countries, in case of war, to quickly ship weapons down from the Moon.

proggy 5y ago

I disagree. While I concede that, if there is credible information that an adversary is developing AI weapons that need to be counterbalanced to reach some sort of Nash equilibrium, simply putting time and effort into the development of any adversarial technology will turn up the temperature in the space, so to speak. More work in this area means more capability, more complexity, more risk of misuse, and a heightened risk of the possibility of additional actors. Look no further than the theory of deterrence, and how it shifted from a tolerable short term strategy during the Cold War (one with few actors), to a high risk strategy as the trove of technology was enlarged, the likelihood of it spreading increased, and the number of active and potential actors increased dramatically also. In my view AI weapons development demands a judicious, steady, paced, and thoroughly informed approach to keep the whole space from burning up —- figuratively and literally.

elefanten 5y ago
High risk strategy? Deterrence via MAD is overwhelmingly what keeps the relative peace between the major nuclear powers. It’s not a great situation overall, but you couldn’t uninvent nukes at that point, so it has proven remarkably effective thus far. Same dynamic with AI/ML —- can’t uninvent. If one aggression-minded power has it, balance will be required for stability.

yboris 5y ago

100% disagree. We have treaties that prevent use of gas weapons. We need governments to coordinate and cooperate rather than race each other to murderous death. https://www.stopkillerrobots.org/ https://www.stopkillerrobots.org/

srcmap 5y ago
How do we enforce any treaties the prevent AI usage?
- mLuby 5y ago
  With an AI, specifically an AI that kills younger AIs (called the Great Old 1).
ethbr0 5y ago
A cynic would say that we only banned chemical weapons because they weren't particularly useful in major power combat. See: cluster munition and mine bans
- munk-a 5y ago
  That cynic would be wrong since chemical weapons are extremely effective in combat. When the scales aren't even (i.e. something like the vietnam war) the stronger side can afford equipment that allows their soldiers to operate effectively while the other side is unable to respond to aggression greatly lowering expected casualties. With powers on par you still have the factor of well coordinated chemical attacks allowing breakthroughs to be executed much easier. Chemical weapons being banned is an example of a mostly inexplicably altruistic action on the part of major powers. Well, inexplicable if you're a cynic.
  - dmitriid 5y ago
    > That cynic would be wrong since chemical weapons are extremely effective in combat. They are not. It's the most ineffective weapon outside swords and bows. A sibling comment provided the link discussing it: https://acoup.blog/2020/03/20/collections-why-dont-we-use-chemical-weapons-anymore/ https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...

jldugger 5y ago

> it would be tremendously irresponsible to let our adversaries own this space Can't wait for your finished script of Dr Strangelove 2.

mLuby 5y ago
Wonder when we'll get to the MAINPART: Militarized Artificial Intelligence Non-Proliferation And Reduction Treaty.
tablespoon 5y ago
>> it would be tremendously irresponsible to let our adversaries own this space > Can't wait for your finished script of Dr Strangelove 2. It's easy to mock and parody MAD, arms races, etc., but certain conditions make them necessary and rational, and those conditions can't easily be changed.

sitkack 5y ago

I have only a single downvote, so I'll leave a comment instead. This will directly lead to an AI arms race. We need to put a pin in this just like chemical weapons. Our civilization might not survive a WW1 level experiment. If we go down this path, and we survive, earth will be the best approximation of hell in heaven.

alexfromapex 5y ago

We can use AI to control things but having an AI decide to attack or not without human sign off is a terrible terrible idea

gus_massa 5y ago

Do you realize it's the same reason used to start the Manhattan project? Do you realize that only one country nuked civilian cities?