4 ms·

How does it actually kill a person? When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire?

dragonwriter 3y ago
> When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire? When someone runs a model in a reasonably durable housing with a battery? (I'm not big on the AI as destroyer or saviour cult myself, but that particular question doesn't seem like all that big of a refutation of it.)
- idontwantthis 3y ago
  But my point is what is it actually doing to reach out and touch someone in the doomsday scenario?
  - LordDragonfang 3y ago
    I mean, the cliched answer is "when it figures out how to override the nuclear launch process". And while that cliche might have a certain degree of unrealism, it would certainly be possible for a system with access to arbitrary compute power that's specifically trained to impersonate human personas to use social engineering to precipitate WW3. And even that isn't the easiest scenario if an AI just wants us dead; a smart enough AI could just as easily use send a request to any of the the many labs that will synthesize/print genetic sequences for you and create things that combine into a plague worse than covid. And if it's really smart, it can figure out how to use those same labs to begin producing self-replicating nanomachines (because that's what viruses are) that give it substrate to run on. Oh, and good luck destroying it when it can copy and shard itself onto every unpatched smarthome device on Earth. Now, granted, none of these individual scenarios have a high absolute likelihood. That said, even at a 10% (or 0.1%) chance of destroying all life, you should probably at least give it some thought.
    - idontwantthis 3y ago
      How can it call one of those labs and place an order for the apocalypse and I can’t right now? Also about the smart home devices: if a current iPhone can’t run Siri locally then how is a Roomba supposed to run an AGI?
      - LordDragonfang 3y ago
        You could if you were educated enough in DNA synthesis and customer service manipulation to do so, and were smart enough to figure out a novel rna sequence based in publicly available data. I'm not, you're not. A superintelligence would be. The base assumption is that any superintelligence is smarter than us, and can solve problems we can't. AI can already come up with novel chemical weapons thousands of times faster than us[1], and it's way dumber than we are. And the roomba isn't running the model, it's just storing a portion of the model for backup. Or only running a fraction of it (very different from an iPhone trying to run the whole model). Instead, the proper model is running on the best computer from the Russian botnet it purchased using crypto it scammed from a discord NFT server. Once again, the premise is that AI is smarter than you or anyone else, and way faster. It can solve any problem that a human like me can figure out a solution for in 30 seconds of spitballing, and it can be an expert in everything. [1]https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx https://www.theverge.com/2022/3/17/22983197/ai-new-possible-...
  - Auracle 3y ago
    Nukes, power grids, planes, blackmail, etc. Surely you’ve seen plenty of media over the years that’s explored this.
    - idontwantthis 3y ago
      What is “nukes” though? Like the missiles in silos that could have been networked decades ago but still require mechanical keys in order to fire? Like is it just making phone calls pretending to be the president and everyone down the line says “ok let’s destroy the world”?
      - Auracle 3y ago
        Perhaps emailing members of whatever terrorist group the exact location, codes, personnel they would need to seize a nuke themselves? I'm not actively worried about it, but let's not pretend something with all of the information in the world and great intelligence couldn't pull it off.
upwardbound 3y ago
One route is if AI (not through malice but simply through incompetence) plays a part in a terrorist plan to trick the US and China or US and Russia into fighting an unwanted nuclear war. A working group I’m a part of, DISARM:SIMC4, has a lot of papers about this here: https://simc4.org https://simc4.org
- justcool393 3y ago
  so the plot of WarGames?
  - upwardbound 3y ago
    Exactly. WarGames is very similar to a true incident that occurred in 1979, four years before the release of the film. https://blog.ucsusa.org/david-wright/nuclear-false-alarm-950/ https://blog.ucsusa.org/david-wright/nuclear-false-alarm-950... In this case, it turns out that a technician mistakenly inserted into a NORAD computer a training tape that simulated a large Soviet attack on the United States. Because of the design of the warning system, that information was sent out widely through the U.S. nuclear command network.
- hurryer 3y ago
  Since you work on this, do you think leaders will wait until confirmation of actual nuclear detonations, maybe on TV, before believing that a massive attack was launched?
  - upwardbound 3y ago
    According to current nuclear doctrine, no, they won’t wait. The current doctrine is called Launch On Warning which means you retaliate immediately after receiving the first indications of incoming missiles. This is incredibly dumb, which is why those of us who study the intersection of AI and global strategic stability are advocating a change to a different doctrine called Decide Under Attack. Decide Under Attack has been shown by game theory to have equally strong deterrence as Launch On Warning, while also having a much much lower chance of accidental or terrorist-triggered war. Here is the paper that introduced Decide Under Attack: A Commonsense Policy for Avoiding a Disastrous Nuclear Decision, Admiral James A Winnefeld, Jr. https://carnegieendowment.org/2019/09/10/commonsense-policy-for-avoiding-disastrous-nuclear-decision-pub-79799 https://carnegieendowment.org/2019/09/10/commonsense-policy-...
grey-area 3y ago
The network is the computer. If you live in a city right now there are millions of networked computers that humans depend on in their everyday life and do not want to turn off. Many of those computers keep humans alive (grid control, traffic control, comms, hospitals etc). Some are actual robotic killing machines but most have other purposes. Hardly any are air-gapped nowadays and all our security assumes the network nodes have no agency. A super intelligence residing in that network would be very difficult to kill and could very easily kill lots of people (destroy a dam for example), however that sort of crude threat is unlikely to be a problem. There are lots of potentially bad scenarios though many of them involving the wrong sort of dictator getting control of such an intelligence. There are legitimate concerns here IMO.

dragonwriter 3y ago

> When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire? When someone runs a model in a reasonably durable housing with a battery? (I'm not big on the AI as destroyer or saviour cult myself, but that particular question doesn't seem like all that big of a refutation of it.)

idontwantthis 3y ago
But my point is what is it actually doing to reach out and touch someone in the doomsday scenario?
- LordDragonfang 3y ago
  I mean, the cliched answer is "when it figures out how to override the nuclear launch process". And while that cliche might have a certain degree of unrealism, it would certainly be possible for a system with access to arbitrary compute power that's specifically trained to impersonate human personas to use social engineering to precipitate WW3. And even that isn't the easiest scenario if an AI just wants us dead; a smart enough AI could just as easily use send a request to any of the the many labs that will synthesize/print genetic sequences for you and create things that combine into a plague worse than covid. And if it's really smart, it can figure out how to use those same labs to begin producing self-replicating nanomachines (because that's what viruses are) that give it substrate to run on. Oh, and good luck destroying it when it can copy and shard itself onto every unpatched smarthome device on Earth. Now, granted, none of these individual scenarios have a high absolute likelihood. That said, even at a 10% (or 0.1%) chance of destroying all life, you should probably at least give it some thought.
  - idontwantthis 3y ago
    How can it call one of those labs and place an order for the apocalypse and I can’t right now? Also about the smart home devices: if a current iPhone can’t run Siri locally then how is a Roomba supposed to run an AGI?
    - LordDragonfang 3y ago
      You could if you were educated enough in DNA synthesis and customer service manipulation to do so, and were smart enough to figure out a novel rna sequence based in publicly available data. I'm not, you're not. A superintelligence would be. The base assumption is that any superintelligence is smarter than us, and can solve problems we can't. AI can already come up with novel chemical weapons thousands of times faster than us[1], and it's way dumber than we are. And the roomba isn't running the model, it's just storing a portion of the model for backup. Or only running a fraction of it (very different from an iPhone trying to run the whole model). Instead, the proper model is running on the best computer from the Russian botnet it purchased using crypto it scammed from a discord NFT server. Once again, the premise is that AI is smarter than you or anyone else, and way faster. It can solve any problem that a human like me can figure out a solution for in 30 seconds of spitballing, and it can be an expert in everything. [1]https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx https://www.theverge.com/2022/3/17/22983197/ai-new-possible-...
- Auracle 3y ago
  Nukes, power grids, planes, blackmail, etc. Surely you’ve seen plenty of media over the years that’s explored this.
  - idontwantthis 3y ago
    What is “nukes” though? Like the missiles in silos that could have been networked decades ago but still require mechanical keys in order to fire? Like is it just making phone calls pretending to be the president and everyone down the line says “ok let’s destroy the world”?
    - Auracle 3y ago
      Perhaps emailing members of whatever terrorist group the exact location, codes, personnel they would need to seize a nuke themselves? I'm not actively worried about it, but let's not pretend something with all of the information in the world and great intelligence couldn't pull it off.

upwardbound 3y ago

One route is if AI (not through malice but simply through incompetence) plays a part in a terrorist plan to trick the US and China or US and Russia into fighting an unwanted nuclear war. A working group I’m a part of, DISARM:SIMC4, has a lot of papers about this here: https://simc4.org https://simc4.org

justcool393 3y ago
so the plot of WarGames?
- upwardbound 3y ago
  Exactly. WarGames is very similar to a true incident that occurred in 1979, four years before the release of the film. https://blog.ucsusa.org/david-wright/nuclear-false-alarm-950/ https://blog.ucsusa.org/david-wright/nuclear-false-alarm-950... In this case, it turns out that a technician mistakenly inserted into a NORAD computer a training tape that simulated a large Soviet attack on the United States. Because of the design of the warning system, that information was sent out widely through the U.S. nuclear command network.
hurryer 3y ago
Since you work on this, do you think leaders will wait until confirmation of actual nuclear detonations, maybe on TV, before believing that a massive attack was launched?
- upwardbound 3y ago
  According to current nuclear doctrine, no, they won’t wait. The current doctrine is called Launch On Warning which means you retaliate immediately after receiving the first indications of incoming missiles. This is incredibly dumb, which is why those of us who study the intersection of AI and global strategic stability are advocating a change to a different doctrine called Decide Under Attack. Decide Under Attack has been shown by game theory to have equally strong deterrence as Launch On Warning, while also having a much much lower chance of accidental or terrorist-triggered war. Here is the paper that introduced Decide Under Attack: A Commonsense Policy for Avoiding a Disastrous Nuclear Decision, Admiral James A Winnefeld, Jr. https://carnegieendowment.org/2019/09/10/commonsense-policy-for-avoiding-disastrous-nuclear-decision-pub-79799 https://carnegieendowment.org/2019/09/10/commonsense-policy-...

grey-area 3y ago

The network is the computer. If you live in a city right now there are millions of networked computers that humans depend on in their everyday life and do not want to turn off. Many of those computers keep humans alive (grid control, traffic control, comms, hospitals etc). Some are actual robotic killing machines but most have other purposes. Hardly any are air-gapped nowadays and all our security assumes the network nodes have no agency. A super intelligence residing in that network would be very difficult to kill and could very easily kill lots of people (destroy a dam for example), however that sort of crude threat is unlikely to be a problem. There are lots of potentially bad scenarios though many of them involving the wrong sort of dictator getting control of such an intelligence. There are legitimate concerns here IMO.