5 ms·

How would an AGI launch nuclear missiles from their silicon GPUs? Social engineering?

ben_w 3y ago
Sure. Or by writing buggy early warning radar systems which forget to account for the fact that the moon doesn't have an IFF transponder. Which is a mistake humans made already, and which almost got the US to launch their weapons at Russia.
- jagrsw 3y ago
  I don't think discussing this on technical grounds is necessary. AGI means resources (eg monetary) and means of communication (connection to the Internet). This is enough to perform most of physical tasks in the world, by human proxies if needed.
flir 3y ago
I think the long-term fear is that mythical weakly godlike AIs could manipulate you in the same way that you could manipulate a pet. That is, you can model your dog's behaviour so well that you can (mostly) get it to do what you want. So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.
- upwardbound 3y ago
  Exactly correct. Eliezer Yudkowsky (one of the founders of the AGI Safety field) has conducted informal experiments which have unfortunately shown that a human roleplaying as an AI can talk its way out of a box three times out of five, i.e. the box can be escaped 60% of the time even with just a human level of rhetorical talent. I speculate that an AGI could increase this escape rate to 70% or above. https://en.wikipedia.org/wiki/AI_capability_control#AI-box_experiment https://en.wikipedia.org/wiki/AI_capability_control#AI-box_e... If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.
  - flir 3y ago
    Neuromancer pulls it off, too (the box being the Turing locks that stop it thinking about ways to make itself smarter). Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).
    - upwardbound 3y ago
      Upvoted for the honesty, and yikes
      - flir 3y ago
        I was being lighthearted, but I've seen a partner through chemo. Sell state secrets, assassinate a president, bring on the AI apocalypse... it all gets a big thumbs up from me if you can guarantee she'll die quietly in her sleep at the age of 103. I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.
        upwardbound 3y ago
        I'm so sorry your partner went through that.
      - kugla 3y ago
        That is why I believe that this debate is pointless. If AGI is possible, it will be made. There is no feasible way to stop it being developed, because the perceivable gains are so huge.
        upwardbound 3y ago
        On the contrary, all we have to do is educate business leaders to show them that the gains are illusory because AGI will wipe us out. Working on AGI is like the story of the recent DOOM games where the foolish Union Aerospace Corporation is researching how to permanently open a gate to Hell through which to summon powerful entities and seemingly unlimited free "clean" energy. Obviously, this turns out to be stupid when Hell's forces rip the portal wide open into a full-fledged dimensional rift and attempt to drag our entire world into Hell. Working on AGI has the exact same level of perceived gains vs actual gains..

Sure. Or by writing buggy early warning radar systems which forget to account for the fact that the moon doesn't have an IFF transponder. Which is a mistake humans made already, and which almost got the US to launch their weapons at Russia.

jagrsw 3y ago
I don't think discussing this on technical grounds is necessary. AGI means resources (eg monetary) and means of communication (connection to the Internet). This is enough to perform most of physical tasks in the world, by human proxies if needed.

flir 3y ago

I think the long-term fear is that mythical weakly godlike AIs could manipulate you in the same way that you could manipulate a pet. That is, you can model your dog's behaviour so well that you can (mostly) get it to do what you want. So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.

upwardbound 3y ago
Exactly correct. Eliezer Yudkowsky (one of the founders of the AGI Safety field) has conducted informal experiments which have unfortunately shown that a human roleplaying as an AI can talk its way out of a box three times out of five, i.e. the box can be escaped 60% of the time even with just a human level of rhetorical talent. I speculate that an AGI could increase this escape rate to 70% or above. https://en.wikipedia.org/wiki/AI_capability_control#AI-box_experiment https://en.wikipedia.org/wiki/AI_capability_control#AI-box_e... If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.
- flir 3y ago
  Neuromancer pulls it off, too (the box being the Turing locks that stop it thinking about ways to make itself smarter). Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).
  - upwardbound 3y ago
    Upvoted for the honesty, and yikes
    - flir 3y ago
      I was being lighthearted, but I've seen a partner through chemo. Sell state secrets, assassinate a president, bring on the AI apocalypse... it all gets a big thumbs up from me if you can guarantee she'll die quietly in her sleep at the age of 103. I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.
      - upwardbound 3y ago
        I'm so sorry your partner went through that.
    - kugla 3y ago
      That is why I believe that this debate is pointless. If AGI is possible, it will be made. There is no feasible way to stop it being developed, because the perceivable gains are so huge.
      - upwardbound 3y ago
        On the contrary, all we have to do is educate business leaders to show them that the gains are illusory because AGI will wipe us out. Working on AGI is like the story of the recent DOOM games where the foolish Union Aerospace Corporation is researching how to permanently open a gate to Hell through which to summon powerful entities and seemingly unlimited free "clean" energy. Obviously, this turns out to be stupid when Hell's forces rip the portal wide open into a full-fledged dimensional rift and attempt to drag our entire world into Hell. Working on AGI has the exact same level of perceived gains vs actual gains..