5 ms·

> It’s also going to happen in the blink of an eye — because once it gets loose, there is no stopping it from scaling itself incredibly rapidly. Whether we want

by skulk 5y ago

> It’s also going to happen in the blink of an eye — because once it gets loose, there is no stopping it from scaling itself incredibly rapidly. Whether we want it to or not, it will impact every human being’s life. What does "get loose" mean here? Is someone going to say, "Here little computer program, take this IP address. If you ever need something, call over TCP, send your request as plaintext English and someone will set it up for you. Now, go forth and do whatever you want." I really wish people would talk about this more. There's something missing between our programs that are utterly confined in the boxes we make and these hypothetical skynets that have unlimited influence over the material world. > Once we have AGI, there’s no feasible way to contain it, so it will be free to improve itself and replicate in a runaway exponential fashion — and that’s basically what the idea of a technological singularity describes anyways. Seriously, how? How is it going to replicate? Is it going to figure out how to hack remote systems over the internet and take control of them? Is it going to smooth-talk me into letting it out? It's impossible for me to take anyone seriously if they deify AGI like this.

wincy 5y ago
I recently read this[0], someone posted it on HN the other day, even if I don’t make any claims of the likelihood it at least tries to portray what it’d look like for the singularity to suddenly happen. [0] https://www.gwern.net/Clippy https://www.gwern.net/Clippy
- skulk 5y ago
  Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is. > HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world. So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability. So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.
  - mkaic 5y ago
    Hmm, as I mentioned in my above comment, I think you might be overlooking the wildcard in all of this that is humanity itself. There is absolutely zero guarantee that only one person/group will discover AI, and zero guarantee that any given group that does discover it won't just release it. Sure, the first people to successfully create AGI might be responsible with it and keep it airgapped, but at some point, someone will purposefully or accidentally let it escape.
mkaic 5y ago
Hi, author here. I think there are at least two plausible ways this could happen -- one, a terrorist/anarchist deliberately does exactly what you joked about (gives the AI everything it needs to go rogue), or two, the AI is supposed to be contained but is clever enough to engineer a way out (think Ava in Ex Machina learning to invert the power flow to cause a generator shutdown). I think the only way you'd be able to be safe would be to completely airgap and immobilize the AI, and give it a killswitch, but even then it could use plausibly psychological manipulation to get a human operator to let it out.
MauranKilom 5y ago
It seems impossible that an AI could trick a human to "let it out of the box". But you might find this interesting: https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment (With sufficiently nasty psychological tactics it is apparently feasible - even for humans - to make someone fail at a game where the only objective is "don't release the AI").
cmsj 5y ago
Your skepticism rests on the assumptions that a) we would recognise AGI when we create it, b) we would be smart enough to contain it. I'm not at all sure that a) is true, and I'm certain that b) is not true. We can't even prevent other dumb humans from breaching our technological fortresses.

by skulk 5y ago

wincy 5y ago

I recently read this[0], someone posted it on HN the other day, even if I don’t make any claims of the likelihood it at least tries to portray what it’d look like for the singularity to suddenly happen. [0] https://www.gwern.net/Clippy https://www.gwern.net/Clippy

skulk 5y ago
Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is. > HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world. So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability. So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.
- mkaic 5y ago
  Hmm, as I mentioned in my above comment, I think you might be overlooking the wildcard in all of this that is humanity itself. There is absolutely zero guarantee that only one person/group will discover AI, and zero guarantee that any given group that does discover it won't just release it. Sure, the first people to successfully create AGI might be responsible with it and keep it airgapped, but at some point, someone will purposefully or accidentally let it escape.

mkaic 5y ago

Hi, author here. I think there are at least two plausible ways this could happen -- one, a terrorist/anarchist deliberately does exactly what you joked about (gives the AI everything it needs to go rogue), or two, the AI is supposed to be contained but is clever enough to engineer a way out (think Ava in Ex Machina learning to invert the power flow to cause a generator shutdown). I think the only way you'd be able to be safe would be to completely airgap and immobilize the AI, and give it a killswitch, but even then it could use plausibly psychological manipulation to get a human operator to let it out.

MauranKilom 5y ago

It seems impossible that an AI could trick a human to "let it out of the box". But you might find this interesting: https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment (With sufficiently nasty psychological tactics it is apparently feasible - even for humans - to make someone fail at a game where the only objective is "don't release the AI").

cmsj 5y ago

Your skepticism rests on the assumptions that a) we would recognise AGI when we create it, b) we would be smart enough to contain it. I'm not at all sure that a) is true, and I'm certain that b) is not true. We can't even prevent other dumb humans from breaching our technological fortresses.