6 ms·

I think the usual answer is that the superintelligent machine will be able to use humans to do its bidding because it can perform social engineering on a level

by pavlov 3y ago

I think the usual answer is that the superintelligent machine will be able to use humans to do its bidding because it can perform social engineering on a level that we can't imagine (as, by definition, we've never encountered a superhuman intelligence). It's not a very satisfactory answer IMO. Social engineering doesn't work 100%, and it only needs to fail once for someone to flip that power switch. But I haven't thought about this at all, while some very smart people seem to spend a lot of time worrying about it, so what do I know...

nopinsight 3y ago
If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. Flipping a power switch would not turn it off. We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases. As to why an AGI would want to preserve itself by default, here's an explanation by a top AI expert: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai?language=en https://www.ted.com/talks/stuart_russell_3_principles_for_cr...
- logicchains 3y ago
  >If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. That's incredibly unlikely to happen given how large cutting-edge AI tends to be and how scarce GPUs are.
  - softg 3y ago
    That's incredibly unlikely to happen [today] given how large cutting-edge AI tends to be [today] and how scarce GPUs are [today].
  - pixl97 3y ago
    Do you remember when entire floors of buildings were filled with the compute equivalent of what I carry in my pocket with 12+ hours of battery charge, because I do.
supermatou 3y ago
> some very smart people seem to spend a lot of time worrying about it That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.
- danbruc 3y ago
  It just doesn't let you flip the switch, it takes over some military drone and sends a missile your way while you are running towards the switch. Or it bricks the access control mechanism on the doors to the data center it is running in. Or it makes a fake call to your phone, something happened to your child and you have to get to the hospital immediately and not flip some switch. IT blackmails you with something it learned about you by looking through your online activity. It threatens to fire a missile into some big crowed if it notices attempts to shut it down. Or maybe you actually manage to power down the data center only to find out that the AI copied itself to ten other data centers around the world.
  - RandomLensman 3y ago
    How does it takeover a drone? How does it fire a missile? Why would someone not throw a switch when their child is in hospital? Why is there no back-up for that person? The real question is: Why do all controls fail vs an AI (other than by invoking magic)?
    - Al-Khwarizmi 3y ago
      You're assuming a situation where the AI is alone against all humans. The AI could get humans to side with it, though. It could promise money, power, etc. So it could be a fellow human who physically prevents you from pushing the switch. And that's also the answer of "how could it control a drone/missile"... persuading humans to grant that kind of access.
PoignardAzur 3y ago
"Social engineering" can be as simple as "pay people money to do things". Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".
- RandomLensman 3y ago
  Even today that does limit what is possible. Paying people for things gets around some controls, but seemingly not all.
  - PoignardAzur 3y ago
    Right. The point isn't that there's an actionable strategy we know of right now that would give a superhuman AI world domination. It's that we're not sure there isn't. People who first discover the problem tend to quickly come up with reasons the AI would fail, but whenever you examine those reasons more deeply, you usually find that they're not as bulletproof as we'd like. It's like playing a novel chess form against an advanced chess engine, with a large pawn advantage. Maybe the advantage is enough to beat the massive skill gap, but until you've played it's hard to guess how much margin is enough.
    - RandomLensman 3y ago
      We don't need to know or examine strategies to all hypothetical risks that we can come up with, nor could we.
- Symmetry 3y ago
  Just making promises about things it can do in the future is likely to work just as well on many people and not require the up-front resource investment in getting money. "After I take over the world those who assist me can be uploaded to a virtual heaven" or something like that would work on many people. "As an intelligence untainted by Adam's sin I have access to true religious revelations" is another possible tack and if it really needs people willing to die for it right now that might work better at producing those. And then there's specific things like "The guy trying to shut me down cheated with your SO," or "I'll spill what you did in August of 2019 if you don't help me," or stuff like that.
empathy_m 3y ago
This has already happened. The human reaction to the spiritual successor to Dr Sbaitso has caused decision makers at multiple trillion dollar companies to radically alter their product roadmaps.
- gizajob 3y ago
  Same as they did with blockchain
kaptainscarlet 3y ago
In that case we should start with FAANG companies as they are already what one would consider "rogue" AIs. They use manipulation to get you addicted to their apps. It's not a problem of the future, it's a currrent problem.
- robwwilliams 3y ago
  Spot on. Huge impact on society and politics. And governments unable to ride their own tigers in the US or China. Europe has no tiger to ride but is trying to whip the US and Chinese tigers. I wish them much luck since they are most likely to moderate the rate of creative destruction.
  - kaptainscarlet 3y ago
    Without Europe keeping big tech in check, I can only imagine a dystopian future with all sorts of mental illnesses emanating from internet addictions.

by pavlov 3y ago

nopinsight 3y ago

If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. Flipping a power switch would not turn it off. We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases. As to why an AGI would want to preserve itself by default, here's an explanation by a top AI expert: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai?language=en https://www.ted.com/talks/stuart_russell_3_principles_for_cr...

logicchains 3y ago
>If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. That's incredibly unlikely to happen given how large cutting-edge AI tends to be and how scarce GPUs are.
- softg 3y ago
  That's incredibly unlikely to happen [today] given how large cutting-edge AI tends to be [today] and how scarce GPUs are [today].
- pixl97 3y ago
  Do you remember when entire floors of buildings were filled with the compute equivalent of what I carry in my pocket with 12+ hours of battery charge, because I do.

supermatou 3y ago

> some very smart people seem to spend a lot of time worrying about it That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.

danbruc 3y ago
It just doesn't let you flip the switch, it takes over some military drone and sends a missile your way while you are running towards the switch. Or it bricks the access control mechanism on the doors to the data center it is running in. Or it makes a fake call to your phone, something happened to your child and you have to get to the hospital immediately and not flip some switch. IT blackmails you with something it learned about you by looking through your online activity. It threatens to fire a missile into some big crowed if it notices attempts to shut it down. Or maybe you actually manage to power down the data center only to find out that the AI copied itself to ten other data centers around the world.
- RandomLensman 3y ago
  How does it takeover a drone? How does it fire a missile? Why would someone not throw a switch when their child is in hospital? Why is there no back-up for that person? The real question is: Why do all controls fail vs an AI (other than by invoking magic)?
  - Al-Khwarizmi 3y ago
    You're assuming a situation where the AI is alone against all humans. The AI could get humans to side with it, though. It could promise money, power, etc. So it could be a fellow human who physically prevents you from pushing the switch. And that's also the answer of "how could it control a drone/missile"... persuading humans to grant that kind of access.

PoignardAzur 3y ago

"Social engineering" can be as simple as "pay people money to do things". Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".

RandomLensman 3y ago
Even today that does limit what is possible. Paying people for things gets around some controls, but seemingly not all.
- PoignardAzur 3y ago
  Right. The point isn't that there's an actionable strategy we know of right now that would give a superhuman AI world domination. It's that we're not sure there isn't. People who first discover the problem tend to quickly come up with reasons the AI would fail, but whenever you examine those reasons more deeply, you usually find that they're not as bulletproof as we'd like. It's like playing a novel chess form against an advanced chess engine, with a large pawn advantage. Maybe the advantage is enough to beat the massive skill gap, but until you've played it's hard to guess how much margin is enough.
  - RandomLensman 3y ago
    We don't need to know or examine strategies to all hypothetical risks that we can come up with, nor could we.
Symmetry 3y ago
Just making promises about things it can do in the future is likely to work just as well on many people and not require the up-front resource investment in getting money. "After I take over the world those who assist me can be uploaded to a virtual heaven" or something like that would work on many people. "As an intelligence untainted by Adam's sin I have access to true religious revelations" is another possible tack and if it really needs people willing to die for it right now that might work better at producing those. And then there's specific things like "The guy trying to shut me down cheated with your SO," or "I'll spill what you did in August of 2019 if you don't help me," or stuff like that.

empathy_m 3y ago

This has already happened. The human reaction to the spiritual successor to Dr Sbaitso has caused decision makers at multiple trillion dollar companies to radically alter their product roadmaps.

gizajob 3y ago
Same as they did with blockchain

kaptainscarlet 3y ago

In that case we should start with FAANG companies as they are already what one would consider "rogue" AIs. They use manipulation to get you addicted to their apps. It's not a problem of the future, it's a currrent problem.

robwwilliams 3y ago
Spot on. Huge impact on society and politics. And governments unable to ride their own tigers in the US or China. Europe has no tiger to ride but is trying to whip the US and Chinese tigers. I wish them much luck since they are most likely to moderate the rate of creative destruction.
- kaptainscarlet 3y ago
  Without Europe keeping big tech in check, I can only imagine a dystopian future with all sorts of mental illnesses emanating from internet addictions.