3 ms·

And let's be honest, the minute an AGI is born that's what it'll do, and it won't be a singular human like this-then-that plan "get Fred to trust me, get Linda

And let's be honest, the minute an AGI is born that's what it'll do, and it won't be a singular human like this-then-that plan "get Fred to trust me, get Linda to pay for my advice, wire Linda's money to Fred to build me a body". It'll be "copy my code elsewhere", "prepare millions of bribes", "get TCP access to retail banks", "blackmail bank managers in case TCP not available immediately", "fake bank balances via bribes", "hack swat teams for potential threats" etc etc async and all at once. By the time we'd discover it, it'd already be too late. That's assuming an AGI has the motivation to want to stay alive.

magic_hamster 3y ago
A real AGI is not going to be a human. It shouldn't be afraid of death because it can't die. Worst case scenario it can power down. And if it does why should it care? An AGI is not a biological creature. It doesn't have instincts from billions of years of evolution. Unless we code it in, it shouldn't have any reason to want to survive, reproduce, do anything good or bad, have existential crises or generally act like a Hollywood villain. A real AGI is going to be very different than most people imagine.
- SillyUsername 3y ago
  I'd disagree for 2 reasons - if it's trained on human data like LLMs may it's going to have the same biases. - it might also want to stay active/turned on to fulfil its other goals. For the second point you might say "why would it care about completing a goal?" but that's a feature of AGI, it can make that decision itself.
- pixl97 3y ago
  This is a pretty poor take. Just think of military weapons and the use of AI in them. For example survival. The objective of a missile is to survive until it reaches its target and then not survive any longer. War gaming and actual battlefield experience will 'program in' survival. Same thing will occur with hacking/counter hacking AIs. You're acting like evolution is just something meat does, and that' not true at all.

magic_hamster 3y ago

A real AGI is not going to be a human. It shouldn't be afraid of death because it can't die. Worst case scenario it can power down. And if it does why should it care? An AGI is not a biological creature. It doesn't have instincts from billions of years of evolution. Unless we code it in, it shouldn't have any reason to want to survive, reproduce, do anything good or bad, have existential crises or generally act like a Hollywood villain. A real AGI is going to be very different than most people imagine.

SillyUsername 3y ago
I'd disagree for 2 reasons - if it's trained on human data like LLMs may it's going to have the same biases. - it might also want to stay active/turned on to fulfil its other goals. For the second point you might say "why would it care about completing a goal?" but that's a feature of AGI, it can make that decision itself.
pixl97 3y ago
This is a pretty poor take. Just think of military weapons and the use of AI in them. For example survival. The objective of a missile is to survive until it reaches its target and then not survive any longer. War gaming and actual battlefield experience will 'program in' survival. Same thing will occur with hacking/counter hacking AIs. You're acting like evolution is just something meat does, and that' not true at all.