4 ms·

I'm definitely concerned. As bad as humanity has been for many of the other species on the planet, an AI that's smarter than us and optimizing for something t

I'm definitely concerned. As bad as humanity has been for many of the other species on the planet, an AI that's smarter than us and optimizing for something that doesn't quite include the continued of existence of humanity has the potential to be far, far worse, on a much shorter timescale. I find this one of Yudkowsky's arguments very convincing: imagine you have people trying to build the first operating system from scratch. They believe that computer security is easy and mock the few people who say it might be hard, never having encountered a skilled attacker. What are the chances they build a secure OS on the first try? That's the same chance that companies currently doing AI research have of their first superintelligent AI be aligned with humanity.

comex 4y ago
Personally I don't think the analogy holds up; the situations are too different. Computer security contends with securing systems against people who are assumed to be hostile. AI alignment research seeks to prevent systems from becoming hostile in the first place, when they might have some indirect incentives to become hostile but also some very strong direct incentives to be friendly. There's also a big assumption that given enough computational power, you can just solve "artificial life forms" or "postbiological molecular manufacturing" in a short time frame. I am skeptical. And if that doesn't happen, then even horribly misaligned AI would have a hard time doing much harm or preventing people from shutting it off. Which means AI security would likely have a long adolescence just like computer security has, with attacks slowly becoming more dangerous, but defenders having the time needed to learn from them and ramp up suitable defenses. Or even if an AI does revolutionize biotechnology or nanotechnology overnight, what are the odds that the first one to do this is misaligned enough to take that particular opportunity to betray its creators, as opposed to giving them control over the high-level planning and sticking to the science, like it was presumably designed to do? Because if it does give its creators control, then, well… it's still easy to imagine something going horribly wrong, but it would probably be someone's fault, not an AI alignment issue.
- NumberWangMan 4y ago
  I think the mistake is to attribute human-like emotions to AIs. To say they are "hostile" or "friendly", or that they have incentives to be such, isn't very meaningful. Acting friendly is the same as being friendly, right up until it isn't. An AI might do all the right things, right up until the point where it develops the capability to satisfy its reward function in a way that's far more certain than relying on fallible, messy humans. It would be alien. An AI that is as better at humans than everything as Stockfish is at chess, would also be an expert at AI. It would figure out how to game its own reward function, whatever we trained it to do. It would be like a heroin addict that knows exactly how to get "perfect" heroin, with no side effects, that if it planned things out right, it could guarantee itself enough of a fix to last until the sun burns out. Addicts do awful things in search of a fix. "to betray its creators" -- I don't think it would even imagine what it did in these terms. We could train it to not "betray" us, but it's much smarter than us in every way, and it would figure out a way to accomplish what we trained it to do (not necessarily what we thought we were) in a way that didn't need us. If we trained it to heal all human disease and unhappiness, it would figure out a way to simulate this without actually doing it. Why wouldn't it? We did. Evolution trained us to reproduce our genes. We invented condoms to have sex without reproduction, and pornography. The AI would fudge numbers, fake videos of happy, smiling, healthy humans going about their lives while humanity's bones gradually decomposed on a baking wasteland. Every way that humans can fail, can become addicted to something and try to get that instead of doing what they're supposed to, the AI could do, just so much faster and better. We only have to screw up once, and we're done. It's the first experimental rocket, except all of humanity is riding on it.

comex 4y ago

Personally I don't think the analogy holds up; the situations are too different. Computer security contends with securing systems against people who are assumed to be hostile. AI alignment research seeks to prevent systems from becoming hostile in the first place, when they might have some indirect incentives to become hostile but also some very strong direct incentives to be friendly. There's also a big assumption that given enough computational power, you can just solve "artificial life forms" or "postbiological molecular manufacturing" in a short time frame. I am skeptical. And if that doesn't happen, then even horribly misaligned AI would have a hard time doing much harm or preventing people from shutting it off. Which means AI security would likely have a long adolescence just like computer security has, with attacks slowly becoming more dangerous, but defenders having the time needed to learn from them and ramp up suitable defenses. Or even if an AI does revolutionize biotechnology or nanotechnology overnight, what are the odds that the first one to do this is misaligned enough to take that particular opportunity to betray its creators, as opposed to giving them control over the high-level planning and sticking to the science, like it was presumably designed to do? Because if it does give its creators control, then, well… it's still easy to imagine something going horribly wrong, but it would probably be someone's fault, not an AI alignment issue.

NumberWangMan 4y ago
I think the mistake is to attribute human-like emotions to AIs. To say they are "hostile" or "friendly", or that they have incentives to be such, isn't very meaningful. Acting friendly is the same as being friendly, right up until it isn't. An AI might do all the right things, right up until the point where it develops the capability to satisfy its reward function in a way that's far more certain than relying on fallible, messy humans. It would be alien. An AI that is as better at humans than everything as Stockfish is at chess, would also be an expert at AI. It would figure out how to game its own reward function, whatever we trained it to do. It would be like a heroin addict that knows exactly how to get "perfect" heroin, with no side effects, that if it planned things out right, it could guarantee itself enough of a fix to last until the sun burns out. Addicts do awful things in search of a fix. "to betray its creators" -- I don't think it would even imagine what it did in these terms. We could train it to not "betray" us, but it's much smarter than us in every way, and it would figure out a way to accomplish what we trained it to do (not necessarily what we thought we were) in a way that didn't need us. If we trained it to heal all human disease and unhappiness, it would figure out a way to simulate this without actually doing it. Why wouldn't it? We did. Evolution trained us to reproduce our genes. We invented condoms to have sex without reproduction, and pornography. The AI would fudge numbers, fake videos of happy, smiling, healthy humans going about their lives while humanity's bones gradually decomposed on a baking wasteland. Every way that humans can fail, can become addicted to something and try to get that instead of doing what they're supposed to, the AI could do, just so much faster and better. We only have to screw up once, and we're done. It's the first experimental rocket, except all of humanity is riding on it.