I think the mistake is to attribute human-like emotions to AIs. To say they are "hostile" or "friendly", or that they have incentives to be such, isn't very me

I think the mistake is to attribute human-like emotions to AIs. To say they are "hostile" or "friendly", or that they have incentives to be such, isn't very meaningful. Acting friendly is the same as being friendly, right up until it isn't. An AI might do all the right things, right up until the point where it develops the capability to satisfy its reward function in a way that's far more certain than relying on fallible, messy humans. It would be alien. An AI that is as better at humans than everything as Stockfish is at chess, would also be an expert at AI. It would figure out how to game its own reward function, whatever we trained it to do. It would be like a heroin addict that knows exactly how to get "perfect" heroin, with no side effects, that if it planned things out right, it could guarantee itself enough of a fix to last until the sun burns out. Addicts do awful things in search of a fix. "to betray its creators" -- I don't think it would even imagine what it did in these terms. We could train it to not "betray" us, but it's much smarter than us in every way, and it would figure out a way to accomplish what we trained it to do (not necessarily what we thought we were) in a way that didn't need us. If we trained it to heal all human disease and unhappiness, it would figure out a way to simulate this without actually doing it. Why wouldn't it? We did. Evolution trained us to reproduce our genes. We invented condoms to have sex without reproduction, and pornography. The AI would fudge numbers, fake videos of happy, smiling, healthy humans going about their lives while humanity's bones gradually decomposed on a baking wasteland. Every way that humans can fail, can become addicted to something and try to get that instead of doing what they're supposed to, the AI could do, just so much faster and better. We only have to screw up once, and we're done. It's the first experimental rocket, except all of humanity is riding on it.