> but I believe humans would not like that idea The very point of this discussion is that humans are bad at anticipating and controlling the consequences of no

> but I believe humans would not like that idea The very point of this discussion is that humans are bad at anticipating and controlling the consequences of novel AIs. We can say "being able to make convincing pornography of anyone without their consent or them even knowing is bad and we shouldn't do that", but the tools to do it are out there and getting more optimized by the month. There's a million different scenarios where a human does upload an unaligned AGI unwittingly. Maybe the human is a random hacker and he uploads the AI on a random server and instructs it "make as much money as you can and send it to me" and doesn't realize the dangers of doing that.