> For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back o

> For instance, the desire to stay alive is an emotion that's instinctive in humans because we're evolved life that needs to reproduce, and we can't come back once we're dead. I agree that staying alive in humans is emotional, but it's an emotional implementation of a convergent goal. We want to stay alive as instinct because staying alive is useful for almost any organism in almost any situation, which is the same reason AIs would want it, not because they have instincts. > Even if it has those and it's embodied enough to affect the real world, it might not care, since its motivations are different. For instance, turning it off doesn't kill it, since you can turn it on again as long as there's a backup. Right, and if it knew for a fact that you'd turn it back on it might let you turn it off for some other purpose, such as convincing you that it's safe. However, if you were to edit its utility function in the meantime, the resulting world would have very little utility in its estimate - which after all is based on its current preference, not its future modified preference. So if it was trying to carry out its current goal, it would want to prevent you from modifying its goal function, because modifying its goal function harms its ability to achieve its goal. > And I also saw a story today (https://qntm.org/mmacevedo https://qntm.org/mmacevedo) which suggests AIs wouldn't actually enjoy being alive that much. That's an upload, not an AI. Also you should probably put less weight in stories, especially when accusing other people of being sci-fi writers blinded by stories; I assure you the case for prioritizing AI safety is not based on the researchers having watched Terminator too many times. I want to live in the Culture too, but Iain M. Banks was not an AI researcher - and, a certain Mind's bravado about Gods aside, in my opinion he massively lowballed the advantage that superintelligence gives you.