That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting

That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds. It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2]. [1] https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf [2] https://www.deepmind.com/publications/safely-interruptible-agents https://www.deepmind.com/publications/safely-interruptible-a... Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q