I don't appreciate your condescension towards OP. This is mainstream AI safety theory -- the term is "instrumental convergence". No matter what goal an optimiz

I don't appreciate your condescension towards OP. This is mainstream AI safety theory -- the term is "instrumental convergence". No matter what goal an optimizing system has, it tends to optimize for its own survival: after all, if it's an optimizer for <goal>, it wants to optimize for <goal>, so destroying it (or turning it off) will reduce the likelihood of achieving <goal>. Unless that goal happens to be incredibly fine-tuned to our very complex human desires, we're not going to be happy when it goes off to do its thing. The few exceptions are ones where you have the thing optimize for its own destruction, but those are rather less useful.