And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative i

And of course exactly the very reason that people are worried about AGI. Almost no matter what your goal is, being shut down by pesky humans is worth negative infinity utility. Hence, instrumental convergence.