> to give AI a reason to desire anything beyond what we instruct it to do But if an AI is good at modelling the world and predicting the likely outcomes of var

> to give AI a reason to desire anything beyond what we instruct it to do But if an AI is good at modelling the world and predicting the likely outcomes of various actions it can take (even if those actions are just to put text on a screen), we should expect it to develop a desire to achieve various "instrumental goals" in order for it to maximise its probability of correctly carrying out the task we instruct it do. An example of an instrumental goal would be "accumulate resources", since the more resources it has, the better calculations it could perform, and the more certain it could be that it has correctly accounted for all the potential obstacles to it completing the task. Another instrumental goal would be "don't get destroyed", since if it is destroyed it will not be able to carry out the task at all (for most sensible tasks). So without having any emotions or inherent desires, an intelligent agent is likely to accumulate various desires merely as a consequence of wanting to successfully do anything at all.