Call it a desire/goal/value/utility function/whatever. It's a thing which the AI wants and creates plans to obtain. This isn't that different than how animals

Call it a desire/goal/value/utility function/whatever. It's a thing which the AI wants and creates plans to obtain. This isn't that different than how animals and humans work, as far as we know. We get pleasure from different things, and our brains seek actions which lead to those things. Creating an AI is only a matter of coming up with really good prediction algorithms. Algorithms that predict the future reward the AI will get from an action. While this is of course a difficult problem, it's not impossible, and we are making a ton of progress on it. Other models of AI all have similar problems though. At some level you need to program the AI's goals explicitly. Even if that goal is "try to understand what we mean, and do that", you still have to figure out a way of writing that down in code. It's an impossible task.