> without any specifics of what works for an implementation As I understand it, if you assume that your agent is rational in certain basic ways, for instance t

> without any specifics of what works for an implementation As I understand it, if you assume that your agent is rational in certain basic ways, for instance that it has ordered, rather than circular preferences, or it can't be Dutch booked/money pumped, it can modelled as having a utility function. Note that this is different from assuming that an explicit utility function will be programmed in, rather the basic level of rationality implies it into existence. Once you know that an agent has a utility function, you can use that to do a fair amount of reasoning about its reasoning. Better (and less briefly) explained here: https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/#coherent-decisions-imply-a-utility-function https://intelligence.org/2016/12/28/ai-alignment-why-its-har...