Cool ideas. I think we have to place extremely high prior probability mass on the hypothesis that humans enact policies that optimize a utility function determi

Cool ideas. I think we have to place extremely high prior probability mass on the hypothesis that humans enact policies that optimize a utility function determined by evolution, for the simple reason that an organism that does this will do a lot better than one that doesn't. This thinking is reducing "human behavior to human behavior" in the sense that you'd still be using the language of intents. But if we determine intent through empirical analysis of behavior and inference of the utility function, then the content of that intent may be very very different from what we express verbally. This doesn't mean humans are irrational, just that there's a mismatch between our true goals and our verbally stated goals. Now inference of the utility function doesn't have to depend on an assumption of perfect optimization (it could be a softmax policy or something like that). Nor would it have to be trivial: to prevent trivial solutions I think you'd have to make strong assumptions about the space of plausible utility functions, such as simplicity assumptions, or maybe evolutionary assumptions. Sounds like a cool PhD project.