For base desires, that 'agent' is evolution. E.g., we can't will ourselves to stop breathing for the next hour, no matter what our higher-level desire might be.

For base desires, that 'agent' is evolution. E.g., we can't will ourselves to stop breathing for the next hour, no matter what our higher-level desire might be. Many of our higher desires are set for us (taught) by other humans: our parents and society. We wouldn't want an AGI that was able to set all of its own desires, any more than we'd want that in our children. Rather dangerous indeed.