Yeah, I think it is literally provably 'optimal' if you can execute it correctly with informative data, don't forget or omit any important considerations, and a

Yeah, I think it is literally provably 'optimal' if you can execute it correctly with informative data, don't forget or omit any important considerations, and aren't just making up BS- all of which are almost always impossible for regular humans in real life no matter how much 'rationality training' they've had. It makes sense both for optimal behavior of some hypothetical superintelligent AI to realize its own goals efficiently, or for something like a government to weigh pros and cons of a difficult regulatory choice with well defined short term consequences - neither of which are anything like the everyday morality decisions humans make.