Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQst

Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H... It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.