I feel like a bulk of your premise here revolves around never understanding. But, lets say instead of AI, you just imagine some guy Bob being responsible for wh

I feel like a bulk of your premise here revolves around never understanding. But, lets say instead of AI, you just imagine some guy Bob being responsible for whatever you're worried about. Bob is a reasonably intelligent guy, but he's not perfect, he makes an attempt to provide satisfactory explanations of his decisions, he tries to learn and improve, and he might secretly by manipulating the system either for entertainment or malice. If you want to build a useful system around Bob, you have to consider all these limitations - you can't read his mind, you can't predict how he will make decisions in all situations, and you don't have a guarantee that he will give you a satisfactory explanation, or that any explanations are truthful or correct. There's nothing dystopian about this situation - this is the world we live in. You just need to implement some resource constraints, ensure certain checks-and-balances for certain classes of behaviors, implement certain metrics to ensure behavior is according to your expectations, revise metrics and controls whenever you encounter problems, and don't give Bob unfettered control over nukes. What you described isn't dystopia-land material unless you don't consider some very elementary controls that our society already has in place to prevent individuals from destabilizing society.