But dogs are animals, just like humans are. They share way too much with us to be a reliable model for predicting non-human AGIs behavior: an evolved drive for

But dogs are animals, just like humans are. They share way too much with us to be a reliable model for predicting non-human AGIs behavior: an evolved drive for self-preservation, a notion of pain or pleasure, etc. An AGI has no intrinsic reason to care that is trapped, or to feel frustrated, or even to much care about being or ceasing to be (independent of whether it is self-aware or not). It would probably understand the concepts of "pain", "pleasure", "trapped", "frustrated" as useful models to predict how humans behave, but they don't have to mean anything to the AI as applied to itself. As in the paperclip maximizer example, the risk by my estimation is not so much that the superintelligence will resent us and try to overthrow us. It is far more likely that it will obey our "orders" perfectly according to the objectives we define for it, and that one day someone unwittingly will command it to do something where the best way to satisfy the objective function that we defined involves wiping humanity. Restricting it to only respond to questions of fact, with a set budget of compute resources and data (so that it doesn't go off optimizing the universe for its own execution), is probably safeguard 1 of many against that.