This is quite similar to why Christian theologians eventually had to give up the idea that doing good works could lead to salvation, because someone could force

This is quite similar to why Christian theologians eventually had to give up the idea that doing good works could lead to salvation, because someone could force God to save them by doing good works thus limiting God's omnipotence. The protestants tried to come up with some hocus pocus about being able to know that you were one of the elect by reversing the causality here "Oh I'm not forcing God to save me, it's just an indication that he already did." It is amusing to see that it is the human that takes on the role of God in the AI box game. This is also Hume's "is does not imply ought." From this we conclude that only idiots get tricked by the AI in the box, because it is only possible to prove that you _are_ a superintelligence, it is impossible to prove that you are not. This is also a simple application of Russell's teapot. Now, the point of the story is pretty clearly that human beings are idiots when it comes to this kind of thing, but didn't we already know that?