In the case of automated theorem provers, they should generate a certificate together with "true", so their proof can be checked. And that is also the answer to

In the case of automated theorem provers, they should generate a certificate together with "true", so their proof can be checked. And that is also the answer to checking AI: let them generate a proof that what they did was correct. Of course, that only works if you ask the AI for a theorem, but a lot of tasks can be framed that way.