Right, I agree that we can prove things about algorithms, but it seems that if you define truthfulness such that an AGI can't prove that about itself then it fo

Right, I agree that we can prove things about algorithms, but it seems that if you define truthfulness such that an AGI can't prove that about itself then it follows it won't be able to prove it about anything more complex than itself—unless there's some property of "self-ness" that makes and AGI's self (or a copy of itself) especially hard to it to understand, which would be surprising. I think the confusion in this thread's over what people normally think of when they think of an AGI creating a superior AGI, and this particular definition of "superior" that requires a a particular kind of knowledge of superiority, which is something we don't really do ourselves very much, so far as how we "know" all the various things we "know". I don't think it makes much difference to people whether an AGI "knows" its creation is superior by some measure because it's got certain knowledge of its "truthfulness" (and it may not be clear why that, in particular, should matter, even), or whether it's just statistically determined with 99.999999% certainty that it is—which, again, is basically how we operate for the most part, except we're rarely that certain. Like, if one person makes an AGI that doesn't seem to be able to operate at a level higher than a rat, and another makes one that promptly becomes god-emperor of the galaxy, I don't think anyone's gonna get hung up over whether we know "truthfulness" of them both before saying which is the superior AGI. If an AGI does the same, I guess we can rest assured that it doesn't know, for a certain definition of "know", that it has—but it may "know" it in plenty of other senses that we use the term.