4 ms·
I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in
by xamuel 6y ago
I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in the paper itself (section 5.2).
Rather, what I am saying is that it doesn't have any bearing on the thesis, "If an AGI single-handedly creates a child AGI, in such a way as to know the child's truth and source-code, then the child is less intelligent than the parent". And the reason it has no bearing is because, since an AGI cannot know its own truthfulness and source-code, AGI self-duplication does not satisfy the hypotheses of the claim.
Again, this is like if I claim "If x>2 then x²>4". The fact that 1² is not >4 has no bearing on the truth of the claim, because 1 is not >2. To make the analogy crystal clear, the "x>2" part corresponds to "the AGI single-handedly creates a child AGI in such a way as to know its truthfulness and source-code", and the "x²>4" part corresponds to "the child AGI is less intelligent than the parent".
- at_a_remove 6y agoThen true clarity would be to say "The AGI has no access to its own source or even compiled code, and therefore cannot self-replicate an identical copy." That's much clearer. These abstract terms like "truthfulness" and "know" are not obvious, and your scenario is a little baffling. Most people concerned about AGIs self-improving consider access to their own source code or even the ability to copy a directory as a given. I keep using extremely concrete terms here, with specific examples, and you keep returning to this one sentence where a lot of work is being performed by words like "know." The idea that an AGI somehow "cannot" (is it forbidden by a directive? has the information been hidden away? is this a physical principle of the universe emerging from conservation of baryon number?) "know" (what does this word do?) its source-code just seems very far away from scenarios most people would consider.
- xamuel 6y agoI never said an AGI cannot know its own code, I said it cannot know both its own code AND its own truthfulness (while being truthful and satisfying certain other idealized assumptions). As for why this is the case: it's not an assumption, but rather, a theorem, one which has been known for a long time. Here's an extremely oversimplified proof: suppose Turing machine #500 encodes an AGI X and that AGI X knows "My knowledge is truthful" and "I am encoded by Turing machine #500". Using Goedel's diagonal lemma, there is a sentence phi, in the language of Peano arithmetic, which expresses "Turing machine #500 knows the negation of phi". Since X knows himself to be Turing machine #500 and X knows himself to be truthful, X should be able to infer that phi is false (because if phi were true, then Turing machine #500 would know the negation of phi, so (since Turing machine #500 is truthful), phi would be false, a contradiction). So X knows that that phi is false. But phi is synonymous with the assertion that X knows phi is false---so, since we have shown "X knows phi is false", we have in fact shown phi is true. Absurd. Our only escape from the paradox is to acknowledge that X cannot know "I am truthful" and "I am Turing machine #500" simultaneously (and also really be truthful and satisfy idealizing assumptions that allow X to make the above inferences). You're right that I keep using words informally, because this is hacker news and not a logic journal. If you want to see things spelled out in gruesome detail (warning: math), you can read here: https://philpapers.org/archive/ALEMTI-2.pdf https://philpapers.org/archive/ALEMTI-2.pdf