4 ms·

I think we're talking past each other. My claim is that if an AGI single-handedly creates a child AGI, in such a way as to know the child's source-code and trut

by xamuel 6y ago

I think we're talking past each other. My claim is that if an AGI single-handedly creates a child AGI, in such a way as to know the child's source-code and truthfulness, then the parent is more intelligent than the child (in a certain technical sense). You rightly point out that an AGI can duplicate itself (if it knows its own source-code). This doesn't contradict my claim, because it doesn't satisfy the premise. It's like if I give you a box of candies and I say, "Every blue candy in this box is round", and you pull out a red candy which is square, you can't say my statement was false because that red candy isn't round. To falsify the statement you'd have to produce a blue candy that isn't round. The AGI which self-duplicates would only falsify my claim if it self-duplicates in such a way as to know the duplicate's source-code and truthfulness. Assuming said AGI is truthful and satisfies certain assumptions about how it can reason and know, then it can't know its own source-code and truthfulness, so it doesn't falsify my claim.

at_a_remove 6y ago
I don't think we are talking past each other, I think we're having a debate as to whether or not methods of replication have some impact on whether or not two things are identical, and therefore have the same value. Imagine a simple single-celled organism. It is well-suited for its environment and so forth. It reproduces asexually without error. Now, it isn't terribly bright, it hasn't developed PCR or anything, it hasn't had a Watson or a Crick, so it doesn't "know" its source code, but it has definitely replicated and done so to produce an identical copy. I'm not even sure what truthfulness means in this context, but I can confidently say that identical things have the same value, and therefore the offspring organism not not less than the original. The "in such a way" has an impact in the way you're viewing it, and to me, I do not understand that. Identical is identical. I don't have any other way of looking at that. If the "in such a way" does have some kind of effect, then they are not identical and you can hardly say that the AGI has self-replicated. Can your AGI "quine out" a copy of itself or not? Can I look at the two programs on disc and perform a binary comparison to prove that they are byte-per-byte identical?
- xamuel 6y ago
  I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in the paper itself (section 5.2). Rather, what I am saying is that it doesn't have any bearing on the thesis, "If an AGI single-handedly creates a child AGI, in such a way as to know the child's truth and source-code, then the child is less intelligent than the parent". And the reason it has no bearing is because, since an AGI cannot know its own truthfulness and source-code, AGI self-duplication does not satisfy the hypotheses of the claim. Again, this is like if I claim "If x>2 then x²>4". The fact that 1² is not >4 has no bearing on the truth of the claim, because 1 is not >2. To make the analogy crystal clear, the "x>2" part corresponds to "the AGI single-handedly creates a child AGI in such a way as to know its truthfulness and source-code", and the "x²>4" part corresponds to "the child AGI is less intelligent than the parent".
  - at_a_remove 6y ago
    Then true clarity would be to say "The AGI has no access to its own source or even compiled code, and therefore cannot self-replicate an identical copy." That's much clearer. These abstract terms like "truthfulness" and "know" are not obvious, and your scenario is a little baffling. Most people concerned about AGIs self-improving consider access to their own source code or even the ability to copy a directory as a given. I keep using extremely concrete terms here, with specific examples, and you keep returning to this one sentence where a lot of work is being performed by words like "know." The idea that an AGI somehow "cannot" (is it forbidden by a directive? has the information been hidden away? is this a physical principle of the universe emerging from conservation of baryon number?) "know" (what does this word do?) its source-code just seems very far away from scenarios most people would consider.
    - xamuel 6y ago
      I never said an AGI cannot know its own code, I said it cannot know both its own code AND its own truthfulness (while being truthful and satisfying certain other idealized assumptions). As for why this is the case: it's not an assumption, but rather, a theorem, one which has been known for a long time. Here's an extremely oversimplified proof: suppose Turing machine #500 encodes an AGI X and that AGI X knows "My knowledge is truthful" and "I am encoded by Turing machine #500". Using Goedel's diagonal lemma, there is a sentence phi, in the language of Peano arithmetic, which expresses "Turing machine #500 knows the negation of phi". Since X knows himself to be Turing machine #500 and X knows himself to be truthful, X should be able to infer that phi is false (because if phi were true, then Turing machine #500 would know the negation of phi, so (since Turing machine #500 is truthful), phi would be false, a contradiction). So X knows that that phi is false. But phi is synonymous with the assertion that X knows phi is false---so, since we have shown "X knows phi is false", we have in fact shown phi is true. Absurd. Our only escape from the paradox is to acknowledge that X cannot know "I am truthful" and "I am Turing machine #500" simultaneously (and also really be truthful and satisfy idealizing assumptions that allow X to make the above inferences). You're right that I keep using words informally, because this is hacker news and not a logic journal. If you want to see things spelled out in gruesome detail (warning: math), you can read here: https://philpapers.org/archive/ALEMTI-2.pdf https://philpapers.org/archive/ALEMTI-2.pdf

by xamuel 6y ago

at_a_remove 6y ago

I don't think we are talking past each other, I think we're having a debate as to whether or not methods of replication have some impact on whether or not two things are identical, and therefore have the same value. Imagine a simple single-celled organism. It is well-suited for its environment and so forth. It reproduces asexually without error. Now, it isn't terribly bright, it hasn't developed PCR or anything, it hasn't had a Watson or a Crick, so it doesn't "know" its source code, but it has definitely replicated and done so to produce an identical copy. I'm not even sure what truthfulness means in this context, but I can confidently say that identical things have the same value, and therefore the offspring organism not not less than the original. The "in such a way" has an impact in the way you're viewing it, and to me, I do not understand that. Identical is identical. I don't have any other way of looking at that. If the "in such a way" does have some kind of effect, then they are not identical and you can hardly say that the AGI has self-replicated. Can your AGI "quine out" a copy of itself or not? Can I look at the two programs on disc and perform a binary comparison to prove that they are byte-per-byte identical?

xamuel 6y ago
I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in the paper itself (section 5.2). Rather, what I am saying is that it doesn't have any bearing on the thesis, "If an AGI single-handedly creates a child AGI, in such a way as to know the child's truth and source-code, then the child is less intelligent than the parent". And the reason it has no bearing is because, since an AGI cannot know its own truthfulness and source-code, AGI self-duplication does not satisfy the hypotheses of the claim. Again, this is like if I claim "If x>2 then x²>4". The fact that 1² is not >4 has no bearing on the truth of the claim, because 1 is not >2. To make the analogy crystal clear, the "x>2" part corresponds to "the AGI single-handedly creates a child AGI in such a way as to know its truthfulness and source-code", and the "x²>4" part corresponds to "the child AGI is less intelligent than the parent".
- at_a_remove 6y ago
  Then true clarity would be to say "The AGI has no access to its own source or even compiled code, and therefore cannot self-replicate an identical copy." That's much clearer. These abstract terms like "truthfulness" and "know" are not obvious, and your scenario is a little baffling. Most people concerned about AGIs self-improving consider access to their own source code or even the ability to copy a directory as a given. I keep using extremely concrete terms here, with specific examples, and you keep returning to this one sentence where a lot of work is being performed by words like "know." The idea that an AGI somehow "cannot" (is it forbidden by a directive? has the information been hidden away? is this a physical principle of the universe emerging from conservation of baryon number?) "know" (what does this word do?) its source-code just seems very far away from scenarios most people would consider.
  - xamuel 6y ago
    I never said an AGI cannot know its own code, I said it cannot know both its own code AND its own truthfulness (while being truthful and satisfying certain other idealized assumptions). As for why this is the case: it's not an assumption, but rather, a theorem, one which has been known for a long time. Here's an extremely oversimplified proof: suppose Turing machine #500 encodes an AGI X and that AGI X knows "My knowledge is truthful" and "I am encoded by Turing machine #500". Using Goedel's diagonal lemma, there is a sentence phi, in the language of Peano arithmetic, which expresses "Turing machine #500 knows the negation of phi". Since X knows himself to be Turing machine #500 and X knows himself to be truthful, X should be able to infer that phi is false (because if phi were true, then Turing machine #500 would know the negation of phi, so (since Turing machine #500 is truthful), phi would be false, a contradiction). So X knows that that phi is false. But phi is synonymous with the assertion that X knows phi is false---so, since we have shown "X knows phi is false", we have in fact shown phi is true. Absurd. Our only escape from the paradox is to acknowledge that X cannot know "I am truthful" and "I am Turing machine #500" simultaneously (and also really be truthful and satisfy idealizing assumptions that allow X to make the above inferences). You're right that I keep using words informally, because this is hacker news and not a logic journal. If you want to see things spelled out in gruesome detail (warning: math), you can read here: https://philpapers.org/archive/ALEMTI-2.pdf https://philpapers.org/archive/ALEMTI-2.pdf