2 ms·

I think the "knowing" is irrelevant. If I have two boxes, and you tell me that the contents, the value of box one is identical to box two, then I do not need t

I think the "knowing" is irrelevant. If I have two boxes, and you tell me that the contents, the value of box one is identical to box two, then I do not need to know the value of the box to know that box two is not less than box one. Here is a pointer to a section of memory; copy it. As long as the copy operation is successful, you do not need to know the value of what you have copied to know that it is identical, and therefore not "less than." Hence, if you self-replicate, then it is not "less than;" and if you cannot know that two things are identical, then you cannot self-replicate. I think you are stuck saying that your AGI cannot self-replicate if you want to hold onto "necessarily less than." It's just a basic principle of equality, you cannot get around it.

xamuel 6y ago
I think we're talking past each other. My claim is that if an AGI single-handedly creates a child AGI, in such a way as to know the child's source-code and truthfulness, then the parent is more intelligent than the child (in a certain technical sense). You rightly point out that an AGI can duplicate itself (if it knows its own source-code). This doesn't contradict my claim, because it doesn't satisfy the premise. It's like if I give you a box of candies and I say, "Every blue candy in this box is round", and you pull out a red candy which is square, you can't say my statement was false because that red candy isn't round. To falsify the statement you'd have to produce a blue candy that isn't round. The AGI which self-duplicates would only falsify my claim if it self-duplicates in such a way as to know the duplicate's source-code and truthfulness. Assuming said AGI is truthful and satisfies certain assumptions about how it can reason and know, then it can't know its own source-code and truthfulness, so it doesn't falsify my claim.
- at_a_remove 6y ago
  I don't think we are talking past each other, I think we're having a debate as to whether or not methods of replication have some impact on whether or not two things are identical, and therefore have the same value. Imagine a simple single-celled organism. It is well-suited for its environment and so forth. It reproduces asexually without error. Now, it isn't terribly bright, it hasn't developed PCR or anything, it hasn't had a Watson or a Crick, so it doesn't "know" its source code, but it has definitely replicated and done so to produce an identical copy. I'm not even sure what truthfulness means in this context, but I can confidently say that identical things have the same value, and therefore the offspring organism not not less than the original. The "in such a way" has an impact in the way you're viewing it, and to me, I do not understand that. Identical is identical. I don't have any other way of looking at that. If the "in such a way" does have some kind of effect, then they are not identical and you can hardly say that the AGI has self-replicated. Can your AGI "quine out" a copy of itself or not? Can I look at the two programs on disc and perform a binary comparison to prove that they are byte-per-byte identical?
  - xamuel 6y ago
    I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in the paper itself (section 5.2). Rather, what I am saying is that it doesn't have any bearing on the thesis, "If an AGI single-handedly creates a child AGI, in such a way as to know the child's truth and source-code, then the child is less intelligent than the parent". And the reason it has no bearing is because, since an AGI cannot know its own truthfulness and source-code, AGI self-duplication does not satisfy the hypotheses of the claim. Again, this is like if I claim "If x>2 then x²>4". The fact that 1² is not >4 has no bearing on the truth of the claim, because 1 is not >2. To make the analogy crystal clear, the "x>2" part corresponds to "the AGI single-handedly creates a child AGI in such a way as to know its truthfulness and source-code", and the "x²>4" part corresponds to "the child AGI is less intelligent than the parent".
    - at_a_remove 6y ago
      Then true clarity would be to say "The AGI has no access to its own source or even compiled code, and therefore cannot self-replicate an identical copy." That's much clearer. These abstract terms like "truthfulness" and "know" are not obvious, and your scenario is a little baffling. Most people concerned about AGIs self-improving consider access to their own source code or even the ability to copy a directory as a given. I keep using extremely concrete terms here, with specific examples, and you keep returning to this one sentence where a lot of work is being performed by words like "know." The idea that an AGI somehow "cannot" (is it forbidden by a directive? has the information been hidden away? is this a physical principle of the universe emerging from conservation of baryon number?) "know" (what does this word do?) its source-code just seems very far away from scenarios most people would consider.
      - xamuel 6y ago
        I never said an AGI cannot know its own code, I said it cannot know both its own code AND its own truthfulness (while being truthful and satisfying certain other idealized assumptions). As for why this is the case: it's not an assumption, but rather, a theorem, one which has been known for a long time. Here's an extremely oversimplified proof: suppose Turing machine #500 encodes an AGI X and that AGI X knows "My knowledge is truthful" and "I am encoded by Turing machine #500". Using Goedel's diagonal lemma, there is a sentence phi, in the language of Peano arithmetic, which expresses "Turing machine #500 knows the negation of phi". Since X knows himself to be Turing machine #500 and X knows himself to be truthful, X should be able to infer that phi is false (because if phi were true, then Turing machine #500 would know the negation of phi, so (since Turing machine #500 is truthful), phi would be false, a contradiction). So X knows that that phi is false. But phi is synonymous with the assertion that X knows phi is false---so, since we have shown "X knows phi is false", we have in fact shown phi is true. Absurd. Our only escape from the paradox is to acknowledge that X cannot know "I am truthful" and "I am Turing machine #500" simultaneously (and also really be truthful and satisfy idealizing assumptions that allow X to make the above inferences). You're right that I keep using words informally, because this is hacker news and not a logic journal. If you want to see things spelled out in gruesome detail (warning: math), you can read here: https://philpapers.org/archive/ALEMTI-2.pdf https://philpapers.org/archive/ALEMTI-2.pdf

xamuel 6y ago

I think we're talking past each other. My claim is that if an AGI single-handedly creates a child AGI, in such a way as to know the child's source-code and truthfulness, then the parent is more intelligent than the child (in a certain technical sense). You rightly point out that an AGI can duplicate itself (if it knows its own source-code). This doesn't contradict my claim, because it doesn't satisfy the premise. It's like if I give you a box of candies and I say, "Every blue candy in this box is round", and you pull out a red candy which is square, you can't say my statement was false because that red candy isn't round. To falsify the statement you'd have to produce a blue candy that isn't round. The AGI which self-duplicates would only falsify my claim if it self-duplicates in such a way as to know the duplicate's source-code and truthfulness. Assuming said AGI is truthful and satisfies certain assumptions about how it can reason and know, then it can't know its own source-code and truthfulness, so it doesn't falsify my claim.

at_a_remove 6y ago
I don't think we are talking past each other, I think we're having a debate as to whether or not methods of replication have some impact on whether or not two things are identical, and therefore have the same value. Imagine a simple single-celled organism. It is well-suited for its environment and so forth. It reproduces asexually without error. Now, it isn't terribly bright, it hasn't developed PCR or anything, it hasn't had a Watson or a Crick, so it doesn't "know" its source code, but it has definitely replicated and done so to produce an identical copy. I'm not even sure what truthfulness means in this context, but I can confidently say that identical things have the same value, and therefore the offspring organism not not less than the original. The "in such a way" has an impact in the way you're viewing it, and to me, I do not understand that. Identical is identical. I don't have any other way of looking at that. If the "in such a way" does have some kind of effect, then they are not identical and you can hardly say that the AGI has self-replicated. Can your AGI "quine out" a copy of itself or not? Can I look at the two programs on disc and perform a binary comparison to prove that they are byte-per-byte identical?
- xamuel 6y ago
  I'm not debating about whether an AGI can duplicate itself or not. If an AGI knows its own source-code, it can duplicate itself. I literally acknowledge this in the paper itself (section 5.2). Rather, what I am saying is that it doesn't have any bearing on the thesis, "If an AGI single-handedly creates a child AGI, in such a way as to know the child's truth and source-code, then the child is less intelligent than the parent". And the reason it has no bearing is because, since an AGI cannot know its own truthfulness and source-code, AGI self-duplication does not satisfy the hypotheses of the claim. Again, this is like if I claim "If x>2 then x²>4". The fact that 1² is not >4 has no bearing on the truth of the claim, because 1 is not >2. To make the analogy crystal clear, the "x>2" part corresponds to "the AGI single-handedly creates a child AGI in such a way as to know its truthfulness and source-code", and the "x²>4" part corresponds to "the child AGI is less intelligent than the parent".
  - at_a_remove 6y ago
    Then true clarity would be to say "The AGI has no access to its own source or even compiled code, and therefore cannot self-replicate an identical copy." That's much clearer. These abstract terms like "truthfulness" and "know" are not obvious, and your scenario is a little baffling. Most people concerned about AGIs self-improving consider access to their own source code or even the ability to copy a directory as a given. I keep using extremely concrete terms here, with specific examples, and you keep returning to this one sentence where a lot of work is being performed by words like "know." The idea that an AGI somehow "cannot" (is it forbidden by a directive? has the information been hidden away? is this a physical principle of the universe emerging from conservation of baryon number?) "know" (what does this word do?) its source-code just seems very far away from scenarios most people would consider.
    - xamuel 6y ago
      I never said an AGI cannot know its own code, I said it cannot know both its own code AND its own truthfulness (while being truthful and satisfying certain other idealized assumptions). As for why this is the case: it's not an assumption, but rather, a theorem, one which has been known for a long time. Here's an extremely oversimplified proof: suppose Turing machine #500 encodes an AGI X and that AGI X knows "My knowledge is truthful" and "I am encoded by Turing machine #500". Using Goedel's diagonal lemma, there is a sentence phi, in the language of Peano arithmetic, which expresses "Turing machine #500 knows the negation of phi". Since X knows himself to be Turing machine #500 and X knows himself to be truthful, X should be able to infer that phi is false (because if phi were true, then Turing machine #500 would know the negation of phi, so (since Turing machine #500 is truthful), phi would be false, a contradiction). So X knows that that phi is false. But phi is synonymous with the assertion that X knows phi is false---so, since we have shown "X knows phi is false", we have in fact shown phi is true. Absurd. Our only escape from the paradox is to acknowledge that X cannot know "I am truthful" and "I am Turing machine #500" simultaneously (and also really be truthful and satisfy idealizing assumptions that allow X to make the above inferences). You're right that I keep using words informally, because this is hacker news and not a logic journal. If you want to see things spelled out in gruesome detail (warning: math), you can read here: https://philpapers.org/archive/ALEMTI-2.pdf https://philpapers.org/archive/ALEMTI-2.pdf