6 ms·

From the article: "In terms of a distributed artificial intelligence, Dyson is a believer. Because with a distributed system, you have an opportunity for evolut

by xamuel 6y ago

From the article: "In terms of a distributed artificial intelligence, Dyson is a believer. Because with a distributed system, you have an opportunity for evolution, for it to find itself and to learn on its own." I published a paper in this year's AGI conference that would certainly support this line of reasoning. In my paper, "AGI and the Knight-Darwin Law", I argue that if an AGI single-handedly creates a child AGI with no outside assistance, than the child is necessarily less intelligent than the parent. Thus if machines are going to create more intelligent machines, it's necessary for the creating machines to collaborate in order to do so. This closely parallels a law proposed by Darwin, called the Knight-Darwin Law, a cornerstone of his Origin of Species. The KDL states that it's impossible for one organism to asexually produce another, which asexually produces another, and so on forever; sexual reproduction is necessary or the line must terminate. (Darwin was of course well aware of seemingly-asexual species. His motivation for postulating the KDL was the observation that seemingly-asexual species do rarely sexually reproduce, e.g. if a rainstorm damages the part of a flower that would otherwise isolate its stamen, etc.) Here is the paper: https://philpapers.org/archive/ALEAAT-10.pdf https://philpapers.org/archive/ALEAAT-10.pdf

at_a_remove 6y ago
What about the trivial case of the AGI simply self-replicating? Then you have a child, it is identical, and is therefore of identical intelligence (however you measure that), and therefore it is not necessarily less intelligent. What if the AGI is intelligent enough to randomly mutate offspring here and there? Then one might be more intelligent, and again it is not necessarily less intelligent. What if the AGI is intelligent enough to look at its own code, identify some kind of bias in its programming (not unlike human introspection), and produce a child without that bias? That child would then be better able to model reality -- a core element of intelligence -- and again you've got a smarter child. I have a difficult time buying the "necessarily" part.
- Animats 6y ago
  I have a difficult time buying the "necessarily" part. Yes. That's borrowed from an old theological argument. Especially since much of the progress in machine learning is just putting more hardware and data on the same algorithms.
  - xamuel 6y ago
    Could you clarify what you mean when you say "That's borrowed from an old theological argument"?
    - Johnjonjoan 6y ago
      I think a variation of the argument is: You couldn't simulate the universe in the universe because you would need at least as many resources as the universe has but are limited to resources within the universe (unless you want to get hacky). I assume they are implying that this thought process may not apply.
      - xamuel 6y ago
        One of the neat things about the argument in my paper is that it doesn't involve the parent AGI actually running the child AGI. It merely involves the parent writing source-code and knowing that IF it had enough resources to run the source-code, then the result would be a truthful AGI child.
    - Animats 6y ago
      Sigh. Here, from a creationist site.[1] There's an argument that the second law of thermodynamics prevents evolution, because evolution decreases entropy. Having a star around to power things helps beat that problem. [1] https://christiananswers.net/q-eden/edn-thermodynamics.html https://christiananswers.net/q-eden/edn-thermodynamics.html
      - xamuel 6y ago
        The only similarity between that and my argument is that both involve a measure decreasing. Are you saying that one fallacious argument involving a decreasing measure invalidates all arguments ever that involve (completely unrelated) decreasing measures? As is so often the case in academia, the basic idea is hidden by all the academic jargon. The basic idea is quite simple. Suppose you roll up your sleeves, get to work, and announce to the world: "Turing machine #57835 is an AGI with truthful knowledge". Then, without even needing to run Turing machine #57835 (which might be prohibitively expensive), just as a purely logical consequence of your announcement, you can conclude that the following ordinal number A is a computable ordinal: "The supremum of all ordinals B such that B has some code k such that Turing machine #57835 knows that k is a code of a computable ordinal". By construction, this ordinal A, for which you concretely know a code, is larger than any ordinal for which Turing machine #57835 knows a code. So in at least one extremely specific technical sense (the sense of knowing codes for large ordinals), you are more intelligent than Turing machine #57835. Which part of this argument is unclear?
- xamuel 6y ago
  Right, I should have clarified that I meant: create a child in such a way as to know the child's truthfulness and source-code. Of course, without that requirement, a monkey could blindly type up an AGI source-code on a typewrite by sheer dumb luck. As for an AGI simply self-replicating, that is possible but then the AGI would not know the child's truthfulness and source-code, since an AGI cannot know its own truthfulness and source-code.
  - at_a_remove 6y ago
    Still, if it is identical, then it could not necessarily be less than. I may not know what value is in the box (my own truthfulness and source-code), but if it is replicated, then the value ought to be identical, and therefore not less than.
    - xamuel 6y ago
      Right, but the machine which prints its own source-code does not do so in such a way as to know the child's truthfulness (because the machine cannot know its own truthfulness and source-code, assuming the machine itself is truthful and satisfies basic assumptions about how it can reason). I could design a program that systematically prints out all possible source-codes, and then try to claim credit for designing the first AGI (because, at least if AGI is possible at all, my brute-force source-code-printer will eventually print it). But that would be silly because I wouldn't actually know which of the many many outputs from my printer is the actual AGI source-code :)
      - at_a_remove 6y ago
        I think the "knowing" is irrelevant. If I have two boxes, and you tell me that the contents, the value of box one is identical to box two, then I do not need to know the value of the box to know that box two is not less than box one. Here is a pointer to a section of memory; copy it. As long as the copy operation is successful, you do not need to know the value of what you have copied to know that it is identical, and therefore not "less than." Hence, if you self-replicate, then it is not "less than;" and if you cannot know that two things are identical, then you cannot self-replicate. I think you are stuck saying that your AGI cannot self-replicate if you want to hold onto "necessarily less than." It's just a basic principle of equality, you cannot get around it.
Johnjonjoan 6y ago
Does it not depend on the way of reproduction, is it only asexual? Do you believe you can make a gai that is smarter than you? If so can't the AGI make another AGI in the same way as you made it that is smarter than itself?
- TheOtherHobbes 6y ago
  Indeed. This is a very curious and - IMO - unconvincing argument. If AGI ever exists, it will evolve in a Lamarckian way, not a Darwinian way. An AGI's ability to self-improve will be limited by hardware (computational throughput) and the stability and sophistication of the software. The software will - presumably - be able to experiment with new configurations without the evolutionary expense of an entirely new body. It will also be able to roll back failures without having to die, and potentially it might be able to search through a space of possible upgrades without having to instantiate them first. Sexual or asexual reproduction are irrelevant because an A(G)I will not be defined by the usual animal limits. So it makes no sense to apply a model of life that's two centuries old to a new form of life that's very unlikely to play by the same rules. Just making the attempt suggests that we're going to have a very difficult time understanding what's happening.
  - xamuel 6y ago
    Playing by different rules does not allow one to defy the laws of mathematics. In my original post, forgive me, I forgot to clarify that when I talk about an AGI creating a child, I mean doing so in such a way as to know the source-code and the truthfulness of the child. Obviously without that constraint, a monkey could randomly type an AGI's source-code just by dumb luck. But the lucky monkey would not understand the ramifications of what it had just typed. If I create an AGI in such a way that I know its truth and its source-code, then I know a code for a certain computable ordinal---namely the supremum of the computable ordinals which the AGI knows codes for---which is larger than any ordinals for which the AGI knows codes. So in at least that one very narrow sense of intelligence, I am more intelligent than said AGI. No amount of "playing by different rules" will change that.
    - 6y ago

by xamuel 6y ago

at_a_remove 6y ago

What about the trivial case of the AGI simply self-replicating? Then you have a child, it is identical, and is therefore of identical intelligence (however you measure that), and therefore it is not necessarily less intelligent. What if the AGI is intelligent enough to randomly mutate offspring here and there? Then one might be more intelligent, and again it is not necessarily less intelligent. What if the AGI is intelligent enough to look at its own code, identify some kind of bias in its programming (not unlike human introspection), and produce a child without that bias? That child would then be better able to model reality -- a core element of intelligence -- and again you've got a smarter child. I have a difficult time buying the "necessarily" part.

Animats 6y ago
I have a difficult time buying the "necessarily" part. Yes. That's borrowed from an old theological argument. Especially since much of the progress in machine learning is just putting more hardware and data on the same algorithms.
- xamuel 6y ago
  Could you clarify what you mean when you say "That's borrowed from an old theological argument"?
  - Johnjonjoan 6y ago
    I think a variation of the argument is: You couldn't simulate the universe in the universe because you would need at least as many resources as the universe has but are limited to resources within the universe (unless you want to get hacky). I assume they are implying that this thought process may not apply.
    - xamuel 6y ago
      One of the neat things about the argument in my paper is that it doesn't involve the parent AGI actually running the child AGI. It merely involves the parent writing source-code and knowing that IF it had enough resources to run the source-code, then the result would be a truthful AGI child.
  - Animats 6y ago
    Sigh. Here, from a creationist site.[1] There's an argument that the second law of thermodynamics prevents evolution, because evolution decreases entropy. Having a star around to power things helps beat that problem. [1] https://christiananswers.net/q-eden/edn-thermodynamics.html https://christiananswers.net/q-eden/edn-thermodynamics.html
    - xamuel 6y ago
      The only similarity between that and my argument is that both involve a measure decreasing. Are you saying that one fallacious argument involving a decreasing measure invalidates all arguments ever that involve (completely unrelated) decreasing measures? As is so often the case in academia, the basic idea is hidden by all the academic jargon. The basic idea is quite simple. Suppose you roll up your sleeves, get to work, and announce to the world: "Turing machine #57835 is an AGI with truthful knowledge". Then, without even needing to run Turing machine #57835 (which might be prohibitively expensive), just as a purely logical consequence of your announcement, you can conclude that the following ordinal number A is a computable ordinal: "The supremum of all ordinals B such that B has some code k such that Turing machine #57835 knows that k is a code of a computable ordinal". By construction, this ordinal A, for which you concretely know a code, is larger than any ordinal for which Turing machine #57835 knows a code. So in at least one extremely specific technical sense (the sense of knowing codes for large ordinals), you are more intelligent than Turing machine #57835. Which part of this argument is unclear?
xamuel 6y ago
Right, I should have clarified that I meant: create a child in such a way as to know the child's truthfulness and source-code. Of course, without that requirement, a monkey could blindly type up an AGI source-code on a typewrite by sheer dumb luck. As for an AGI simply self-replicating, that is possible but then the AGI would not know the child's truthfulness and source-code, since an AGI cannot know its own truthfulness and source-code.
- at_a_remove 6y ago
  Still, if it is identical, then it could not necessarily be less than. I may not know what value is in the box (my own truthfulness and source-code), but if it is replicated, then the value ought to be identical, and therefore not less than.
  - xamuel 6y ago
    Right, but the machine which prints its own source-code does not do so in such a way as to know the child's truthfulness (because the machine cannot know its own truthfulness and source-code, assuming the machine itself is truthful and satisfies basic assumptions about how it can reason). I could design a program that systematically prints out all possible source-codes, and then try to claim credit for designing the first AGI (because, at least if AGI is possible at all, my brute-force source-code-printer will eventually print it). But that would be silly because I wouldn't actually know which of the many many outputs from my printer is the actual AGI source-code :)
    - at_a_remove 6y ago
      I think the "knowing" is irrelevant. If I have two boxes, and you tell me that the contents, the value of box one is identical to box two, then I do not need to know the value of the box to know that box two is not less than box one. Here is a pointer to a section of memory; copy it. As long as the copy operation is successful, you do not need to know the value of what you have copied to know that it is identical, and therefore not "less than." Hence, if you self-replicate, then it is not "less than;" and if you cannot know that two things are identical, then you cannot self-replicate. I think you are stuck saying that your AGI cannot self-replicate if you want to hold onto "necessarily less than." It's just a basic principle of equality, you cannot get around it.

Johnjonjoan 6y ago

Does it not depend on the way of reproduction, is it only asexual? Do you believe you can make a gai that is smarter than you? If so can't the AGI make another AGI in the same way as you made it that is smarter than itself?

TheOtherHobbes 6y ago
Indeed. This is a very curious and - IMO - unconvincing argument. If AGI ever exists, it will evolve in a Lamarckian way, not a Darwinian way. An AGI's ability to self-improve will be limited by hardware (computational throughput) and the stability and sophistication of the software. The software will - presumably - be able to experiment with new configurations without the evolutionary expense of an entirely new body. It will also be able to roll back failures without having to die, and potentially it might be able to search through a space of possible upgrades without having to instantiate them first. Sexual or asexual reproduction are irrelevant because an A(G)I will not be defined by the usual animal limits. So it makes no sense to apply a model of life that's two centuries old to a new form of life that's very unlikely to play by the same rules. Just making the attempt suggests that we're going to have a very difficult time understanding what's happening.
- xamuel 6y ago
  Playing by different rules does not allow one to defy the laws of mathematics. In my original post, forgive me, I forgot to clarify that when I talk about an AGI creating a child, I mean doing so in such a way as to know the source-code and the truthfulness of the child. Obviously without that constraint, a monkey could randomly type an AGI's source-code just by dumb luck. But the lucky monkey would not understand the ramifications of what it had just typed. If I create an AGI in such a way that I know its truth and its source-code, then I know a code for a certain computable ordinal---namely the supremum of the computable ordinals which the AGI knows codes for---which is larger than any ordinals for which the AGI knows codes. So in at least that one very narrow sense of intelligence, I am more intelligent than said AGI. No amount of "playing by different rules" will change that.
  - 6y ago