4 ms·

Does it not depend on the way of reproduction, is it only asexual? Do you believe you can make a gai that is smarter than you? If so can't the AGI make another

Does it not depend on the way of reproduction, is it only asexual? Do you believe you can make a gai that is smarter than you? If so can't the AGI make another AGI in the same way as you made it that is smarter than itself?

TheOtherHobbes 6y ago
Indeed. This is a very curious and - IMO - unconvincing argument. If AGI ever exists, it will evolve in a Lamarckian way, not a Darwinian way. An AGI's ability to self-improve will be limited by hardware (computational throughput) and the stability and sophistication of the software. The software will - presumably - be able to experiment with new configurations without the evolutionary expense of an entirely new body. It will also be able to roll back failures without having to die, and potentially it might be able to search through a space of possible upgrades without having to instantiate them first. Sexual or asexual reproduction are irrelevant because an A(G)I will not be defined by the usual animal limits. So it makes no sense to apply a model of life that's two centuries old to a new form of life that's very unlikely to play by the same rules. Just making the attempt suggests that we're going to have a very difficult time understanding what's happening.
- xamuel 6y ago
  Playing by different rules does not allow one to defy the laws of mathematics. In my original post, forgive me, I forgot to clarify that when I talk about an AGI creating a child, I mean doing so in such a way as to know the source-code and the truthfulness of the child. Obviously without that constraint, a monkey could randomly type an AGI's source-code just by dumb luck. But the lucky monkey would not understand the ramifications of what it had just typed. If I create an AGI in such a way that I know its truth and its source-code, then I know a code for a certain computable ordinal---namely the supremum of the computable ordinals which the AGI knows codes for---which is larger than any ordinals for which the AGI knows codes. So in at least that one very narrow sense of intelligence, I am more intelligent than said AGI. No amount of "playing by different rules" will change that.
  - spanhandler 6y ago
    > Playing by different rules does not allow one to defy the laws of mathematics. OK, but the rules in the paper seem pretty arbitrary and unintuitive to me, so by picking different rules one is no longer defying anything. In particular the requirements for exceptionally perfectly detailed knowledge of the child seem designed to make the conclusion tautologically true, but also not very interesting, as far as I can tell. "If I define impossible requirements, [thing] is impossible". OK, that's... fine, I guess. Allowing for "trust" between AGIs to overcome issues with their properly knowing one another's "truthfulness" but not allowing the same shortcut when it comes time to judge their knowledge of the child's "truthfulness" especially seems odd. I'm also not sure how all this perfect knowledge is justified before we allow that an AGI has created a superior AGI—it seems impossible for any system, biological or AGI, so I'm not sure what use the conclusion is. "Perfect play always results in cat, in tic-tac-toe, so you cannot win this game" well, that's nice but we're talking about poker...
    - xamuel 6y ago
      Thanks for the feedback. Regarding "impossible requirements", does it really go without saying that knowing an AGI's truthfulness is so impossible? I could write a computer program which does nothing but print "1+1=2", "1+2=3", "1+3=4", ... onto the screen, and know (with "exceptionally perfectly detailed knowledge") everything about this program and know that the things it prints on the screen are truthful. So it's not that it's inherently impossible to have such detailed knowledge about just any computer program. What makes AGI special? Why is it that we should take it for granted that it's impossible to know the truthfulness of an AGI's knowledge? I'm not saying that's not impossible (indeed, by tweaking the assumptions, the argument in my paper could be considered an argument for said impossibility), I'm just trying to understand why you think it's so blatantly obvious that knowing the truthfulness of the AGI's knowledge is impossible.
      - spanhandler 6y ago
        Right, I agree that we can prove things about algorithms, but it seems that if you define truthfulness such that an AGI can't prove that about itself then it follows it won't be able to prove it about anything more complex than itself—unless there's some property of "self-ness" that makes and AGI's self (or a copy of itself) especially hard to it to understand, which would be surprising. I think the confusion in this thread's over what people normally think of when they think of an AGI creating a superior AGI, and this particular definition of "superior" that requires a a particular kind of knowledge of superiority, which is something we don't really do ourselves very much, so far as how we "know" all the various things we "know". I don't think it makes much difference to people whether an AGI "knows" its creation is superior by some measure because it's got certain knowledge of its "truthfulness" (and it may not be clear why that, in particular, should matter, even), or whether it's just statistically determined with 99.999999% certainty that it is—which, again, is basically how we operate for the most part, except we're rarely that certain. Like, if one person makes an AGI that doesn't seem to be able to operate at a level higher than a rat, and another makes one that promptly becomes god-emperor of the galaxy, I don't think anyone's gonna get hung up over whether we know "truthfulness" of them both before saying which is the superior AGI. If an AGI does the same, I guess we can rest assured that it doesn't know, for a certain definition of "know", that it has—but it may "know" it in plenty of other senses that we use the term.
xamuel 6y ago
Sorry, I forgot to clarify that when I talk about the AGI creating a child, I mean the AGI creating a child in such a way as to know the child's truthfulness and source-code. Otherwise, of course, a monkey could "create" an AGI by blindly typing source-code on a typewrite and getting lucky. >Do you believe you can make a gai that is smarter than you? There are different ways of measuring intelligence. For the measure addressed in the paper, no, I can't single-handedly create an AGI smarter than myself (at least not in such a way as to know its truthfulness and its source-code).

TheOtherHobbes 6y ago

Indeed. This is a very curious and - IMO - unconvincing argument. If AGI ever exists, it will evolve in a Lamarckian way, not a Darwinian way. An AGI's ability to self-improve will be limited by hardware (computational throughput) and the stability and sophistication of the software. The software will - presumably - be able to experiment with new configurations without the evolutionary expense of an entirely new body. It will also be able to roll back failures without having to die, and potentially it might be able to search through a space of possible upgrades without having to instantiate them first. Sexual or asexual reproduction are irrelevant because an A(G)I will not be defined by the usual animal limits. So it makes no sense to apply a model of life that's two centuries old to a new form of life that's very unlikely to play by the same rules. Just making the attempt suggests that we're going to have a very difficult time understanding what's happening.

xamuel 6y ago
Playing by different rules does not allow one to defy the laws of mathematics. In my original post, forgive me, I forgot to clarify that when I talk about an AGI creating a child, I mean doing so in such a way as to know the source-code and the truthfulness of the child. Obviously without that constraint, a monkey could randomly type an AGI's source-code just by dumb luck. But the lucky monkey would not understand the ramifications of what it had just typed. If I create an AGI in such a way that I know its truth and its source-code, then I know a code for a certain computable ordinal---namely the supremum of the computable ordinals which the AGI knows codes for---which is larger than any ordinals for which the AGI knows codes. So in at least that one very narrow sense of intelligence, I am more intelligent than said AGI. No amount of "playing by different rules" will change that.
- spanhandler 6y ago
  > Playing by different rules does not allow one to defy the laws of mathematics. OK, but the rules in the paper seem pretty arbitrary and unintuitive to me, so by picking different rules one is no longer defying anything. In particular the requirements for exceptionally perfectly detailed knowledge of the child seem designed to make the conclusion tautologically true, but also not very interesting, as far as I can tell. "If I define impossible requirements, [thing] is impossible". OK, that's... fine, I guess. Allowing for "trust" between AGIs to overcome issues with their properly knowing one another's "truthfulness" but not allowing the same shortcut when it comes time to judge their knowledge of the child's "truthfulness" especially seems odd. I'm also not sure how all this perfect knowledge is justified before we allow that an AGI has created a superior AGI—it seems impossible for any system, biological or AGI, so I'm not sure what use the conclusion is. "Perfect play always results in cat, in tic-tac-toe, so you cannot win this game" well, that's nice but we're talking about poker...
  - xamuel 6y ago
    Thanks for the feedback. Regarding "impossible requirements", does it really go without saying that knowing an AGI's truthfulness is so impossible? I could write a computer program which does nothing but print "1+1=2", "1+2=3", "1+3=4", ... onto the screen, and know (with "exceptionally perfectly detailed knowledge") everything about this program and know that the things it prints on the screen are truthful. So it's not that it's inherently impossible to have such detailed knowledge about just any computer program. What makes AGI special? Why is it that we should take it for granted that it's impossible to know the truthfulness of an AGI's knowledge? I'm not saying that's not impossible (indeed, by tweaking the assumptions, the argument in my paper could be considered an argument for said impossibility), I'm just trying to understand why you think it's so blatantly obvious that knowing the truthfulness of the AGI's knowledge is impossible.
    - spanhandler 6y ago
      Right, I agree that we can prove things about algorithms, but it seems that if you define truthfulness such that an AGI can't prove that about itself then it follows it won't be able to prove it about anything more complex than itself—unless there's some property of "self-ness" that makes and AGI's self (or a copy of itself) especially hard to it to understand, which would be surprising. I think the confusion in this thread's over what people normally think of when they think of an AGI creating a superior AGI, and this particular definition of "superior" that requires a a particular kind of knowledge of superiority, which is something we don't really do ourselves very much, so far as how we "know" all the various things we "know". I don't think it makes much difference to people whether an AGI "knows" its creation is superior by some measure because it's got certain knowledge of its "truthfulness" (and it may not be clear why that, in particular, should matter, even), or whether it's just statistically determined with 99.999999% certainty that it is—which, again, is basically how we operate for the most part, except we're rarely that certain. Like, if one person makes an AGI that doesn't seem to be able to operate at a level higher than a rat, and another makes one that promptly becomes god-emperor of the galaxy, I don't think anyone's gonna get hung up over whether we know "truthfulness" of them both before saying which is the superior AGI. If an AGI does the same, I guess we can rest assured that it doesn't know, for a certain definition of "know", that it has—but it may "know" it in plenty of other senses that we use the term.

xamuel 6y ago

Sorry, I forgot to clarify that when I talk about the AGI creating a child, I mean the AGI creating a child in such a way as to know the child's truthfulness and source-code. Otherwise, of course, a monkey could "create" an AGI by blindly typing source-code on a typewrite and getting lucky. >Do you believe you can make a gai that is smarter than you? There are different ways of measuring intelligence. For the measure addressed in the paper, no, I can't single-handedly create an AGI smarter than myself (at least not in such a way as to know its truthfulness and its source-code).