30 ms·

I think AGI is usually equated with “superhuman” intelligence, though obviously not a requirement. If superhuman intelligence were easy, why aren’t we already

by yojo 5y ago

I think AGI is usually equated with “superhuman” intelligence, though obviously not a requirement. If superhuman intelligence were easy, why aren’t we already more intelligent? Being smart has already demonstrated considerable fitness benefits for the human race. It seems like we haven’t gotten much smarter in the last few thousand years though, at least from a raw horsepower point of view. My intuition is that past a certain level of complexity intelligent systems become inherently unstable. This is all just a handwave, but there’s some circumstantial evidence in the confluence of genius and mental disorder in our species. If that were true, the question then becomes: is biology an inferior substrate for general intelligence compared to silicon. Obviously a cpu can add two integers more efficiently than a meat brain, but it’s not self evident that this will hold true for more complex computation. Put another way, if you try to make an AGI agent “smarter” than a person, you might just end up with an irrational system that spits out nonsense. So it may be that for complexity reasons you can’t beat human-level GI. At which point the question becomes whether AGI is cheaper to produce and operate than feeding and sheltering a human. If the answer is “no”, then AGI may be both possible and irrelevant.

greggman3 5y ago
It's pretty hard to believe that being able transfer knowledge between digital brains (something that seems arguably inevitable) as well as having faster access to more knowledge in general (because you can make a bigger brain with more memory) it's just pretty obvious that AGI will beat humans. If all they had was human brains but could transfer thoughts faster they'd still beat us.
- yojo 5y ago
  Again, I will hand-wave about complexity. A larger memory and the ability to load/unload large corpora rapidly implies a very different kind of cognition than the one we enjoy. I’m not saying you’re wrong, only that I can imagine a reality where you can’t just arbitrarily scale components of cognition and still have a functioning system.
  - gonehome 5y ago
    It's more likely (imo) that humans are constrained by biological limits (energy, head size) than some true maximum. A simple thought experiment is the same architecture of the human brain (whatever that is), but just with way more power to run at more operations per second.
    - breuleux 5y ago
      Would that actually work, though, or would it just cook itself? I think the general question to ask is, for a given organism or machine, at what point does it stop being cost effective to make it smarter? Nature doesn't just keep human intelligence at the level it currently is, it's doing so for all other organisms. There seems to be tradeoffs. Which ones would apply to AI? And which ones are not currently relevant but will apply to future AI? For example, AI has no physical predators currently, but you could imagine a future where they do, or where nanobots or special bacteria attack silicon. In such a future, it is possible that very intelligent AI would suddenly become uncompetitive, relative to dumber and leaner ones. Viable superintelligent AI might then turn out to be just as difficult or circumstantial to "evolve" as humans were.
      - eru 5y ago
        > Would that actually work, though, or would it just cook itself? You could add better cooling than what the human hardware provides. Even just taking a normal human head and dunking it in cold water dissipates a lot more head. (If you try this at home, I suggest getting a snorkel.)
      - gonehome 5y ago
        Why would it cook itself if it was replicated in silicon? I think it's a lot less likely humanity just happens to be at some natural limit of intelligence rather than being bounded by other unrelated constraints. AGI tradeoffs would be different because it'd be unconstrained from biological natural selection. If you're interested in this stuff it's worth checking out the work that specifies the AGI goal alignment problem more specifically. The most commonly recommended book is Bostrom's Superintelligence, but it's fairly dry. Tegmark's Life 3.0 is on the other extreme of too pop-sci like. I think Yudkowsky's writing is the best for explaining the issue (specifically AGI goal alignment and what the specific problem is) in a way that's accessible, but not dumbed down.
- shkkmo 5y ago
  I suspect that intra network latency has a significant impact on intelligent systems. Interconnectivity also seems crucial but as you add nodes the number of connections needed to keep that closeness in the networks scales dramatically. These connections also need space so if you keep the same level of interconnection as you add more nodes, your density drops and so your latency goes up. There is thus a fundamental tradeoff between network size, network latency, and network interconnectivity that can't be avoided. While we may eventually beat the human brain on some or all of these measures, there is good reason to believe there are fundamental limits on the scalability of intelligence.
  - yojo 5y ago
    Thank you for this. I was hoping someone would contribute a better explanation than my own vague intuition.
  - FeepingCreature 5y ago
    There is simply no reason to believe that the human brain is anywhere near optimal on this tradeoff.
- breuleux 5y ago
  > It's pretty hard to believe that being able transfer knowledge between digital brains (something that seems arguably inevitable) I don't think it's necessarily inevitable. It is not a given that future hardware architectures for AI would be inspectable or copiable, because the extra wiring required to do that is space and energy overhead. It may also be the case that greater intelligence comes through better distributed representations and that it simply isn't possible to cheaply translate knowledge from an inferior representation to a superior one (you may need to relearn from scratch). The ability to transfer thoughts may therefore require a sort of lowest-common-denominator representation, in other words, a language. I imagine that language could be more efficient, though.
  - eru 5y ago
    You are right that it's not inevitable, but still pretty likely. Also keep in mind: Human nerves work (roughly) at the speed of sound. Computers work (roughly) at the speed of light. We already make computers that are much, much faster than humans at sequential processing. About the representation thing: if you start with AIs that are copies from each other, you can probably keep the 'superior' internal representation the same or nearly the same. So I would expect AI clones to be able to exchange knowledge much quicker than unrelated AIs.
AgentME 5y ago
Human ancestors evolved larger and larger brain sizes until they reached the limits of what we can calorically support and by what fits through the birth canal. It seems unlikely to me that these limits just happen to line up with general limits on intelligence. An AGI will be easier to scale further up without these biological limits. (And even if those biological limits coincidentally lined up with general limits on intelligence, just the fact that AGIs will be able to duplicate themselves or share knowledge directly would itself be a huge practical increase to their intelligence over us.)
- chrisco255 5y ago
  There's no proof that humans have hit any such limits. But that's not how genes and evolution work. We got intelligent enough to adapt to the environment and conditions that were thrown at us to survive the ice age and through the many inter-tribal wars that have been fought since. But once a species hits a plateau in selection pressure there is little improvement, except via sexual competition.
- eru 5y ago
  I'm not sure the birth canal is a real limit. It's a limit on how wide your head can be when you are born. That's not the same as a limit on brain size in adults. (Nor even directly a limit on brain size in newborns: up to a point they could always get longer heads instead of wider ones.) > (And even if those biological limits coincidentally lined up with general limits on intelligence, just the fact that AGIs will be able to duplicate themselves or share knowledge directly would itself be a huge practical increase to their intelligence over us.) Yes. A human with access to a calculator is much more capable than one without. An otherwise human-level AGI with hard-wired direct-'brain'-access would be even more capable---without having to manipulate the calculator with clumsy fingers and interpreting its outputs with a general visual system. Add direct-'brain'-access to Google and Wikipedia and to huge amounts of raw storage, and your human-level AGI would already be super human.
  - tacocataco 5y ago
    "(Nor even directly a limit on brain size in newborns: up to a point they could always get longer heads instead of wider ones.)" Coneheads confirmed? Surly I wasn't the only one to watch it.
PeterisP 5y ago
There are many aspects how biology can make "better" humans - faster, stronger, more resilient, etc, but they are not evolutionary useful because they cost extra calories and apparently are not worth the cost in a calorie restricted environment. Our bodies have been optimized to have as little of the "good stuff" as possible to conserve calories. We have mechanisms that will prevent building more muscle unless it's really necessary (as shown by exercise) and we have spare food - but it does not have to be this way, e.g. gorillas don't have to exercise to develop their huge muscles, it's just as our bodies (unlike theirs) is heavily optimized in favor of "cheaping out on features" to be more resistant to starvation. IMHO that also fully 'explains away' the issue of "why aren't we already more intelligent". Brains are excessive consumers of energy compared to other organs, so extra brain mass costs calories. At a certain point (which seems to be our current brains) being a bit smarter does not allow a hunter-gatherer to harvest much more calories per day in the seasons/crisis events where food sources are scarce (which are the only times that matter for evolution), so the brain increase stops there, just to save calories. Improvements that are 'zero cost' (i.e. brain structure changes which give more intelligence for the same calorie expenditure) are welcome, but the trivial way of getting more 'processing power' through larger brains is aggressively selected against. And all this calorie saving apparently was worthwhile - e.g. neanderthals were stronger and had larger brains than homo sapiens , but we were a bit more efficient, so we are here and they are not.
- eru 5y ago
  And we _are_ already more intelligent. Eg we are more intelligent than rats, even though rats are pretty smart and evolution had plenty of time to make them smarter.

by yojo 5y ago

greggman3 5y ago

It's pretty hard to believe that being able transfer knowledge between digital brains (something that seems arguably inevitable) as well as having faster access to more knowledge in general (because you can make a bigger brain with more memory) it's just pretty obvious that AGI will beat humans. If all they had was human brains but could transfer thoughts faster they'd still beat us.

yojo 5y ago
Again, I will hand-wave about complexity. A larger memory and the ability to load/unload large corpora rapidly implies a very different kind of cognition than the one we enjoy. I’m not saying you’re wrong, only that I can imagine a reality where you can’t just arbitrarily scale components of cognition and still have a functioning system.
- gonehome 5y ago
  It's more likely (imo) that humans are constrained by biological limits (energy, head size) than some true maximum. A simple thought experiment is the same architecture of the human brain (whatever that is), but just with way more power to run at more operations per second.
  - breuleux 5y ago
    Would that actually work, though, or would it just cook itself? I think the general question to ask is, for a given organism or machine, at what point does it stop being cost effective to make it smarter? Nature doesn't just keep human intelligence at the level it currently is, it's doing so for all other organisms. There seems to be tradeoffs. Which ones would apply to AI? And which ones are not currently relevant but will apply to future AI? For example, AI has no physical predators currently, but you could imagine a future where they do, or where nanobots or special bacteria attack silicon. In such a future, it is possible that very intelligent AI would suddenly become uncompetitive, relative to dumber and leaner ones. Viable superintelligent AI might then turn out to be just as difficult or circumstantial to "evolve" as humans were.
    - eru 5y ago
      > Would that actually work, though, or would it just cook itself? You could add better cooling than what the human hardware provides. Even just taking a normal human head and dunking it in cold water dissipates a lot more head. (If you try this at home, I suggest getting a snorkel.)
    - gonehome 5y ago
      Why would it cook itself if it was replicated in silicon? I think it's a lot less likely humanity just happens to be at some natural limit of intelligence rather than being bounded by other unrelated constraints. AGI tradeoffs would be different because it'd be unconstrained from biological natural selection. If you're interested in this stuff it's worth checking out the work that specifies the AGI goal alignment problem more specifically. The most commonly recommended book is Bostrom's Superintelligence, but it's fairly dry. Tegmark's Life 3.0 is on the other extreme of too pop-sci like. I think Yudkowsky's writing is the best for explaining the issue (specifically AGI goal alignment and what the specific problem is) in a way that's accessible, but not dumbed down.
shkkmo 5y ago
I suspect that intra network latency has a significant impact on intelligent systems. Interconnectivity also seems crucial but as you add nodes the number of connections needed to keep that closeness in the networks scales dramatically. These connections also need space so if you keep the same level of interconnection as you add more nodes, your density drops and so your latency goes up. There is thus a fundamental tradeoff between network size, network latency, and network interconnectivity that can't be avoided. While we may eventually beat the human brain on some or all of these measures, there is good reason to believe there are fundamental limits on the scalability of intelligence.
- yojo 5y ago
  Thank you for this. I was hoping someone would contribute a better explanation than my own vague intuition.
- FeepingCreature 5y ago
  There is simply no reason to believe that the human brain is anywhere near optimal on this tradeoff.
breuleux 5y ago
> It's pretty hard to believe that being able transfer knowledge between digital brains (something that seems arguably inevitable) I don't think it's necessarily inevitable. It is not a given that future hardware architectures for AI would be inspectable or copiable, because the extra wiring required to do that is space and energy overhead. It may also be the case that greater intelligence comes through better distributed representations and that it simply isn't possible to cheaply translate knowledge from an inferior representation to a superior one (you may need to relearn from scratch). The ability to transfer thoughts may therefore require a sort of lowest-common-denominator representation, in other words, a language. I imagine that language could be more efficient, though.
- eru 5y ago
  You are right that it's not inevitable, but still pretty likely. Also keep in mind: Human nerves work (roughly) at the speed of sound. Computers work (roughly) at the speed of light. We already make computers that are much, much faster than humans at sequential processing. About the representation thing: if you start with AIs that are copies from each other, you can probably keep the 'superior' internal representation the same or nearly the same. So I would expect AI clones to be able to exchange knowledge much quicker than unrelated AIs.

AgentME 5y ago

Human ancestors evolved larger and larger brain sizes until they reached the limits of what we can calorically support and by what fits through the birth canal. It seems unlikely to me that these limits just happen to line up with general limits on intelligence. An AGI will be easier to scale further up without these biological limits. (And even if those biological limits coincidentally lined up with general limits on intelligence, just the fact that AGIs will be able to duplicate themselves or share knowledge directly would itself be a huge practical increase to their intelligence over us.)

chrisco255 5y ago
There's no proof that humans have hit any such limits. But that's not how genes and evolution work. We got intelligent enough to adapt to the environment and conditions that were thrown at us to survive the ice age and through the many inter-tribal wars that have been fought since. But once a species hits a plateau in selection pressure there is little improvement, except via sexual competition.
eru 5y ago
I'm not sure the birth canal is a real limit. It's a limit on how wide your head can be when you are born. That's not the same as a limit on brain size in adults. (Nor even directly a limit on brain size in newborns: up to a point they could always get longer heads instead of wider ones.) > (And even if those biological limits coincidentally lined up with general limits on intelligence, just the fact that AGIs will be able to duplicate themselves or share knowledge directly would itself be a huge practical increase to their intelligence over us.) Yes. A human with access to a calculator is much more capable than one without. An otherwise human-level AGI with hard-wired direct-'brain'-access would be even more capable---without having to manipulate the calculator with clumsy fingers and interpreting its outputs with a general visual system. Add direct-'brain'-access to Google and Wikipedia and to huge amounts of raw storage, and your human-level AGI would already be super human.
- tacocataco 5y ago
  "(Nor even directly a limit on brain size in newborns: up to a point they could always get longer heads instead of wider ones.)" Coneheads confirmed? Surly I wasn't the only one to watch it.

PeterisP 5y ago

There are many aspects how biology can make "better" humans - faster, stronger, more resilient, etc, but they are not evolutionary useful because they cost extra calories and apparently are not worth the cost in a calorie restricted environment. Our bodies have been optimized to have as little of the "good stuff" as possible to conserve calories. We have mechanisms that will prevent building more muscle unless it's really necessary (as shown by exercise) and we have spare food - but it does not have to be this way, e.g. gorillas don't have to exercise to develop their huge muscles, it's just as our bodies (unlike theirs) is heavily optimized in favor of "cheaping out on features" to be more resistant to starvation. IMHO that also fully 'explains away' the issue of "why aren't we already more intelligent". Brains are excessive consumers of energy compared to other organs, so extra brain mass costs calories. At a certain point (which seems to be our current brains) being a bit smarter does not allow a hunter-gatherer to harvest much more calories per day in the seasons/crisis events where food sources are scarce (which are the only times that matter for evolution), so the brain increase stops there, just to save calories. Improvements that are 'zero cost' (i.e. brain structure changes which give more intelligence for the same calorie expenditure) are welcome, but the trivial way of getting more 'processing power' through larger brains is aggressively selected against. And all this calorie saving apparently was worthwhile - e.g. neanderthals were stronger and had larger brains than homo sapiens , but we were a bit more efficient, so we are here and they are not.

eru 5y ago
And we _are_ already more intelligent. Eg we are more intelligent than rats, even though rats are pretty smart and evolution had plenty of time to make them smarter.