7 ms·

Saying RL is sufficient to (eventually) achieve AGI is a bit misleading. One might similarly state that biological evolution is sufficient to (eventually) achie

Saying RL is sufficient to (eventually) achieve AGI is a bit misleading. One might similarly state that biological evolution is sufficient to (eventually) achieve biological general intelligence. Both statements are probably true, but the parenthetical (eventually) is doing an awful lot of heavy lifting.

neltnerb 5y ago
Yes, it's easy to be convinced on either side, the arguments write themselves. Yes, eventually a learning system might learn enough to be indistinguishable from intelligence. Or this might be entirely the wrong path and detracting from genuine new innovations in how we think about AI. We won't be able to tell whether it's AGI or just good enough at trained tasks to trick us.
- exporectomy 5y ago
  It can prove its intelligence by making testable predictions of the future better than us. As for whether it's "real" AGI or just acts like it, doesn't really matter. I think the Chinese room problem has been agreed on as not a problem, hasn't it?
  - neltnerb 5y ago
    My last sentence was a statement of that problem, not a question.
  - chriswarbo 5y ago
    > As for whether it's "real" AGI or just acts like it, doesn't really matter. Absolutely. The term "AGI" came about specifically to avoid existing philosophical arguments about "strong AI", "real AI", "synthetic intelligence", etc. Those wanting to discuss "true intelligence", etc. should use those other terms, or define new ones, rather than misuse the term AGI. AGI requires nothing more (or less!) than a widely-applicable optimisation algorithm. For example, it's easy to argue that a paperclip maximiser isn't "truly intelligent", but that won't stop it smelting your haemoglobin into more paperclips!
  - 6gvONxR4sf7o 5y ago
    I think proof of “real” intelligence by answering harder and harder questions is barking up the wrong tree. I think evidence and proof are a better way to denote varying levels of understanding. A deductive system can come with an answer and a proof of that answer, where proof is whatever counts as proof in that system. So the notion of “does it really understand it’s answers” gets punted off its Q&A abilities and onto its ability to justify its answers.
    - exporectomy 5y ago
      That's an interesting idea but it would exclude many humans who can make correct predictions using their experience and intuition but can't justify them correctly. Those people are still very useful. What you're describing is what we do at school. We can't assess understanding so we assess justification of answers as well as other things like ability to do X (we don't care if they understood or not, just be capable).
- prometheus76 5y ago
  Let's say I'm standing next to a table. The computer recognizes it as a table. Now I sit on the table. Is it a chair or a table now? Something that we do automatically is a LONG way away from being automatic for AI.
visarga 5y ago
RL + piggybacking on human culture might be enough, or evolution + RL for biological agents.
- webmaven 5y ago
  > RL + piggybacking on human culture might be enough, or evolution + RL for biological agents. Yes, but over what timeframe? Will there be any diminishing returns plateaus along the way?
  - visarga 5y ago
    We still have unknown unknowns but we also know a lot more about how neural nets deal with various tasks and dataset preparations. We know what kind of applications are good enough and where they still fail, which is much more than a decade ago. If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance. GPT-3 taught us that good sounding text is not that hard to generate if you have ample training data, but modeling the larger context is still hard. These kind of fine distinctions are what I call progress.
    - webmaven 5y ago
      > If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance. I waffle a lot on whether that aspect of 1968's '2001: A Space Odyssey' is evidence of genius or just survivorship bias.
ms1 5y ago
I think, in really broad terms, in order to get AGI actually we would need to do better than nature. If our metric is (intelligence)/(joule), nature seems pretty bad at a first glance: it took many trillions of lifetimes to achieve "general intelligence" * But then again, on the big stuff like this, have we ever really beat nature? That asterisk is there because, sure, turning the earth's biosphere into computers would make us smarter, but... are we sure? (And also: human = general?)
- londons_explore 5y ago
  Nature has a massive incentive to make good use of energy from light through photosynthesis. Billions of plants compete, and whoever can get most out of the sun will win out. Yet manmade solar cells are more efficient by nearly all measures.
  - Clewza313 5y ago
    Except that manmade solar cells are pretty bad at repairing or replicating themselves.
    - rich_sasha 5y ago
      Or growing out of literally nothing but dirt and water.
      - 21eleven 5y ago
        And air. That's what is crazy about plants, their carbon comes from the CO2 in the air. Also if someone loses weight, most of the carbon that made up their fat leaves the body as breath.
        khimaros 5y ago
        and are inedible
        khimaros 5y ago
deleted 5y ago
[deleted]
admk 5y ago
That is if you believe biological general intelligence is the end goal of evolution, which I believe is highly unlikely. Intelligence is simply a special side-product of evolution, there is nothing general about general intelligence. Many organisms can thrive without it. There is also a non-negligible chance that all organisms would die out before reaching intelligence. We are fortunate to live in a world that produced us.
- arketyp 5y ago
  That's a bit besides OP's point though, which is about vacuous claims. Humans are the existence proof that there is some sequence of circumstances where evolution reaches GI. There's an analogous sequence of circumstances in the RL case, which happens to be the hard part.
- Filligree 5y ago
  > That is if you believe biological general intelligence is the end goal of evolution, which I believe is highly unlikely. I would agree, but might add that evolution doesn't have 'goals'. Is that the point you were trying to make?
  - didericis 5y ago
    Not OP, but yeah, evolution doesn’t have goals in the same sense that people do, just like gravity doesn’t “want” to pull things, it just kind of “is”, and simply acts as reality permits based on prior and current conditions. That’s reasonable to say. Convergent evolution exists for at least some adaptations though, like the eye. It’s not unreasonable to think that there may be some sort of equivalent convergence which creates a high general intelligence adaptation given enough time, at least for social creatures. I think it’s pretty much impossible to know whether intelligence is a convergent adaptation without some kind of perfect simulation of evolution over billions of years. You’d have to tweak starting conditions and see if you kept getting smart creatures.
    - Filligree 5y ago
      Ah. So that’s why we exist. I was wondering.
burning_hamster 5y ago
I think the title of the paper makes more sense if you consider that ten years ago, someone could have written a paper in a similar spirit with a different take on "what is enough". Back then, it would probably have been titled: "Backpropagation of errors is enough". The last ten years have shown that backpropagation -- while a crucial component -- is not enough. Personally, I would not be shocked to find out in the next ten years that reinforcement learning is not enough for an AGI (as there are aspects like one-shot learning, forgetting, sleep, and other phenomena for which the RL framework seems not a natural fit).
- whatshisface 5y ago
  RL can forget, just start training it on a dataset that is different from what it was originally trained on.
  - burning_hamster 5y ago
    You are right; I should have been more specific. RL does forget in the simplest sense, i.e. that certain weights in your model drift away if the data distribution is non-stationary. Humans seem to be a bit more targeted in their forgetting.
- deleted 5y ago
  [deleted]
- endtime 5y ago
  Why are forgetting and sleep relevant? If someone invented a pill that gave you a perfect memory and removed the need to sleep, would you stop being generally intelligent if you took it?
  - skywhopper 5y ago
    Because they are strongly associated with all known examples of generalized intelligence. Why wouldn’t they be relevant?
criddell 5y ago
Does AGI imply human-level intelligence, or would the intelligence of a housefly qualify?
- 8ytecoder 5y ago
  I'm assuming you are aware of the difficulties for machines to do even the most basic of things that a living being can do with a brain the size of a pea. A housefly can fly and navigate effortless through most complex scenarios that it evolved to navigate (even though the same fly can get stuck behind a glass window and eventually die). So yeah, even getting that level of intelligence would be a huge win. However, most people mean close to human level intelligence when they mean AGI even if it's one narrow specialization.
  - Blikkentrekker 5y ago
    > even if it's one narrow specialization. Obviously that already exists even with g.o.f.a.i.s so that is not that impressive. The impressive thing is something more general than that.
  - criddell 5y ago
    Doesn't the G in AGI imply that narrow specializations aren't the target?
- flylikeabanana 5y ago
  AGI implies it can pass a Turing test, which means it has a better-than-average chance of acting more "human" than a competing human.
  - deleted 5y ago
    [deleted]
- mjburgess 5y ago
  It's a very interesting question. Personally I take mammalian intelligence as the relevant standard we're actually aiming at. So I'd say mouse+. Houseflys, I think, are closer to non-intelligent than intelligent.
xamuel 5y ago
I thought it was a fun position paper, if not exactly groundbreaking. They did avoid one common pitfall at least. They are (intentionally?) vague about which number systems the rewards can come from, apparently leaving it open whether the rewards need be real-valued or whether they can be, say, hyperreals, surreals, computable ordinals, etc. This avoids a trap I've written about elsewhere [1]: traditionally, RL rewards are limited to be real-valued (usually rational-valued). I argue that RL with real-valued rewards is NOT enough to reach AGI, because the real numbers have a constrained structure making them not flexible enough to express certain goals which an AGI should nevertheless have no problem comprehending (whether or not the AGI can actually solve them---that's a different question). In other words: if real-valued RL is enough for AGI, but real-valued RL is strictly less expressive than more general RL, then what is more general RL good enough for? "Artificial Better-Than-General Intelligence"? Note, however, that almost all [2] practical RL agent technology (certainly any based on neural nets or backprop) very fundamentally assumes real-valued rewards. So if it is true that "RL is enough" but also that "real-valued RL is not enough", then the bad news is all that progress on real-valued RL is not guaranteed to help us reach AGI. [1] "The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI", JAGI 2020, https://philpapers.org/archive/ALETAT-12.pdf https://philpapers.org/archive/ALETAT-12.pdf [2] A notable exception is preference-based RL
- robrenaud 5y ago
  There are more real numbers than programs. Computers cannot represent the vast majority of real numbers. AFAICT, it's not even clear that the universe is continuous rather than discrete. I really don't believe that using approximations of real numbers is going to be the bottleneck for AGI.
  - YinLuck- 5y ago
    I thought the whole point of quantum mechanics was to prove the universe was discrete and thus not continuous, no?
  - xamuel 5y ago
    And yet computers have no problem symbolically representing non-rational numbers like sqrt(2), pi, etc. Neither is there any inherent reason why they cannot symbolically represent various levels of infinity, nor why those would be incomprehensible to AGIs (even if the universe is discrete). You're right that only countably many numbers can be represented, but nevertheless even countable subsets of extended number systems can exhibit structural properties that the reals do not exhibit.

neltnerb 5y ago

Yes, it's easy to be convinced on either side, the arguments write themselves. Yes, eventually a learning system might learn enough to be indistinguishable from intelligence. Or this might be entirely the wrong path and detracting from genuine new innovations in how we think about AI. We won't be able to tell whether it's AGI or just good enough at trained tasks to trick us.

exporectomy 5y ago
It can prove its intelligence by making testable predictions of the future better than us. As for whether it's "real" AGI or just acts like it, doesn't really matter. I think the Chinese room problem has been agreed on as not a problem, hasn't it?
- neltnerb 5y ago
  My last sentence was a statement of that problem, not a question.
- chriswarbo 5y ago
  > As for whether it's "real" AGI or just acts like it, doesn't really matter. Absolutely. The term "AGI" came about specifically to avoid existing philosophical arguments about "strong AI", "real AI", "synthetic intelligence", etc. Those wanting to discuss "true intelligence", etc. should use those other terms, or define new ones, rather than misuse the term AGI. AGI requires nothing more (or less!) than a widely-applicable optimisation algorithm. For example, it's easy to argue that a paperclip maximiser isn't "truly intelligent", but that won't stop it smelting your haemoglobin into more paperclips!
- 6gvONxR4sf7o 5y ago
  I think proof of “real” intelligence by answering harder and harder questions is barking up the wrong tree. I think evidence and proof are a better way to denote varying levels of understanding. A deductive system can come with an answer and a proof of that answer, where proof is whatever counts as proof in that system. So the notion of “does it really understand it’s answers” gets punted off its Q&A abilities and onto its ability to justify its answers.
  - exporectomy 5y ago
    That's an interesting idea but it would exclude many humans who can make correct predictions using their experience and intuition but can't justify them correctly. Those people are still very useful. What you're describing is what we do at school. We can't assess understanding so we assess justification of answers as well as other things like ability to do X (we don't care if they understood or not, just be capable).
prometheus76 5y ago
Let's say I'm standing next to a table. The computer recognizes it as a table. Now I sit on the table. Is it a chair or a table now? Something that we do automatically is a LONG way away from being automatic for AI.

visarga 5y ago

RL + piggybacking on human culture might be enough, or evolution + RL for biological agents.

webmaven 5y ago
> RL + piggybacking on human culture might be enough, or evolution + RL for biological agents. Yes, but over what timeframe? Will there be any diminishing returns plateaus along the way?
- visarga 5y ago
  We still have unknown unknowns but we also know a lot more about how neural nets deal with various tasks and dataset preparations. We know what kind of applications are good enough and where they still fail, which is much more than a decade ago. If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance. GPT-3 taught us that good sounding text is not that hard to generate if you have ample training data, but modeling the larger context is still hard. These kind of fine distinctions are what I call progress.
  - webmaven 5y ago
    > If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance. I waffle a lot on whether that aspect of 1968's '2001: A Space Odyssey' is evidence of genius or just survivorship bias.

ms1 5y ago

I think, in really broad terms, in order to get AGI actually we would need to do better than nature. If our metric is (intelligence)/(joule), nature seems pretty bad at a first glance: it took many trillions of lifetimes to achieve "general intelligence" * But then again, on the big stuff like this, have we ever really beat nature? That asterisk is there because, sure, turning the earth's biosphere into computers would make us smarter, but... are we sure? (And also: human = general?)

londons_explore 5y ago
Nature has a massive incentive to make good use of energy from light through photosynthesis. Billions of plants compete, and whoever can get most out of the sun will win out. Yet manmade solar cells are more efficient by nearly all measures.
- Clewza313 5y ago
  Except that manmade solar cells are pretty bad at repairing or replicating themselves.
  - rich_sasha 5y ago
    Or growing out of literally nothing but dirt and water.
    - 21eleven 5y ago
      And air. That's what is crazy about plants, their carbon comes from the CO2 in the air. Also if someone loses weight, most of the carbon that made up their fat leaves the body as breath.
      - khimaros 5y ago
        and are inedible
      - khimaros 5y ago

deleted 5y ago

[deleted]

admk 5y ago

That is if you believe biological general intelligence is the end goal of evolution, which I believe is highly unlikely. Intelligence is simply a special side-product of evolution, there is nothing general about general intelligence. Many organisms can thrive without it. There is also a non-negligible chance that all organisms would die out before reaching intelligence. We are fortunate to live in a world that produced us.

arketyp 5y ago
That's a bit besides OP's point though, which is about vacuous claims. Humans are the existence proof that there is some sequence of circumstances where evolution reaches GI. There's an analogous sequence of circumstances in the RL case, which happens to be the hard part.
Filligree 5y ago
> That is if you believe biological general intelligence is the end goal of evolution, which I believe is highly unlikely. I would agree, but might add that evolution doesn't have 'goals'. Is that the point you were trying to make?
- didericis 5y ago
  Not OP, but yeah, evolution doesn’t have goals in the same sense that people do, just like gravity doesn’t “want” to pull things, it just kind of “is”, and simply acts as reality permits based on prior and current conditions. That’s reasonable to say. Convergent evolution exists for at least some adaptations though, like the eye. It’s not unreasonable to think that there may be some sort of equivalent convergence which creates a high general intelligence adaptation given enough time, at least for social creatures. I think it’s pretty much impossible to know whether intelligence is a convergent adaptation without some kind of perfect simulation of evolution over billions of years. You’d have to tweak starting conditions and see if you kept getting smart creatures.
  - Filligree 5y ago
    Ah. So that’s why we exist. I was wondering.

burning_hamster 5y ago

I think the title of the paper makes more sense if you consider that ten years ago, someone could have written a paper in a similar spirit with a different take on "what is enough". Back then, it would probably have been titled: "Backpropagation of errors is enough". The last ten years have shown that backpropagation -- while a crucial component -- is not enough. Personally, I would not be shocked to find out in the next ten years that reinforcement learning is not enough for an AGI (as there are aspects like one-shot learning, forgetting, sleep, and other phenomena for which the RL framework seems not a natural fit).

whatshisface 5y ago
RL can forget, just start training it on a dataset that is different from what it was originally trained on.
- burning_hamster 5y ago
  You are right; I should have been more specific. RL does forget in the simplest sense, i.e. that certain weights in your model drift away if the data distribution is non-stationary. Humans seem to be a bit more targeted in their forgetting.
deleted 5y ago
[deleted]
endtime 5y ago
Why are forgetting and sleep relevant? If someone invented a pill that gave you a perfect memory and removed the need to sleep, would you stop being generally intelligent if you took it?
- skywhopper 5y ago
  Because they are strongly associated with all known examples of generalized intelligence. Why wouldn’t they be relevant?

criddell 5y ago

Does AGI imply human-level intelligence, or would the intelligence of a housefly qualify?

8ytecoder 5y ago
I'm assuming you are aware of the difficulties for machines to do even the most basic of things that a living being can do with a brain the size of a pea. A housefly can fly and navigate effortless through most complex scenarios that it evolved to navigate (even though the same fly can get stuck behind a glass window and eventually die). So yeah, even getting that level of intelligence would be a huge win. However, most people mean close to human level intelligence when they mean AGI even if it's one narrow specialization.
- Blikkentrekker 5y ago
  > even if it's one narrow specialization. Obviously that already exists even with g.o.f.a.i.s so that is not that impressive. The impressive thing is something more general than that.
- criddell 5y ago
  Doesn't the G in AGI imply that narrow specializations aren't the target?
flylikeabanana 5y ago
AGI implies it can pass a Turing test, which means it has a better-than-average chance of acting more "human" than a competing human.
- deleted 5y ago
  [deleted]
mjburgess 5y ago
It's a very interesting question. Personally I take mammalian intelligence as the relevant standard we're actually aiming at. So I'd say mouse+. Houseflys, I think, are closer to non-intelligent than intelligent.

xamuel 5y ago

I thought it was a fun position paper, if not exactly groundbreaking. They did avoid one common pitfall at least. They are (intentionally?) vague about which number systems the rewards can come from, apparently leaving it open whether the rewards need be real-valued or whether they can be, say, hyperreals, surreals, computable ordinals, etc. This avoids a trap I've written about elsewhere [1]: traditionally, RL rewards are limited to be real-valued (usually rational-valued). I argue that RL with real-valued rewards is NOT enough to reach AGI, because the real numbers have a constrained structure making them not flexible enough to express certain goals which an AGI should nevertheless have no problem comprehending (whether or not the AGI can actually solve them---that's a different question). In other words: if real-valued RL is enough for AGI, but real-valued RL is strictly less expressive than more general RL, then what is more general RL good enough for? "Artificial Better-Than-General Intelligence"? Note, however, that almost all [2] practical RL agent technology (certainly any based on neural nets or backprop) very fundamentally assumes real-valued rewards. So if it is true that "RL is enough" but also that "real-valued RL is not enough", then the bad news is all that progress on real-valued RL is not guaranteed to help us reach AGI. [1] "The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI", JAGI 2020, https://philpapers.org/archive/ALETAT-12.pdf https://philpapers.org/archive/ALETAT-12.pdf [2] A notable exception is preference-based RL

robrenaud 5y ago
There are more real numbers than programs. Computers cannot represent the vast majority of real numbers. AFAICT, it's not even clear that the universe is continuous rather than discrete. I really don't believe that using approximations of real numbers is going to be the bottleneck for AGI.
- YinLuck- 5y ago
  I thought the whole point of quantum mechanics was to prove the universe was discrete and thus not continuous, no?
- xamuel 5y ago
  And yet computers have no problem symbolically representing non-rational numbers like sqrt(2), pi, etc. Neither is there any inherent reason why they cannot symbolically represent various levels of infinity, nor why those would be incomprehensible to AGIs (even if the universe is discrete). You're right that only countably many numbers can be represented, but nevertheless even countable subsets of extended number systems can exhibit structural properties that the reals do not exhibit.