7 ms·

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechani

I applaud the effort but the problem with RL as a model of learning is in the definition of RL itself. The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. Looking at it from CogSci perspective it is essentially an offshoot of behaviorism, using a coarse and extremely inefficient model of learning as reward and punishment, iterative trial and error process. This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently. Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuristics and hardcoded biases instead of doing some kind of brute force optimization. This is the case where computational efficiency [1] comes first and sets the constraints by which cognitive apparatus needs to be developed. As for actual cognition models a good place to start is not ML/AI field (which tends to getting stuck in local minima as a whole), but state-of-the-art cognitive psychology, and may be looking at research in "distributional semantics", "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience. You'd be surprised how much knowledge we gained about the mind since those RL & ANN models developed in the 1940s. [1] https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/0195126688 https://www.amazon.com/Circuits-Mind-Leslie-G-Valiant/dp/019...

anotherProd 6y ago
As someone who's a software engineer (not data scientist) but is interested in consciousness and by extension AGI and dabbled in some ML algorithms, I find it surprising how often I see the sentiments of AGI being possible or impossible using some sort of algorithm. Obviously I could be missing some great breadth and depth of research (there's definitely a lot I don't know) but from what I've read "we have no idea" is a pretty accurate description with how far we've come when it comes to consciousness, and I would imagine even less for the newer field of AI/AGI (consciousness has been around for a while P: and our theories have mostly sidestepped this real world phenomenon). > "The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop." This to me is a huge red flag (mostly of ego/hubris). I think if we rephrased the goal to not talk about "AGI" and maybe around quantitative things like the things you've listed ("computational efficiency", likelihood of being stuck in local minimimas, etc) then I'd happily concede that we should be looking at "X" and not "Y" but unless I've missed something, again likely, when we're talking about AGI, we're talking about consciousness (epiphenomenon that come about through physical/deterministic interactions). A quick way to gut check myself here is twisting what you state is not a good place to start "ML/AI field... gets stuck in local minima" and ask myself is it possible that local minima (which we consider "bad" for current/traditional tasks) could be necessary for consciousness ? I think the widely accepted answer to this is currently "We don't know". If I think that achieving AGI is going to be similar to what the algorithms and architecture we currently use (where the likelihood of being stuck in a local minima is something we can look at) then sure, your opinions stand. But that is just a guess and unless I'm mistaken AGI hasn't been achieved because we don't know how to do it. This isn't to say that we should have 100% of the data before making strong judgements like this about a subject. It's just that the subject of "consciousness" is a big one (I'd say THE big one) so making such strong statements about something we know we don't know much about is interesting. <- this is where I get flashbacks to SE world where a missing piece of data can really throw you off or leads to wrong assumptions and when I think about consciousness we know we don't know a lot.
- bra-ket 6y ago
  My point was that RL/DL is being used like some kind of massive hammer to hit all the nails. Cognition requires different, specialized, energy-efficient tools. > consciousness All talk about this is premature and "pre-science", before we figure out more basic, fundamental things like object storage and recall from memory, object recognition from sensory input, concept representation and formation, the exact mechanism of "chunking" [1], "translational invariance" [2], generalization along concept hierarchy and different scales, representation of causal structures, proximity search and heuristics, innate coordinate system, innate "grammar". Even having a working, biologically-plausible model of navigation in 3d spaces by mice, without spending a ton of energy training the model, would be a good first step. In fact there is evidence that navigational capacity [3] is the basis of more abstract forms of thinking. On all of these things we have decades worth of research and widely published, fundamental, Nobel-winning discoveries which are almost completely ignored by the AI field stuck in its comfort zone. Saying "we have no idea" is just being lazy. Edit: As for OP's actual paper I think something like complex-valued RL [4] might bypass his main claims entirely. But my point is that RL itself is a dead end, trivializing the problem at hand. [1] https://en.wikipedia.org/wiki/Chunking_(psychology) https://en.wikipedia.org/wiki/Chunking_(psychology) [2] http://www.moreisdifferent.com/2017/09/hinton-whats-wrong-with-CNNs http://www.moreisdifferent.com/2017/09/hinton-whats-wrong-wi... [3] http://www.scholarpedia.org/article/Grid_cells http://www.scholarpedia.org/article/Grid_cells [4] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22complex-valued+reinforcement+learning%22&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22c...
  - deleted 6y ago
    [deleted]
- xvilka 6y ago
  I think it will require at least a few more centuries to build AGI.
- thomasfromcdnjs 6y ago
  Stumbled upon this the other day, seems interesting ¯\_(ツ)_/¯ https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind "Penrose argues that human consciousness is non-algorithmic, and thus is not capable of being modeled by a conventional Turing machine, which includes a digital computer. Penrose hypothesizes that quantum mechanics plays an essential role in the understanding of human consciousness. The collapse of the quantum wavefunction is seen as playing an important role in brain function."
postnihilism 6y ago
> "This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently." > "Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuristics and hardcoded biases instead of doing some kind of brute force optimization." But those heuristics and hardcoded biases were developed through brute force optimization over the course of billions of years, a massive amount of energy input and many organisms being devoured.
- qyv 6y ago
  > But those heuristics and hardcoded biases were developed through brute force optimization over the course of billions of years, a massive amount of energy input and many organisms being devoured. This is true in the context of the universe as a whole, not by the organism itself.
  - renjimen 6y ago
    Except no organism is born a blank slate. Parent is correct in that our prior was massively expensive to construct
    - AlphaSite 6y ago
      I think the point huh is being mad is that individual people (or models) dot learn that way. It’s not like models training models, all the way down.
      - the8472 6y ago
        Individual people are not trained from scratch. ML models often have to be (modulo fine-tuning) since the field is still young.
        6y ago
gnulinux 6y ago
> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I click my pen and give him his favorite treats. Is this ineffective?
- benchaney 6y ago
  Your cat was already capable cognition before you started training it. GP is talking about generating a cognition where it did not previously exist.
  - deleted 6y ago
    [deleted]
  - joe_the_user 6y ago
    Why couldn't a mechanism (say A Neural Turing or whatever) that you train be "cognition capable" when you start and then be trained to actual behavior after that?
    - benchaney 6y ago
      You would need something that is "cognition capable" first and that has not been invented yet.
      - joe_the_user 6y ago
        It's hard to know. Maybe something "cognition cable" exists, it's just the proper train routine hasn't been provided to it. But regardless, the broader point is yeah, combine something akin cognition capability and the proper training routine and there you go, AGI from "reinforcement learning", broadly defined.
- kypro 6y ago
GregarianChild 6y ago
Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL. cognitive psychology You are overselling the insights of this discipline. Has cognitive psychology solved its replication problems? Where is the world-beating AI that is based on "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience? [1] https://arxiv.org/abs/1505.00521 https://arxiv.org/abs/1505.00521
- ForHackernews 6y ago
  > Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL. This seems like overstating your point. Nobody has been able to rigorously define "AI" yet, so there's no way of saying whether it's possible with a Turing machine architecture. The human brain, at least, doesn't seem that similar to a Turing architecture. Neurons don't carry out anything like discrete operations. Maybe it's possible to run AGI on a Turing machine, maybe it's not, but there are more options than simply "possible with a Turing machine" or "completely impossible".
  - GregarianChild 6y ago
    While I agree that we don't have an agreed upon definition of AI, the problem is firmly in the "I" part of AI! The "A" part is taken to mean implementable by a computer, i.e. a Turing machine. This is the content of the Church–Turing thesis [1]. [1] https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis
    - ForHackernews 6y ago
      > It states that a function on the natural numbers can be calculated by an effective method if and only if it is computable by a Turing machine. The article at the top of this thread is specifically about properties ("The Generalized Archimedean Property") that real numbers do not possess. There's also a little bit of slipperiness around the use of "AI" vs. "AGI" - you could easily argue (and people do!) that we've already achieved "AI" for many specialized domains. It's the General bit that seems to be the sticking point, and that this article focuses on.
odnes 6y ago
> The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. A lot of research in RL is focused on intrinsic motivation and the question of whether we can bootstrap our own 'rewards' from our ability to predict and control the future according to some self-defined goals/hypotheses.
cmrx64 6y ago
do you know about the dopamine reward error hypothesis? https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6721851/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6721851/ is it so wrong? what does cognitive psychology have to say about how these neurons work? this is a lot more recent than the 40s and behaviorism.
- bra-ket 6y ago
  dopamine rewards operate on a different time scale vs. that required by these error correction models. I don't remember the exact paper, will need to look it up, but it was orders of magnitude difference in response times. Edit: for authoritative reference on biologically-plausible learning see anything by Edmund Rolls [1]. He explicitly stated in his recent book [2] that something like back-propagation, or similar error correction mechanisms have no supporting evidence in experimental data collected so far [1] https://www.oxcns.org/profile.html https://www.oxcns.org/profile.html [2] https://www.amazon.com/Cerebral-Cortex-Principles-Edmund-Rolls/dp/0198820348/ https://www.amazon.com/Cerebral-Cortex-Principles-Edmund-Rol...
  - cmrx64 6y ago
    thanks for the link, it's been a lovely rabbit hole :)
deleted 6y ago
[deleted]
eli_gottlieb 6y ago
>state-of-the-art cognitive psychology, and may be looking at research in "distributional semantics", "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience. Look, uh, I've read Gardenfors too, but are those really the state of the art? I don't remember there being anything about them at CogSci this past summer. Maybe I wasn't paying close-enough attention?
- bra-ket 6y ago
  In RL/DL context any CogSci developments after 1943 is the state of the art. Some interesting recent work [1] related to Gardenfors ideas was combining them with discovery of place & grid cells, and extending the "cognitive maps" and spatial navigation machinery into concept spaces, treating the innate coordinate system as foundation for abstraction and generalization facilities. And they actually found empirical data to prove it in [1] and related papers, so Gardenfors was right. I believe it gotta be the starting point for anyone seriously considering an AI, kind of like Cartesian foundation. It also aligns nicely with rich "distributional semantics" work and popular vector space models. [1] https://pubmed.ncbi.nlm.nih.gov/27313047/ https://pubmed.ncbi.nlm.nih.gov/27313047/

anotherProd 6y ago

As someone who's a software engineer (not data scientist) but is interested in consciousness and by extension AGI and dabbled in some ML algorithms, I find it surprising how often I see the sentiments of AGI being possible or impossible using some sort of algorithm. Obviously I could be missing some great breadth and depth of research (there's definitely a lot I don't know) but from what I've read "we have no idea" is a pretty accurate description with how far we've come when it comes to consciousness, and I would imagine even less for the newer field of AI/AGI (consciousness has been around for a while P: and our theories have mostly sidestepped this real world phenomenon). > "The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop." This to me is a huge red flag (mostly of ego/hubris). I think if we rephrased the goal to not talk about "AGI" and maybe around quantitative things like the things you've listed ("computational efficiency", likelihood of being stuck in local minimimas, etc) then I'd happily concede that we should be looking at "X" and not "Y" but unless I've missed something, again likely, when we're talking about AGI, we're talking about consciousness (epiphenomenon that come about through physical/deterministic interactions). A quick way to gut check myself here is twisting what you state is not a good place to start "ML/AI field... gets stuck in local minima" and ask myself is it possible that local minima (which we consider "bad" for current/traditional tasks) could be necessary for consciousness ? I think the widely accepted answer to this is currently "We don't know". If I think that achieving AGI is going to be similar to what the algorithms and architecture we currently use (where the likelihood of being stuck in a local minima is something we can look at) then sure, your opinions stand. But that is just a guess and unless I'm mistaken AGI hasn't been achieved because we don't know how to do it. This isn't to say that we should have 100% of the data before making strong judgements like this about a subject. It's just that the subject of "consciousness" is a big one (I'd say THE big one) so making such strong statements about something we know we don't know much about is interesting. <- this is where I get flashbacks to SE world where a missing piece of data can really throw you off or leads to wrong assumptions and when I think about consciousness we know we don't know a lot.

bra-ket 6y ago
My point was that RL/DL is being used like some kind of massive hammer to hit all the nails. Cognition requires different, specialized, energy-efficient tools. > consciousness All talk about this is premature and "pre-science", before we figure out more basic, fundamental things like object storage and recall from memory, object recognition from sensory input, concept representation and formation, the exact mechanism of "chunking" [1], "translational invariance" [2], generalization along concept hierarchy and different scales, representation of causal structures, proximity search and heuristics, innate coordinate system, innate "grammar". Even having a working, biologically-plausible model of navigation in 3d spaces by mice, without spending a ton of energy training the model, would be a good first step. In fact there is evidence that navigational capacity [3] is the basis of more abstract forms of thinking. On all of these things we have decades worth of research and widely published, fundamental, Nobel-winning discoveries which are almost completely ignored by the AI field stuck in its comfort zone. Saying "we have no idea" is just being lazy. Edit: As for OP's actual paper I think something like complex-valued RL [4] might bypass his main claims entirely. But my point is that RL itself is a dead end, trivializing the problem at hand. [1] https://en.wikipedia.org/wiki/Chunking_(psychology) https://en.wikipedia.org/wiki/Chunking_(psychology) [2] http://www.moreisdifferent.com/2017/09/hinton-whats-wrong-with-CNNs http://www.moreisdifferent.com/2017/09/hinton-whats-wrong-wi... [3] http://www.scholarpedia.org/article/Grid_cells http://www.scholarpedia.org/article/Grid_cells [4] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22complex-valued+reinforcement+learning%22&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22c...
- deleted 6y ago
  [deleted]
xvilka 6y ago
I think it will require at least a few more centuries to build AGI.
thomasfromcdnjs 6y ago
Stumbled upon this the other day, seems interesting ¯\_(ツ)_/¯ https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind https://en.wikipedia.org/wiki/The_Emperor%27s_New_Mind "Penrose argues that human consciousness is non-algorithmic, and thus is not capable of being modeled by a conventional Turing machine, which includes a digital computer. Penrose hypothesizes that quantum mechanics plays an essential role in the understanding of human consciousness. The collapse of the quantum wavefunction is seen as playing an important role in brain function."

postnihilism 6y ago

> "This 'Skinnerism' has been discredited in cognitive psychology decades ago and makes absolutely no biological sense whatsoever for the simple reason that any organism trying to adapt in this way will be eaten by predators before minimizing its "error function" sufficiently." > "Living learning organisms have limited resources (energy and time), and they cut the search space drastically through shortcuts and heuristics and hardcoded biases instead of doing some kind of brute force optimization." But those heuristics and hardcoded biases were developed through brute force optimization over the course of billions of years, a massive amount of energy input and many organisms being devoured.

qyv 6y ago
> But those heuristics and hardcoded biases were developed through brute force optimization over the course of billions of years, a massive amount of energy input and many organisms being devoured. This is true in the context of the universe as a whole, not by the organism itself.
- renjimen 6y ago
  Except no organism is born a blank slate. Parent is correct in that our prior was massively expensive to construct
  - AlphaSite 6y ago
    I think the point huh is being mad is that individual people (or models) dot learn that way. It’s not like models training models, all the way down.
    - the8472 6y ago
      Individual people are not trained from scratch. ML models often have to be (modulo fine-tuning) since the field is still young.
      - 6y ago

gnulinux 6y ago

> The idea of using "rewards" as a learning mechanism and a path to actual cognition is just wrong, full stop. I'm a layman (just a software engineer) but am curious, I train my cat only with rewards (never punishment because apparently doesn't work on cats) and the kitty learned how to high-five me, sit, jump, follow me etc. It seems to work really well for us. Basically, ever time he does something desirable, I click my pen and give him his favorite treats. Is this ineffective?

benchaney 6y ago
Your cat was already capable cognition before you started training it. GP is talking about generating a cognition where it did not previously exist.
- deleted 6y ago
  [deleted]
- joe_the_user 6y ago
  Why couldn't a mechanism (say A Neural Turing or whatever) that you train be "cognition capable" when you start and then be trained to actual behavior after that?
  - benchaney 6y ago
    You would need something that is "cognition capable" first and that has not been invented yet.
    - joe_the_user 6y ago
      It's hard to know. Maybe something "cognition cable" exists, it's just the proper train routine hasn't been provided to it. But regardless, the broader point is yeah, combine something akin cognition capability and the proper training routine and there you go, AGI from "reinforcement learning", broadly defined.
kypro 6y ago

GregarianChild 6y ago

Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL. cognitive psychology You are overselling the insights of this discipline. Has cognitive psychology solved its replication problems? Where is the world-beating AI that is based on "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience? [1] https://arxiv.org/abs/1505.00521 https://arxiv.org/abs/1505.00521

ForHackernews 6y ago
> Reinforcement learning is Turing complete [1], so if AI is possible at all, then it can be realised through RL. This seems like overstating your point. Nobody has been able to rigorously define "AI" yet, so there's no way of saying whether it's possible with a Turing machine architecture. The human brain, at least, doesn't seem that similar to a Turing architecture. Neurons don't carry out anything like discrete operations. Maybe it's possible to run AGI on a Turing machine, maybe it's not, but there are more options than simply "possible with a Turing machine" or "completely impossible".
- GregarianChild 6y ago
  While I agree that we don't have an agreed upon definition of AI, the problem is firmly in the "I" part of AI! The "A" part is taken to mean implementable by a computer, i.e. a Turing machine. This is the content of the Church–Turing thesis [1]. [1] https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis
  - ForHackernews 6y ago
    > It states that a function on the natural numbers can be calculated by an effective method if and only if it is computable by a Turing machine. The article at the top of this thread is specifically about properties ("The Generalized Archimedean Property") that real numbers do not possess. There's also a little bit of slipperiness around the use of "AI" vs. "AGI" - you could easily argue (and people do!) that we've already achieved "AI" for many specialized domains. It's the General bit that seems to be the sticking point, and that this article focuses on.

odnes 6y ago

> The idea of using "rewards" as a primary learning mechanism and a path to actual cognition is just wrong, full stop. It's a wrong level of abstraction and is too wasteful in energy spent. A lot of research in RL is focused on intrinsic motivation and the question of whether we can bootstrap our own 'rewards' from our ability to predict and control the future according to some self-defined goals/hypotheses.

cmrx64 6y ago

do you know about the dopamine reward error hypothesis? https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6721851/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6721851/ is it so wrong? what does cognitive psychology have to say about how these neurons work? this is a lot more recent than the 40s and behaviorism.

bra-ket 6y ago
dopamine rewards operate on a different time scale vs. that required by these error correction models. I don't remember the exact paper, will need to look it up, but it was orders of magnitude difference in response times. Edit: for authoritative reference on biologically-plausible learning see anything by Edmund Rolls [1]. He explicitly stated in his recent book [2] that something like back-propagation, or similar error correction mechanisms have no supporting evidence in experimental data collected so far [1] https://www.oxcns.org/profile.html https://www.oxcns.org/profile.html [2] https://www.amazon.com/Cerebral-Cortex-Principles-Edmund-Rolls/dp/0198820348/ https://www.amazon.com/Cerebral-Cortex-Principles-Edmund-Rol...
- cmrx64 6y ago
  thanks for the link, it's been a lovely rabbit hole :)

deleted 6y ago

[deleted]

eli_gottlieb 6y ago

>state-of-the-art cognitive psychology, and may be looking at research in "distributional semantics", "concept spaces", "sparse representations", "small-world networks" and "learning and memory" neuroscience. Look, uh, I've read Gardenfors too, but are those really the state of the art? I don't remember there being anything about them at CogSci this past summer. Maybe I wasn't paying close-enough attention?

bra-ket 6y ago
In RL/DL context any CogSci developments after 1943 is the state of the art. Some interesting recent work [1] related to Gardenfors ideas was combining them with discovery of place & grid cells, and extending the "cognitive maps" and spatial navigation machinery into concept spaces, treating the innate coordinate system as foundation for abstraction and generalization facilities. And they actually found empirical data to prove it in [1] and related papers, so Gardenfors was right. I believe it gotta be the starting point for anyone seriously considering an AI, kind of like Cartesian foundation. It also aligns nicely with rich "distributional semantics" work and popular vector space models. [1] https://pubmed.ncbi.nlm.nih.gov/27313047/ https://pubmed.ncbi.nlm.nih.gov/27313047/