15 ms·

This whole evolution looks more and more like expert systems from 1980s where people kept adding more and more complexity to "solve" a specific problem. For RL

This whole evolution looks more and more like expert systems from 1980s where people kept adding more and more complexity to "solve" a specific problem. For RL, we started with simple DQN that was elegant but now the new algorithms looks like a massive hodge podge of band aids. NGU, as it is, extraordinarily complex and looks adhoc mix of various patches. Now on the top of NGU, we are also throwing in meta-controller and even bandits among other things to complete the proverbial kitchen sink. Sure, we get to call victory on Atari but this is far and away from elegant and beautiful. It would be surprising if this victory generalizes to other problems where folks have built different "expert systems" specific to those problems. So all this feels a lot like Watson winning jeopardy moment to me... PS: Kudos to DeepMind for pushing for median or even betten bottom percentile instead of simplistic average metric which also hides variance.

chpatrick 7y ago
Doesn't the human brain have millions of years of band aids?
- jacobush 7y ago
  Yes and at this pace it may take thousands (or at least hundreds) of years to patch up our A.I. to general A.I.
  - deleted 7y ago
    [deleted]
- eli_gottlieb 7y ago
  No, most likely not. Our best theories of the brain are both computationally and evolutionarily parsimonious.
  - gnramires 7y ago
    You can make a simple theory of anything. An airplane as a spherical ball of varying friction with a steerable propulsion. Or a massive electro-mechanical system. There's a fidelity curve as you progress in complexity. Fidelity curve of a literal ball stabilizes quickly, while the fidelity curve of an airplane extends far. The brain is massively complex, we still don't understand the ways that it computes and learns in detail except generalities such as using electrical and chemical signals, neurotransmissors, and some knowledge of architecture and region differentiation. Those models all point to an extremely complex system with quite a number of different specialized parts, with neurons individually having quite amazing complexity (and so on for capsules, neuronal systems). I think it's unlikely a "simple" system (if you could consider a vanilla CNN simple, for example) will solve human-like AGI efficiently (either data-efficiently or computationally efficiently). There are too many requirements. Also, while simplicity is good... a researcher shouldn't be caught obsessing over it exclusively, as it is not the end goal, which personally would be the advancement of understanding and of our capabilities instead.
    - eli_gottlieb 7y ago
      >The brain is massively complex, we still don't understand the ways that it computes and learns in detail except generalities such as using electrical and chemical signals, neurotransmissors, and some knowledge of architecture and region differentiation. Again, not really. Broadly speaking, the AI community is pretty far behind the neuroscience community in terms of understanding the basic principles underlying how brains work. https://mitpress.mit.edu/books/principles-neural-design https://mitpress.mit.edu/books/principles-neural-design
    - Retric 7y ago
      The human brian is encoded via surprisingly limited amounts of information in DNA. It’s therefore more elegant than many modern AI systems. The often quoted 6×10^9 base pairs overstates the complexity as a lot of non coding DNA exists, and much of what’s left is redundant or does not relate to the brain. PS: We actually know quite about about how the brain learns and computes information. The are really interesting information about how the optic nerve encodes information for example.
      - visarga 7y ago
        > The human brian is encoded via surprisingly limited amounts of information in DNA. Same DNA that is referred to in other discussions as 'the ultimate spaghetti code'.
      - fg6hr 7y ago
        DNA by itself doesn't produce a human or a cell. You need another functional cell or human to create a replica. I would compare DNA with instructions to create a new car: the instructions are incomplete and assume the existence of a car factory. DNA tells "now paint the car with of these colors", but it doesn't tell how.
        Retric 7y ago
        Computer programs don’t include the CPU design. DNA on the other hand encodes everything which then gets leveraged during cell division. Basically, DNA includes the design for both the car, car factory, and all the component factories that feed the car factory, and the mines that feed the component factories. People have dependencies on their diet for things like vitamin C, but that’s a whole other issue. PS: Mitochondria also have DNA, but that’s just location specific and can simply be added to the total for the cell.
adrianN 7y ago
You're just witnessing the natural progression of research. Occasionally somebody has a brilliant idea that works a lot better than the state of the art, and then people add bells and whistles to that idea until the improvements don't justify a new paper. Sometimes a new brilliant idea emerges from the bells and whistles, sometimes the field gets stuck for a while, sometimes the field dies.
WnZ39p0Dgydaz1 7y ago
I agree with you overall, but while it was relatively simple, DQN was never "elegant". The elegant and most important part of DQN was to use a CNN for learning a good image representation - but CNNs are completely unrelated to RL. The sole contribution of DQN was to introduce two ugly band-aids, experience replay and a target network, to make the CNN training stable in an RL setting. So, DQN = CNN (elegant and simple) + ugly band aid
- pabloem 7y ago
  Just a side note. Probably not very fair to attribue the introduction of experience replay to DQN authors. This technique was popularized in 1992 by Lin[1] and used later in many works to improve performance. The difference, perhaps, is that in DQN, experience replay is not only a way to improve performance, but it's a must to obtain stability. [1] http://www.incompleteideas.net/lin-92.pdf http://www.incompleteideas.net/lin-92.pdf
eli_gottlieb 7y ago
Contrast with papers like "Learning to Fly via Deep Model-Based Reinforcement Learning" (https://arxiv.org/abs/2003.08876 https://arxiv.org/abs/2003.08876), in which a principled approach was refined over years until it could be used for a real-world task.
david_s_krueger 7y ago
I find the comparison a bit facile. Expert systems were taylored to the problem at hand. The components of the solution here are based on learning and are meant to address what are widely believed to be important and general facets of human cognition and/or fundamental machine learning problems (e.g. credit assignment, episodic and short-term memory, meta-cognition). It's unclear how general the solutions here actually are, but they certainly don't look very specific to ATARI, on the face of it. It's also worth noting that DeepMind's research has a pretty good track record of not being overly engineered to solve specific tasks. E.g. DQN, AlphaGo and successors (with the possible exception of Alpha*?)...
antpls 7y ago
To everyone : stop to criticize other people's work about "elegance", both on internet and in real life. This is not NYC nor Paris Fashion Week. Criticisms about the estheticism of an algorithm do not make sense. We have objective metrics to judge codes and algorithms, such as Big O notation, resource usages, epoch, reproductability, etc. The questions the R&D team try to answer are a) "Is this possible?", and b) "Is it efficient?". This has nothing to do with estheticism. Simpleness is not a metric. Neural networks are not simple to understand nor use. The Deep Mind team shows that they are able to use them as building blocks in different places of a bigger algorithm to solve a goal, iteratively. All of these results required time, effort and ressources to obtain. They are not obligated to share their results with the public, so now just stop to criticize without even suggesting a better solution.

chpatrick 7y ago

Doesn't the human brain have millions of years of band aids?

jacobush 7y ago
Yes and at this pace it may take thousands (or at least hundreds) of years to patch up our A.I. to general A.I.
- deleted 7y ago
  [deleted]
eli_gottlieb 7y ago
No, most likely not. Our best theories of the brain are both computationally and evolutionarily parsimonious.
- gnramires 7y ago
  You can make a simple theory of anything. An airplane as a spherical ball of varying friction with a steerable propulsion. Or a massive electro-mechanical system. There's a fidelity curve as you progress in complexity. Fidelity curve of a literal ball stabilizes quickly, while the fidelity curve of an airplane extends far. The brain is massively complex, we still don't understand the ways that it computes and learns in detail except generalities such as using electrical and chemical signals, neurotransmissors, and some knowledge of architecture and region differentiation. Those models all point to an extremely complex system with quite a number of different specialized parts, with neurons individually having quite amazing complexity (and so on for capsules, neuronal systems). I think it's unlikely a "simple" system (if you could consider a vanilla CNN simple, for example) will solve human-like AGI efficiently (either data-efficiently or computationally efficiently). There are too many requirements. Also, while simplicity is good... a researcher shouldn't be caught obsessing over it exclusively, as it is not the end goal, which personally would be the advancement of understanding and of our capabilities instead.
  - eli_gottlieb 7y ago
    >The brain is massively complex, we still don't understand the ways that it computes and learns in detail except generalities such as using electrical and chemical signals, neurotransmissors, and some knowledge of architecture and region differentiation. Again, not really. Broadly speaking, the AI community is pretty far behind the neuroscience community in terms of understanding the basic principles underlying how brains work. https://mitpress.mit.edu/books/principles-neural-design https://mitpress.mit.edu/books/principles-neural-design
  - Retric 7y ago
    The human brian is encoded via surprisingly limited amounts of information in DNA. It’s therefore more elegant than many modern AI systems. The often quoted 6×10^9 base pairs overstates the complexity as a lot of non coding DNA exists, and much of what’s left is redundant or does not relate to the brain. PS: We actually know quite about about how the brain learns and computes information. The are really interesting information about how the optic nerve encodes information for example.
    - visarga 7y ago
      > The human brian is encoded via surprisingly limited amounts of information in DNA. Same DNA that is referred to in other discussions as 'the ultimate spaghetti code'.
    - fg6hr 7y ago
      DNA by itself doesn't produce a human or a cell. You need another functional cell or human to create a replica. I would compare DNA with instructions to create a new car: the instructions are incomplete and assume the existence of a car factory. DNA tells "now paint the car with of these colors", but it doesn't tell how.
      - Retric 7y ago
        Computer programs don’t include the CPU design. DNA on the other hand encodes everything which then gets leveraged during cell division. Basically, DNA includes the design for both the car, car factory, and all the component factories that feed the car factory, and the mines that feed the component factories. People have dependencies on their diet for things like vitamin C, but that’s a whole other issue. PS: Mitochondria also have DNA, but that’s just location specific and can simply be added to the total for the cell.

adrianN 7y ago

You're just witnessing the natural progression of research. Occasionally somebody has a brilliant idea that works a lot better than the state of the art, and then people add bells and whistles to that idea until the improvements don't justify a new paper. Sometimes a new brilliant idea emerges from the bells and whistles, sometimes the field gets stuck for a while, sometimes the field dies.

WnZ39p0Dgydaz1 7y ago

I agree with you overall, but while it was relatively simple, DQN was never "elegant". The elegant and most important part of DQN was to use a CNN for learning a good image representation - but CNNs are completely unrelated to RL. The sole contribution of DQN was to introduce two ugly band-aids, experience replay and a target network, to make the CNN training stable in an RL setting. So, DQN = CNN (elegant and simple) + ugly band aid

pabloem 7y ago
Just a side note. Probably not very fair to attribue the introduction of experience replay to DQN authors. This technique was popularized in 1992 by Lin[1] and used later in many works to improve performance. The difference, perhaps, is that in DQN, experience replay is not only a way to improve performance, but it's a must to obtain stability. [1] http://www.incompleteideas.net/lin-92.pdf http://www.incompleteideas.net/lin-92.pdf

eli_gottlieb 7y ago

Contrast with papers like "Learning to Fly via Deep Model-Based Reinforcement Learning" (https://arxiv.org/abs/2003.08876 https://arxiv.org/abs/2003.08876), in which a principled approach was refined over years until it could be used for a real-world task.

david_s_krueger 7y ago

I find the comparison a bit facile. Expert systems were taylored to the problem at hand. The components of the solution here are based on learning and are meant to address what are widely believed to be important and general facets of human cognition and/or fundamental machine learning problems (e.g. credit assignment, episodic and short-term memory, meta-cognition). It's unclear how general the solutions here actually are, but they certainly don't look very specific to ATARI, on the face of it. It's also worth noting that DeepMind's research has a pretty good track record of not being overly engineered to solve specific tasks. E.g. DQN, AlphaGo and successors (with the possible exception of Alpha*?)...

antpls 7y ago

To everyone : stop to criticize other people's work about "elegance", both on internet and in real life. This is not NYC nor Paris Fashion Week. Criticisms about the estheticism of an algorithm do not make sense. We have objective metrics to judge codes and algorithms, such as Big O notation, resource usages, epoch, reproductability, etc. The questions the R&D team try to answer are a) "Is this possible?", and b) "Is it efficient?". This has nothing to do with estheticism. Simpleness is not a metric. Neural networks are not simple to understand nor use. The Deep Mind team shows that they are able to use them as building blocks in different places of a bigger algorithm to solve a goal, iteratively. All of these results required time, effort and ressources to obtain. They are not obligated to share their results with the public, so now just stop to criticize without even suggesting a better solution.