3 ms·

As of 2021, has anything improved ? I'm not sure if this was before or after alphazero

sdenton4 5y ago
The article is about the failure of reinforcement learning to make it out of games and into any kind of real world task, so I think alpha go doesn't really change the argument.
- nuclearnice3 5y ago
  Maybe AlphaFold does? https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-... Notably it's still an entry in a common task framework contest, not a piece of software for the lab, as far as i know.
  - xiphias2 5y ago
    AlphaFold is from DeepMind, but it uses modern neural networks, but not reinforcement learning. DeepMind is not just simply wasting money, they are doing important AI research in other areas as well.
throwawaygh 5y ago
One thing that's changed is: > So now we have the top machine learning research institutes, DeepMind and OpenAI, still spending the majority of their time and resources on Deep RL DeepMind has diversified at least some since 2019, and I'm fairly confident that OpenAI is spending more resources on huge transformer models than on RL these days. Which is really the only thing that has changed, since even in 2019 there are at least a dozen world-class institutions doing AI/ML research aimed at addressing issues raised in this blog post (and others). The blog post is accurate about OpenAI/Deepmind c. 2019, but is wrong about the overall composition of research effort in the field c. 2019. Outside of two small and very new labs, most ML research wasn't focused on RL, and most RL research wasn't focused on DRL as a silver bullet. Sort of of the west coast SV version of only paying attention to work out of MIT and Stanford and therefore missing most of the interesting things happening in the world.
xiphias2 5y ago
Most of the improvements in real world have been by improving data representation for perception (improvements in transformers, self supervised learning), so so far the article seems right.
timkam 5y ago
I think when looking at least at large scale applications in the context of games (and not just super-expensive showcases), like in the Stockfish chess engine, we see that it's not primarily about depth, it's about architecture design. Reference: start here https://stockfishchess.org/blog/2021/stockfish-14/ https://stockfishchess.org/blog/2021/stockfish-14/ and go down the rabbit hole... In the broadly useful domain of recommender systems (which typically make use of some type of RL-like feedback loop, but can be implemented using simple clustering approaches), at least in 2019, neural network-based approaches didn't seem to fair too well, either: https://arxiv.org/pdf/1907.06902.pdf https://arxiv.org/pdf/1907.06902.pdf (arXiv pre-print, but this is an award-winning paper). Since then, it seems that researchers are moving away from getting deeper and deeper (the low-hanging fruit), and try to be more creative instead: new architectures, combining symbolic (logic-based) and sub-symbolic (ML-based) AI, etc.

The article is about the failure of reinforcement learning to make it out of games and into any kind of real world task, so I think alpha go doesn't really change the argument.

nuclearnice3 5y ago
Maybe AlphaFold does? https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-... Notably it's still an entry in a common task framework contest, not a piece of software for the lab, as far as i know.
- xiphias2 5y ago
  AlphaFold is from DeepMind, but it uses modern neural networks, but not reinforcement learning. DeepMind is not just simply wasting money, they are doing important AI research in other areas as well.

throwawaygh 5y ago

One thing that's changed is: > So now we have the top machine learning research institutes, DeepMind and OpenAI, still spending the majority of their time and resources on Deep RL DeepMind has diversified at least some since 2019, and I'm fairly confident that OpenAI is spending more resources on huge transformer models than on RL these days. Which is really the only thing that has changed, since even in 2019 there are at least a dozen world-class institutions doing AI/ML research aimed at addressing issues raised in this blog post (and others). The blog post is accurate about OpenAI/Deepmind c. 2019, but is wrong about the overall composition of research effort in the field c. 2019. Outside of two small and very new labs, most ML research wasn't focused on RL, and most RL research wasn't focused on DRL as a silver bullet. Sort of of the west coast SV version of only paying attention to work out of MIT and Stanford and therefore missing most of the interesting things happening in the world.

xiphias2 5y ago

Most of the improvements in real world have been by improving data representation for perception (improvements in transformers, self supervised learning), so so far the article seems right.

timkam 5y ago

I think when looking at least at large scale applications in the context of games (and not just super-expensive showcases), like in the Stockfish chess engine, we see that it's not primarily about depth, it's about architecture design. Reference: start here https://stockfishchess.org/blog/2021/stockfish-14/ https://stockfishchess.org/blog/2021/stockfish-14/ and go down the rabbit hole... In the broadly useful domain of recommender systems (which typically make use of some type of RL-like feedback loop, but can be implemented using simple clustering approaches), at least in 2019, neural network-based approaches didn't seem to fair too well, either: https://arxiv.org/pdf/1907.06902.pdf https://arxiv.org/pdf/1907.06902.pdf (arXiv pre-print, but this is an award-winning paper). Since then, it seems that researchers are moving away from getting deeper and deeper (the low-hanging fruit), and try to be more creative instead: new architectures, combining symbolic (logic-based) and sub-symbolic (ML-based) AI, etc.