3 ms·

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lea

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer. People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but that doesn't mean it's the best. RL still faces huge struggles with compute efficiency and it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count.

Isinlor 4y ago
Have you heard about EfficientZero? This is the first algorithm that achieved super-human performance on Atari 100k actions benchmark. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement. So data efficiency was doubling roughly every year for the past 8 years. Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge. https://arxiv.org/abs/2111.00210 https://arxiv.org/abs/2111.00210 Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.
stefan_ 4y ago
They have? That's the approach they are using? Because that doesn't mesh well with practical reality. Where AGI use Deep RL, it's to improve on vision tasks like object classification, none of them are making any driving decisions - that seems to remain the domain of I guess you could call it discrete logic.
- robotresearcher 4y ago
  Here's a survey paper from this year on Deep RL for autonomous driving. https://ieeexplore.ieee.org/document/9351818 https://ieeexplore.ieee.org/document/9351818 B. R. Kiran et al., "Deep Reinforcement Learning for Autonomous Driving: A Survey," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909-4926, June 2022, doi: 10.1109/TITS.2021.3054625. I haven't read the paper, so this is not a reading recommendation. Just posting as evidence that there is work in the area.
gwern 4y ago
> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervised Transformers do. Sorry if you were counting on 'RL doesn't work' for safety or anything. If you don't believe the basic existence proofs of things like OA5 or AlphaStar, which work only because things like larger batch sizes or more diverse agent populations magically make notoriously-unreliable archs work, you can look at Jones's beautiful AlphaZero scaling laws (plural) work https://arxiv.org/abs/2104.03113 https://arxiv.org/abs/2104.03113 , or browse through relevant papers https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restrict_sr=on&include_over_18=on https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restr... https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper Or GPT-f. Then you have stuff like Gato continuing to show scaling even in the Decision Transformer framework. Or consider instances of plugging pretrained models into RL agents, like SayCan-PaLM most recently.
- pixelpoet 4y ago
  While we have the mighty gwern on the line: do you believe we'll have AGI in <= 10 years?
  - telotortium 4y ago
    Obviously yes: https://twitter.com/robbensinger/status/1541228487205236736 https://twitter.com/robbensinger/status/1541228487205236736
- trention 4y ago
  That scaling will eventually hit a wall. What was it about nerds and S-curves?
  - dinosaurdynasty 4y ago
    Sure, most everything does. The question is if it hits that wall before or after human-level (and/or dangerous) capabilities.
kastagg 4y ago
1. What is your alternative? 2. What is your definition of the problem?

Isinlor 4y ago

Have you heard about EfficientZero? This is the first algorithm that achieved super-human performance on Atari 100k actions benchmark. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement. So data efficiency was doubling roughly every year for the past 8 years. Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge. https://arxiv.org/abs/2111.00210 https://arxiv.org/abs/2111.00210 Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

stefan_ 4y ago

They have? That's the approach they are using? Because that doesn't mesh well with practical reality. Where AGI use Deep RL, it's to improve on vision tasks like object classification, none of them are making any driving decisions - that seems to remain the domain of I guess you could call it discrete logic.

robotresearcher 4y ago
Here's a survey paper from this year on Deep RL for autonomous driving. https://ieeexplore.ieee.org/document/9351818 https://ieeexplore.ieee.org/document/9351818 B. R. Kiran et al., "Deep Reinforcement Learning for Autonomous Driving: A Survey," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909-4926, June 2022, doi: 10.1109/TITS.2021.3054625. I haven't read the paper, so this is not a reading recommendation. Just posting as evidence that there is work in the area.

gwern 4y ago

> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervised Transformers do. Sorry if you were counting on 'RL doesn't work' for safety or anything. If you don't believe the basic existence proofs of things like OA5 or AlphaStar, which work only because things like larger batch sizes or more diverse agent populations magically make notoriously-unreliable archs work, you can look at Jones's beautiful AlphaZero scaling laws (plural) work https://arxiv.org/abs/2104.03113 https://arxiv.org/abs/2104.03113 , or browse through relevant papers https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restrict_sr=on&include_over_18=on https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restr... https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper Or GPT-f. Then you have stuff like Gato continuing to show scaling even in the Decision Transformer framework. Or consider instances of plugging pretrained models into RL agents, like SayCan-PaLM most recently.

pixelpoet 4y ago
While we have the mighty gwern on the line: do you believe we'll have AGI in <= 10 years?
- telotortium 4y ago
  Obviously yes: https://twitter.com/robbensinger/status/1541228487205236736 https://twitter.com/robbensinger/status/1541228487205236736
trention 4y ago
That scaling will eventually hit a wall. What was it about nerds and S-curves?
- dinosaurdynasty 4y ago
  Sure, most everything does. The question is if it hits that wall before or after human-level (and/or dangerous) capabilities.

kastagg 4y ago

1. What is your alternative? 2. What is your definition of the problem?