5 ms·

I was really excited when I heard Carmack was focusing on AI and am really looking forward to watching this when the video is up - but just from looking at the

I was really excited when I heard Carmack was focusing on AI and am really looking forward to watching this when the video is up - but just from looking at the slides it seems like he tried to build a system that can play the Atari? Seems like a fun project, but curious what will come out of it or if there is an associated paper being released.

johnb231 1y ago
Atari games are widely used in Reinforcement Learning (RL) research as a standard benchmark. https://github.com/Farama-Foundation/Arcade-Learning-Environment https://github.com/Farama-Foundation/Arcade-Learning-Environ... The goal is to develop algorithms that generalize to other tasks.
- sigmoid10 1y ago
  They were highly used. OpenAI even included them in their RL Gym library back in the old days when they were still doing open research. But if you look at this leaderboard from 7 (yes, seven!) years ago [1], most of them were already solved way beyond human capabilities. But we didn't get a really useful general purpose algorithm out of it. As an AI researcher, I always considered Atari a fun academic exercise, but nothing more. Similar to how recognising characters using convnets was cool in the nineties and early 00s, but didn't give us general purpose image understanding. Only modern GPUs and massive training datasets did. Nowadays most cutting-edge RL game research focuses on much more advanced games like Minecraft which is thought to be better suited. But I'm pretty sure it's still not enough. Even role-playing GTA VI won't be. We probably need a pretty advanced physical simulation of the real world before we can get agents to handle the real world. But that means solving the problem of generating such an environment first, because you can't train on the actual real world due to the sample inefficiency of all current algorithms. Nvidia is doing some really interesting research in this direction by combining physics simulation and image generation models to simulate an environment, while getting accuracy and diversity at the same time into training data. But it still feels like some key ingredient is missing. [1]https://github.com/cshenton/atari-leaderboard https://github.com/cshenton/atari-leaderboard
  - mschuster91 1y ago
    > But it still feels like some key ingredient is missing. Continuous training is the key ingredient. Humans can use existing knowledge and apply it to new scenarios, and so can most AI. But AI cannot permanently remember the result of its actions in the real world, and so its body of knowledge cannot expand. Take a toddler and an oven. The toddler has no concept of what an oven is other than maybe that it smells nice. The toddler will touch the oven, notice that it experiences pain (because the oven is hot) and learn that oven = danger. Place a current AI in a droid toddler body? It will never learn and keep touching the oven as soon as the information of "oven = danger" is out of the context window. For some cases this inability to learn is actually desirable. You don't want anyone and everyone to be able to train ChatGPT unsupervised, otherwise you get 4chan flooding it with offensive crap like they did to Tay [1], but for AI that physically interacts with the meatspace, constant evaluation and learning is all but mandatory if it is to safely interact with its surroundings. "Dumb" robots run regular calibration cycles for their limbs to make sure they are still aligned to compensate for random deviations, and so will AI robots. [1] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
    - sigmoid10 1y ago
      This kind of context management is not that hard, even when building LLMs. Especially when you have huge windows like we do today. Look at how ChatGPT can remember things permanently after you said them once using a function call to edit the permanent memory section inside the context. You can also see that in Anthropic's latest post on Claude 4 where it learns to play Pokemon. The only remaining issue here is maybe how to diffuse explicit knowledge from the stored context into the weights. Andrej Karpathy wrote a good piece on this recently. But personally I believe this might not even be necessary if you can manage your context well enough and see it more like RAM while the LLM is the CPU. For your example you can then always just fetch such information from a permanent storage like a VDB and load it into context once you enter an area in the real world.
      - deleted 1y ago
        [deleted]
cryptoz 1y ago
DeepMind’s original demos were also of Atari gameplay.
modeless 1y ago
He says they will open source it which is cool. I agree that I don't understand what's novel here. Playing with a physical controller and camera on a laptop GPU in real time is cool, and maybe that hasn't specifically been done before, but it doesn't seem surprising that it is possible. If it is substantially more sample efficient, or generalizable, than prior work then that would be exciting. But I'm not sure if it is?
- RetroTechie 1y ago
  Maybe that's exactly his goal: not to come up with something that beats the competition, but play with the architecture, get a feel for what works & what doesn't, how various aspects affect the output, and improve on that. Design more efficient architectures, or come up with something that has unique features compared to other models. If so, scaling up may be more of a distraction rather than helpful (besides wasting resources). I hope he succeeds in whatever he's aiming for.
tschillaci 1y ago
You will find many agents that solved (e.g., finished, reached high score) atari games, but there is still so much more work to do in the field. I wrote my Master's thesis on how to learn from few interactions with the game, so that if the algorithm is ported to actual robots they don't need to walk and fall for centuries before learning behaviors. I think there is more research to do on higher levels of generalization: when you know how to play a few video games, you quickly understand how to play a new one intuitively, and I haven't seen thorough research on that.
- lo0dot0 1y ago
  I can tell you right now without any research that video game designers reuse interface patterns and game mechanics that were already known when making new games. Those patterns and mechanics are also often analogies for real life allowing humans to intuitively play the games. If people can't play your game intuitively, they might say it's a bad game.
  - Jensson 1y ago
    So why can't AI learn those and reapply the same understanding to new games?
albertzeyer 1y ago
His goal was not just to solve Atari games. That was already done. His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.
- mike_hearn 1y ago
  But how is this different to what NVIDIA have already done? They have robots that can achieve arbitrary and fluid actions in the real world by training NNs in very accurate GPU simulated environments using physics engines. Moving a little Atari stick around seems like not much compared to sorting through your groceries etc. The approach NVIDIA are using (and other labs) clearly works. It's not going to be more than a year or two now before robotics is as solved as NLP and chatbots are today.
  - albertzeyer 1y ago
    I think he argues that they would not be able to play Atari games this way (I don't know; maybe I also misunderstood). But also, he argues a lot about sample efficiency. He wants to develop algorithms/methods/models which can learn much faster / with much fewer data.
gadders 1y ago
I want smarter NPCs in games.

johnb231 1y ago

Atari games are widely used in Reinforcement Learning (RL) research as a standard benchmark. https://github.com/Farama-Foundation/Arcade-Learning-Environment https://github.com/Farama-Foundation/Arcade-Learning-Environ... The goal is to develop algorithms that generalize to other tasks.

sigmoid10 1y ago
They were highly used. OpenAI even included them in their RL Gym library back in the old days when they were still doing open research. But if you look at this leaderboard from 7 (yes, seven!) years ago [1], most of them were already solved way beyond human capabilities. But we didn't get a really useful general purpose algorithm out of it. As an AI researcher, I always considered Atari a fun academic exercise, but nothing more. Similar to how recognising characters using convnets was cool in the nineties and early 00s, but didn't give us general purpose image understanding. Only modern GPUs and massive training datasets did. Nowadays most cutting-edge RL game research focuses on much more advanced games like Minecraft which is thought to be better suited. But I'm pretty sure it's still not enough. Even role-playing GTA VI won't be. We probably need a pretty advanced physical simulation of the real world before we can get agents to handle the real world. But that means solving the problem of generating such an environment first, because you can't train on the actual real world due to the sample inefficiency of all current algorithms. Nvidia is doing some really interesting research in this direction by combining physics simulation and image generation models to simulate an environment, while getting accuracy and diversity at the same time into training data. But it still feels like some key ingredient is missing. [1]https://github.com/cshenton/atari-leaderboard https://github.com/cshenton/atari-leaderboard
- mschuster91 1y ago
  > But it still feels like some key ingredient is missing. Continuous training is the key ingredient. Humans can use existing knowledge and apply it to new scenarios, and so can most AI. But AI cannot permanently remember the result of its actions in the real world, and so its body of knowledge cannot expand. Take a toddler and an oven. The toddler has no concept of what an oven is other than maybe that it smells nice. The toddler will touch the oven, notice that it experiences pain (because the oven is hot) and learn that oven = danger. Place a current AI in a droid toddler body? It will never learn and keep touching the oven as soon as the information of "oven = danger" is out of the context window. For some cases this inability to learn is actually desirable. You don't want anyone and everyone to be able to train ChatGPT unsupervised, otherwise you get 4chan flooding it with offensive crap like they did to Tay [1], but for AI that physically interacts with the meatspace, constant evaluation and learning is all but mandatory if it is to safely interact with its surroundings. "Dumb" robots run regular calibration cycles for their limbs to make sure they are still aligned to compensate for random deviations, and so will AI robots. [1] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
  - sigmoid10 1y ago
    This kind of context management is not that hard, even when building LLMs. Especially when you have huge windows like we do today. Look at how ChatGPT can remember things permanently after you said them once using a function call to edit the permanent memory section inside the context. You can also see that in Anthropic's latest post on Claude 4 where it learns to play Pokemon. The only remaining issue here is maybe how to diffuse explicit knowledge from the stored context into the weights. Andrej Karpathy wrote a good piece on this recently. But personally I believe this might not even be necessary if you can manage your context well enough and see it more like RAM while the LLM is the CPU. For your example you can then always just fetch such information from a permanent storage like a VDB and load it into context once you enter an area in the real world.
    - deleted 1y ago
      [deleted]

cryptoz 1y ago

DeepMind’s original demos were also of Atari gameplay.

modeless 1y ago

He says they will open source it which is cool. I agree that I don't understand what's novel here. Playing with a physical controller and camera on a laptop GPU in real time is cool, and maybe that hasn't specifically been done before, but it doesn't seem surprising that it is possible. If it is substantially more sample efficient, or generalizable, than prior work then that would be exciting. But I'm not sure if it is?

RetroTechie 1y ago
Maybe that's exactly his goal: not to come up with something that beats the competition, but play with the architecture, get a feel for what works & what doesn't, how various aspects affect the output, and improve on that. Design more efficient architectures, or come up with something that has unique features compared to other models. If so, scaling up may be more of a distraction rather than helpful (besides wasting resources). I hope he succeeds in whatever he's aiming for.

tschillaci 1y ago

You will find many agents that solved (e.g., finished, reached high score) atari games, but there is still so much more work to do in the field. I wrote my Master's thesis on how to learn from few interactions with the game, so that if the algorithm is ported to actual robots they don't need to walk and fall for centuries before learning behaviors. I think there is more research to do on higher levels of generalization: when you know how to play a few video games, you quickly understand how to play a new one intuitively, and I haven't seen thorough research on that.

lo0dot0 1y ago
I can tell you right now without any research that video game designers reuse interface patterns and game mechanics that were already known when making new games. Those patterns and mechanics are also often analogies for real life allowing humans to intuitively play the games. If people can't play your game intuitively, they might say it's a bad game.
- Jensson 1y ago
  So why can't AI learn those and reapply the same understanding to new games?

albertzeyer 1y ago

His goal was not just to solve Atari games. That was already done. His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.

mike_hearn 1y ago
But how is this different to what NVIDIA have already done? They have robots that can achieve arbitrary and fluid actions in the real world by training NNs in very accurate GPU simulated environments using physics engines. Moving a little Atari stick around seems like not much compared to sorting through your groceries etc. The approach NVIDIA are using (and other labs) clearly works. It's not going to be more than a year or two now before robotics is as solved as NLP and chatbots are today.
- albertzeyer 1y ago
  I think he argues that they would not be able to play Atari games this way (I don't know; maybe I also misunderstood). But also, he argues a lot about sample efficiency. He wants to develop algorithms/methods/models which can learn much faster / with much fewer data.

gadders 1y ago

I want smarter NPCs in games.