5 ms·

Is the article true in claiming that the model doesn't work if we increase the size of the paddle? Or change anything else?

goldenshale 8y ago
Yeah, pretty much. It might continue to work fine with very minor changes, but as we know how to design and build them today deep neural networks are often very sensitive to minor changes in the input distribution. The key insight here though is not about deep nets, but about our true progress towards AGI. These systems don't hypothesize about the best strategy to take, think about different approaches, formulate alternatives, etc. They basically memorize paths that have worked based on random exploration. We have what seems like a long way to go (nobody can know whether we are 1 or 100 key ideas away) before an AI already trained to play 5 Atari games can be turned towards a new game and play very well based on its experience with related genres. Today they are trained from scratch for each game, so although the model might be the same the AI is not able to transfer theories and strategies from one domain to another.
swframe2 8y ago
Note you can create a game where the simulated world has random variations and the RL algorithm will learn to handle it. If you don't train for it, obviously it won't learn it. Check out this video from mit/openai: https://www.youtube.com/watch?v=9EN_HoEk3KY https://www.youtube.com/watch?v=9EN_HoEk3KY The entire talk is interesting but the section at 21:40 talks about "Sim2Real with Meta Learning".
- obastani 8y ago
  It's not at all obvious that "if you don't train for it, it won't learn it". Humans do not at all learn in this way: we are very good at adapting existing knowledge to solve new tasks, and relatedly at learning new tasks from very little data. This challenge is a major issue with deep reinforcement learning (and maybe deep learning more broadly). It's unclear how we might surmount this problem, but I believe it'll involve some combination of model-based approaches and deep learning models that internally use more symbolic structures.

goldenshale 8y ago

Yeah, pretty much. It might continue to work fine with very minor changes, but as we know how to design and build them today deep neural networks are often very sensitive to minor changes in the input distribution. The key insight here though is not about deep nets, but about our true progress towards AGI. These systems don't hypothesize about the best strategy to take, think about different approaches, formulate alternatives, etc. They basically memorize paths that have worked based on random exploration. We have what seems like a long way to go (nobody can know whether we are 1 or 100 key ideas away) before an AI already trained to play 5 Atari games can be turned towards a new game and play very well based on its experience with related genres. Today they are trained from scratch for each game, so although the model might be the same the AI is not able to transfer theories and strategies from one domain to another.

swframe2 8y ago

Note you can create a game where the simulated world has random variations and the RL algorithm will learn to handle it. If you don't train for it, obviously it won't learn it. Check out this video from mit/openai: https://www.youtube.com/watch?v=9EN_HoEk3KY https://www.youtube.com/watch?v=9EN_HoEk3KY The entire talk is interesting but the section at 21:40 talks about "Sim2Real with Meta Learning".

obastani 8y ago
It's not at all obvious that "if you don't train for it, it won't learn it". Humans do not at all learn in this way: we are very good at adapting existing knowledge to solve new tasks, and relatedly at learning new tasks from very little data. This challenge is a major issue with deep reinforcement learning (and maybe deep learning more broadly). It's unclear how we might surmount this problem, but I believe it'll involve some combination of model-based approaches and deep learning models that internally use more symbolic structures.