3 ms·
This is interesting but it's still far from how a human would learn how to play the game. Humans don't have inbuilt rewards for Montezuma's Revenge, they acquir
by chongli 10y ago
This is interesting but it's still far from how a human would learn how to play the game. Humans don't have inbuilt rewards for Montezuma's Revenge, they acquire them culturally. How much of what was learned (by the machine, not the researchers) in playing Montezuma's Revenge could be applied to a game like Zelda? A human would instantly notice many of the connections between the two games: enemies that follow simple patterns and harm the player on contact, rooms that connect to one another laid out on a grid pattern, single use consumable keys that open doors, valuable gems to collect. Is the machine able to make any of these connections on its own?
- aab0 10y agoI think if you watch a child play anything, even a video game, you'll see that when it's not immediately obvious what to do, a human will just 'mess around' and 'try stuff out' to 'just to see what happens'. As the paper says, the idea of novelty bonuses in reinforcement learning draws directly from discussions of curiosity and play and intrinsic motivation in animals and humans.
- chongli 10y agoInitially, yes, but then as a human learns more about the game they begin to make plans at a much higher level than just the basic mechanics of moving around. In the video, the machine continues to die in very basic ways even after a hundred million iterations. A human who had played the game even a tiny fraction of that many times would be expected to complete an entire play-through flawlessly. And this still doesn't address my original question. A human who was able to master Montezuma's Revenge would have a dramatic advantage in learning and mastering the game Zelda compared to somebody who had played neither game before. What experience, if any, could this machine be expected to bring to Zelda, assuming no modification by the researchers?
- gwern 10y agoYou would get some boost if you use a NN factored to have game-specific module feeding into a universal game-playing NN: https://arxiv.org/abs/1511.06342 https://arxiv.org/abs/1511.06342 " Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning", Parisotto et al 2015 > The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.
- mafribe 10y agoA human would instantly notice many ... But isn't that exactly what Schmidthuberian intrinsic motivation is about? In Schmidthuber's account, which inspired the paper under discussion, intrinsic motivation is measured by the improvements (= better compression) to a predictive world model made by the learning algorithm. [1] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation.
- rybosome 10y agoThis ability to apply knowledge gained in one domain to another is called transfer learning, and it's not really known how to do it well for AI. My layman's understanding of memory-augmented neural-networks (e.g. neural turing machines) suggests that this is a promising avenue, but there are no clear answers yet.
- wrsh07 10y agoI think this technique -- in the title of the paper -- would transfer somewhat to other games, because it essentially encourages the player to go places it hasn't been [without simultaneously having it explore everything thoroughly]. So you use existing tools to encourage the player to survive, and then you add additional motivation to encourage the player to learn what it needs to do [ie explore]. And exploration is a common theme in games like Zelda. I actually think that this technique will generalize to learning lots of games [eg the same engine might play Zelda & Montezuma's Revenge], but I don't think techniques in this vein alone will allow it to learn how to play one game and then immediately understand how to play the other. [but "transfer learning" is a separate, challenging problem in AI]
- chongli 10y agoThat's a shame. For a human, each game played helps to make the person better at all games. Play enough adventure games, for example, and one begins to recognize so many of the patterns and tropes at work that the solutions to puzzles jump out obviously and immediately in a single play-through. Moreover, we already have machines that are extremely proficient at solving very complex games when given enough context: expert systems. One very notable example is the bot which successfully completed the game NetHack [0]. Would DeepMind's novelty-based reward technique work for NetHack? [0] https://github.com/krajj7/BotHack https://github.com/krajj7/BotHack
- wrsh07 10y agoAgreed. I think it's a pretty big open problem for ML / AI. How can learning about one thing help you do better at a related thing? Because you're absolutely right that a human who is good at one platformer will take very little time to adjust to a new one. I don't know much about NetHack, but seeing that it's a Roguelike, I think these new techniques should definitely be tried on it or something similar.