6 ms·
Deep Mind Playing Montezuma's Revenge with Intrinsic Motivation [video]
- jerf 10y agoIf you end up blocked by a popup blocker, or just don't feel like reading fluff, or like me you've got the browser locked down too tightly for their integrated video player to work, the paper is at https://arxiv.org/pdf/1606.01868v1.pdf https://arxiv.org/pdf/1606.01868v1.pdf and the video mentioned is (probably, since I didn't see the original) at https://www.youtube.com/watch?v=0yI2wJ6F8r0 https://www.youtube.com/watch?v=0yI2wJ6F8r0 .
- dang 10y agoThanks. We've changed the URL from http://www.wired.co.uk/article/google-ai-montezuma-revenge http://www.wired.co.uk/article/google-ai-montezuma-revenge. At first to https://arxiv.org/abs/1606.01868 https://arxiv.org/abs/1606.01868, but since everyone's going to want to watch the video, that seems a bit too abstruse. Hopefully someone will say meaningful things about the paper too.
- louhike 10y agoIt might be a interesting future if computers become powerful enough to be able to run this kind of IA. Then games companies might be able to use directly a generic IA like this instead of developing specific ones. But I suppose it will be hard to do and won't happen (if it does) before a long time.
- pgrote 10y agoThe text of the article is cut in half for me. Can anyone else confirm the same rendering behavior? http://i.imgur.com/FfcXAmi.png http://i.imgur.com/FfcXAmi.png I am on Windows 7 using Chrome 51.0.2704.79 m and the text issue occurs in incognito mode as well.
- david_for_you 10y agoWorks for me on Windows 7 with Chrome 51.0.2704.84 m.
- dragontamer 10y agoWin7 using Firefox. Text issue for me, I can't read it.
- folli 10y agoIs it really true that Montezuma's Revenge is more challenging for DeepMind than Go, as they mention in the article?
- tensor 10y agoIsn't DeepMind just an umbrella term like Watson? I would guess that AlphaGo and the software that learns games like this are distinct.
- nschucher 10y agoDeepMind is the name of an AI company that was acquired by Google. https://deepmind.com/ https://deepmind.com/ AlphaGo is the program that beat Go. These game playing algorithms are variations of Deep Q Learning Reinforcement Learning algos.
- sehugg 10y agoAnd here's the related paper, "Unifying Count-Based Exploration and Intrinsic Motivation": https://arxiv.org/abs/1606.01868 https://arxiv.org/abs/1606.01868 Tech journalism isn't at its best when it can't distinguish between WOPR and a team of researchers.
- mafribe 10y agoDeepMind's main intellectual property, the reason why it was acquired, was DeepMind's reinforcement algorithm. The present paper, like the work on AlphaGo, is about improving this reinforcement algorithm.
- clickok 10y agoI don't believe that's the case. The Atari results were achieved via Q-Learning with neural networks (DQN stands for Deep Q-Networks), both of which have been around for decades. The basic algorithm is something you can implement yourself fairly easily, but (as with most things involving neural nets) getting it to really perform takes a bit of expertise. So, DeepMind's greatest asset is they employ some really excellent people and have a substantial head start in terms of actually implementing AIs and getting them to work.
- discardorama 10y agoSo lets say I take the trained network, and flip the color of the pixels, and make some other cosmetic changes (keeping the game intact). Will the network then solve it in the first try?
- gwern 10y agoThe ALE agents are almost always using grey-scale pixels in the first place (to save on processing and NN space requirements), so flipping the color makes no difference.
- seanwilson 10y agoIs it learning the specific level layout or can it adapt when the screens are different and order swapped?
- gwern 10y agoDQN is stateless, so at every point it's reacting to just the current frame (technically, the average of the last 4 frames IIRC) being fed into a CNN and outputing a motion. It doesn't care about ordering all that much because it has no memory. In this case, it's probably learning stuff like 'if there's a key, go towards it' and 'if you have a key, go towards a barrier'. If each screen keeps working the same way, then I guess it would achieve about the same performance as it does now. New screens with new layouts/patterns of enemies will test how well its learned heuristics generalize, though. Personally, I think that someone should be trying a DQN with an RNN rather than CNN in it to see if that helps on the harder levels. Or better yet, combine it with some of the memory mechanisms and see if it can start doing some real long-term planning.
- SandB0x 10y agoI believe the input state is the past 4 frames stacked or concatenated.
- drcode 10y ago
- biztos 10y agoOn behalf of all who have "played" Montezuma's Revenge for real, let me say we look forward to the day when it can be outsourced to the Cloud. https://en.wikipedia.org/wiki/Montezuma%27s_Revenge https://en.wikipedia.org/wiki/Montezuma%27s_Revenge
- bra-ket 10y agoI have a feeling DeepMind AI will specialize in playing video games
- thisisdave 10y agoHere's one impressive counterexample: https://arxiv.org/abs/1604.08772 https://arxiv.org/abs/1604.08772
- Houshalter 10y agoOnce it gets really good at video games, you can then put it into a robot. How is the real world different than a video game? Navigating a 3d environment, planning, etc, are all important skills needed in "real life" AI. And deepmind has been making great progress on some of them.
- tyingq 10y agoSuper confused. Mon·te·zu·ma's re·venge (noun, informal) Diarrhea suffered by travelers, especially visitors to Mexico.
- schoen 10y agoIt's a video game for Apple II which is named after the illness. https://en.wikipedia.org/wiki/Montezuma's_Revenge_%28video_game%29 https://en.wikipedia.org/wiki/Montezuma's_Revenge_%28video_g...
- schoen 10y agoI'm confused by the subtitle of the linked article, which says "The AI system was able to solve the complex game in just four tries". But the video shows the AI dying many more times than that, and not ultimately winning the game, just learning to explore a portion of it successfully. What did the Wired editor mean by "solve" and "four tries"? (Or, for that matter, "complete"?)
- dang 10y agoGiven that this is what the paper (linked by jerf) says: within a fraction of the training time, our agent explores a significant portion of the first level and obtains significantly higher scores than previously published agents ... it's probably just the usual hype.
- wrsh07 10y agoOh, I think I get it now. I think they mean that in one play [ie 4 lives], it solves significantly more than any previous network.
- ChuckMcM 10y agoI look forward to the day when you are taunted on the Starcraft boards for playing "like an AI" :-) That said, game theory has always been an excellent way to analyze AI systems. And using "modern" games (which generally provide attractive skins over a classic mechanic) certainly makes it easier to watch/sit through. When DeepMind starts beating people playing Diplomacy then we'll know we're in a whole new game.
- bytefactory 10y agoIndeed, I can't wait for the day we see DeepMind (or its ilk) playing games like StarCraft, I expect it'll be quite uncanny.
- chongli 10y agoThis is interesting but it's still far from how a human would learn how to play the game. Humans don't have inbuilt rewards for Montezuma's Revenge, they acquire them culturally. How much of what was learned (by the machine, not the researchers) in playing Montezuma's Revenge could be applied to a game like Zelda? A human would instantly notice many of the connections between the two games: enemies that follow simple patterns and harm the player on contact, rooms that connect to one another laid out on a grid pattern, single use consumable keys that open doors, valuable gems to collect. Is the machine able to make any of these connections on its own?
- aab0 10y agoI think if you watch a child play anything, even a video game, you'll see that when it's not immediately obvious what to do, a human will just 'mess around' and 'try stuff out' to 'just to see what happens'. As the paper says, the idea of novelty bonuses in reinforcement learning draws directly from discussions of curiosity and play and intrinsic motivation in animals and humans.
- chongli 10y agoInitially, yes, but then as a human learns more about the game they begin to make plans at a much higher level than just the basic mechanics of moving around. In the video, the machine continues to die in very basic ways even after a hundred million iterations. A human who had played the game even a tiny fraction of that many times would be expected to complete an entire play-through flawlessly. And this still doesn't address my original question. A human who was able to master Montezuma's Revenge would have a dramatic advantage in learning and mastering the game Zelda compared to somebody who had played neither game before. What experience, if any, could this machine be expected to bring to Zelda, assuming no modification by the researchers?
- gwern 10y agoYou would get some boost if you use a NN factored to have game-specific module feeding into a universal game-playing NN: https://arxiv.org/abs/1511.06342 https://arxiv.org/abs/1511.06342 " Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning", Parisotto et al 2015 > The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.