14 ms·
Human-level control through deep reinforcement learning
- nl 12y agoIs this a different paper to the original DeepMind video game paper? http://arxiv.org/abs/1312.5602 http://arxiv.org/abs/1312.5602
- sp332 12y agoYes, I can't access the full paper but at least the figures are different :) Edit: Ars Technica has a summary of this new paper. https://arstechnica.com/science/2015/02/ai-masters-49-atari-2600-games-without-instructions/ https://arstechnica.com/science/2015/02/ai-masters-49-atari-...
- Houshalter 12y agoThey claim good results on 49 games while the original paper only worked on a few IIRC.
- devindotcom 12y agoYes, they addressed this on the conference call I was on. That was a sort of 'initial results' they wanted to get out there, this is the full research. Same concept, explored more deeply.
- superfx 12y agoHere's a publicly-accessible link to the full paper: http://rdcu.be/cdlg http://rdcu.be/cdlg
- jmnhr 12y agoOnly the first page, the rest is blurred and has to be paid for.
- frandroid 12y agoLet it load...
- jmnhr 12y agoThey are using tricks. When I tried the first time the pdf was blurred and the page automatically opened the payment menu. On the second try it showed the entire pdf, disallowing download.
- leereeves 12y agoIt loaded completely for me the first time, but doesn't allow download. A step toward open access, but they're still trying to claim copyright and control the work of others.
- deleted 12y ago[deleted]
- bmh100 12y ago> ...the authors used the same algorithm, network architecture, and hyperparameters on each game... This is huge. It shows that the algorithm was able to generalize across multiple problem sets within the same domain of "playing Atari 2600 games", and not simply a "lucky" choice of algorithm, network architecture, or hyperparameters that a random search for each game might choose. This is also not a violation of the No Free Lunch (NFL) Theorem [1], because the domain is limited to playing Atari 2600 games, which share many characteristics. [1]: https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...
- bsdetector 12y ago> the algorithm was able to generalize across multiple problem sets Did it really? I think they reset it and retrained it for each game. I'd like to know how much more is needed to make one instance of the AI that can successfully play any of the games. To play all 49 games that it could learn, does it need to be an extra level deep? Or 49 times larger? Or 2^49 times more?
- ZeroFries 12y agoI think you could establish an upperbound at 50 times larger, because each set of "neurons" could be completely independent for each game, with 49 neurons to determine which set to use. If they have similar features (which they undoubtedly would), the size could drop a lot. Of course, I have no training in neural nets, and so my conclusions are reached from general understanding/reasoning.
- DavidSJ 12y agoWhat they mean by "the algorithm" is the learning algorithm, not the actual learned behavior. Historically in machine learning a great deal of hand tuning of learning algorithms is required for them to even be effective on a single problem, so it's impressive that a single algorithm was able to learn to play many different games effectively. Having a single instance of an algorithm learn to perform well on many different tasks is really a separate problem than what they're dealing with here.
- viggity 12y agoCan someone convert "academia nerd language" down one notch into "regular nerd language". On the surface, this sounds interesting but despite being a huge nerd I'm not really sure what the hell they're talking about.
- discardorama 12y agoThey built a system that automatically learned to play 49 different games, without any human tweaking (aside from the original setup).
- brentjanderson 12y agoHere is a very rough translation from my POV: > The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. Reinforcement (rewards/punishments) is a highly effective way to train autonomous individuals to succeed in arbitrary environments. > To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. This kind of open-ended learning in a simulation is hard: Think of the number of inputs from all your senses being continually processed by the nervous system at a given time. Being able to take all those inputs, each of which changes meaning depending on context, to figure out what to do right now (while learning from the past in the process) is a hard problem, especially for a computer to solve. > Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. Humans and other animals have this figured out through our own brains ("dopaminergic neurons") and the combination of our senses and various parts of our nervous systems/biology (e.g. our reflexes respond faster than our cognition due to the hierarchy of autonomic nervous system response). > While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Past reinforcement-based algorithms have worked well, but require thorough understanding of the problem being solved, or for the problem to be relatively simple and predictable. > Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. By combining advances in deep neural network training with reinforcement learning in a "novel artificial agent" ("a deep Q-network"), our agent can learn sophisticated problems through only reinforcement learning. > We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. Our new approach works across 49 games using the same approach for each game (where each game presumably has different rules and dynamics), and is able to perform at the same level as a professional human being. > This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem.
- discardorama 12y agoIs there a chance this paper will be available as PDF? I' finding it difficult to read the readcube version. :-(
- deleted 12y ago[deleted]
- j_m_b 12y agoIt is interesting how they are using various biological models to develop their own model. They gave their model a reward system and a memory. It will be interesting to see how far deep Q-networks can be extended and at what point they hit the wall of diminishing returns. |Nevertheless, games demanding more temporally extended planning strategies still constitute a major challenge for all existing agents including DQN. |Notably, the succesfsful integration of reinforcement learning with deep network architectures was critically dependent on our incorporation of a replay algorithm involving the storage and representations of recently experienced transitions. I am not for sure what data the replay algorithm has access to, but I wonder what happens if you extend the amount of data it has. This might be the brick wall this algorithm hits of diminishing returns. It would be interesting to hear what the authors think could help help improve how their model deals with temporally extended planning strategies. As someone who grew up on Atari, Nintendo and Sony this is pretty cool work.
- morenoh149 12y agoI expect it could go far. Mind you I only did parts of Artificial Intellegence: A modern approach but the Q-learning algorthim seems very flexible. https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Q-learning It basically keeps doing good stuff, while exploring to get out of local minimas.
- erostrate 12y agoThe code is online if you want to play with it. https://sites.google.com/a/deepmind.com/dqn/ https://sites.google.com/a/deepmind.com/dqn/ If you're interested, one of the main authors (David Silver) teaches a very good and intuitive introductory class on reinforcement learning at UCL: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html
- fchollet 12y agoInteresting that they're using Torch7. The code is pretty concise and readable, very cool stuff.
- yablak 12y agoThey wrote Torch7...
- fchollet 12y agoI'm sure a number of people who have contributed to Torch are working at DeepMind. However, Torch has been around for much longer than DeepMind (about 12 years at this point). Two of the major contributors to Torch, Ronan Collobert and Clement Farabet, were never DeepMind employees. To be fair, another major contributor to Torch is a co-author of this paper (Kavukcuoglu).
- sjtrny 12y agoWatch it play: http://www.nature.com/nature/journal/v518/n7540/extref/nature14236-sv1.mov http://www.nature.com/nature/journal/v518/n7540/extref/natur... http://www.nature.com/nature/journal/v518/n7540/extref/nature14236-sv2.mov http://www.nature.com/nature/journal/v518/n7540/extref/natur...
- dwaltrip 12y agoThis is so cool. I'd love to work on this stuff... Anyone know how hard it would be for someone who is fairly good at programming (works as a full stack developer and feels quite comfortable learning new things) and has strong math skills (undergrad degree) to break into this field? Is going back to school for a masters/phd the best way?
- ratsimihah 12y agoHow about you read the article and get a few books on the relevant topics? It would probably be much cheaper than going back to school.
- dwaltrip 12y agoVery good point. I taught myself web dev (now working at a pretty awesome startup) so I'm definitely familiar with that route. I have a few cool AI ideas I'm hoping to start spending more time in the coming months, and I have heard of some great online courses to check out. I was just curious as to how important institutional credentials are for this kind of thing, seeing as it much more academic than building CRUD web apps.
- meric 12y agoI think there are lots of AI competitions where you can join and make your name.
- dabeeeenster 12y agoI did an M.SC in CogSci a long way back (1998) and funnily enough my thesis area was very very close to this (reinforcement learning with different network topologies). The core area is pretty simple stuff to be honest. Obviously DeepMind are completely next-level but you can get pretty good results with basic understanding...
- nl 12y agoThe most interesting thing about this is that it shows significant progress towards goal-oriented AI. The fact this system is effectively learning what "win" means in the context of a game is something of a breakthrough.
- Houshalter 12y agoReinforcement learning is not new. The same algorithm was used in the 90s to beat backgammon. The only notable thing is they are using raw pixels and vision to play the games.
- eveningcoffee 12y agoI do not think that it figures out "what the win is" as the score parameter is explicitly made available to the algorithm. In some sense this paper even demonstrates that how simple the problem actually is. I think more important question is that what else can be modelled as such problem.
- eveningcoffee 12y agoI am wondering that what kind of real life problems could be modelled this way.
- phkahler 12y ago>> I am wondering that what kind of real life problems could be modelled this way. Driving a car? Making a taco? Working the checkout counter?
- eveningcoffee 12y agoSo could you define an cost function for these activities?
- craftit 12y agoI suspect setting up motivations for the AI is going to be a big research issue before too long. If you can write a simulator for the task you want to it solve, you should be able to train it. Often writing the simulator is much easier than solving a task itself. For example, the atlas robot, then can simulate it but struggle to control it.
- craftit 12y agoKeeping someone entertained in a game, the longer they play the more reward you give it. I was also thinking of hooking up a face recognition system, so it can be rewarded when the user smiles.
- craftit 12y agoAll sorts, we have a machine learning service which lets you do something similar with JSON messages. https://aiseedo.com/demos/cookiemonster/ https://aiseedo.com/demos/cookiemonster/
- Someone 12y agoFor comparison: http://www.cs.cmu.edu/~tom7/mario/ http://www.cs.cmu.edu/~tom7/mario/. That is way more of a hack, but I am not sure this is that big a step forward. Space invaders and breakout aren't the hardest games and I haven't heard a hard argument that it is just a matter of scale to create a machine that, say, plays chess.
- chriswarbo 12y agoThat's pretty cool (especially the ending of Tetris!), but the crucial difference is that it's trying to learn by observing what happens during a known-good run, then it sounds like some game-specific learning is performed. In contrast, the Arcade Learning Environment (the benchmark used in TFA) is completely autonomous; there is no human guidance or game-specific learning: anything it learns must work well across all of the games (ie. it's either generally useful, or it gets surrounded by a bunch of conditionals). In particular, the author of your link says he's surprised that the algorithm can sometimes get further than twice the length of his pre-recorded training sequence.
- TheEzEzz 12y agoThe biggest differences are this: 1. The Mario algo has direct access to the game state, and will only work for games where it has that game state access. The DeepMind algo plays directly from the screen pixels. That means DeepMind has to first learn to interpret time varying (!) visual information correctly, then deduce rules and good play strategy on top of that leaky abstraction. That's hard. It also means the algorithm can be applied to any game with a screen output, not just to an Atari. 2. The Mario algo is doing a direct search through move space. It can back up and explore a different branch of the tree and play differently to see a different outcome. When the DeepMind algo plays Atari, it can't undo a move that it just did. It has to make good choices, using intelligence, just like a human player would. The impressive thing here is not that it plays Atari games. You're right, we have had AIs that can do this for a long time, even better than this. The impressive thing is that it's a single AI algorithm that works for many games, and that is learning directly from the screen. We have not had anything like this before.
- sharemywin 12y agoPDF: http://arxiv.org/pdf/1312.5602v1.pdf http://arxiv.org/pdf/1312.5602v1.pdf
- p1esk 12y agoNo, this is a new paper.
- javierluraschi 12y agoI think qlearning is really interesting, I posted yesterday a simple implementation/demo in Javascript of qlearning. This paper goes way beyond qlearning by deducing states based on a deep neural network from the actual game rendering, really cool. Regardless, as a first intro to qlearning I had fun putting this together https://news.ycombinator.com/item?id=9105818 https://news.ycombinator.com/item?id=9105818
- javierluraschi 12y agoHere is the marketing side of this publication in which Google scientists (aquihired from Deepmind) have developed a way to outperform humans in Atari games: http://m.phys.org/news/2015-02-hal-bests-humans-space-invaders.html http://m.phys.org/news/2015-02-hal-bests-humans-space-invade...
- plinkplonk 12y agoIs the paper available anywhere to read without having to pay Nature? From the comments it seems as if everyone is able to read this but me! Even in their "readcube" access method, only the first page is (barely) visible, the rest seems blurred.
- albertzeyer 12y agoAn interesting critic by Schmidhuber about this publication: https://plus.google.com/100849856540000067209/posts/eLQf4KC97Bs https://plus.google.com/100849856540000067209/posts/eLQf4KC9...
- arvinjoar 12y agoSeems to be critiquing the claim that this is "groundbreaking" and not much else. Nice to get some further context though. :)
- craftit 12y agoIt is an amazingly powerful technique. We've been working on a service which lets you do this kind of learning with any JSON stream. You can see a demo here: https://aiseedo.com/demos/cookiemonster/ https://aiseedo.com/demos/cookiemonster/
- ya3r 12y agoThe amazing part of what DeepMind has achieved is its capability to learn from raw pixel input with deep convolutional neural networks, which as I understand it quite different from what you do. Still the reinforcement learning part is the same, but reinforcement learning was not the main contribution of this nature paper.
- craftit 12y agoIts not all that different, we take multiple asynchronous streams of messages integrate them into a coherent predictive model, and use that to feed the reinforcement learning. The messages can contain images, a simple case can be seen in the demo with a 1d vision sensor.