3 ms·

Can someone convert "academia nerd language" down one notch into "regular nerd language". On the surface, this sounds interesting but despite being a huge nerd

• discardorama 12y ago
  They built a system that automatically learned to play 49 different games, without any human tweaking (aside from the original setup).
• brentjanderson 12y ago
  Here is a very rough translation from my POV: > The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. Reinforcement (rewards/punishments) is a highly effective way to train autonomous individuals to succeed in arbitrary environments. > To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. This kind of open-ended learning in a simulation is hard: Think of the number of inputs from all your senses being continually processed by the nervous system at a given time. Being able to take all those inputs, each of which changes meaning depending on context, to figure out what to do right now (while learning from the past in the process) is a hard problem, especially for a computer to solve. > Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. Humans and other animals have this figured out through our own brains ("dopaminergic neurons") and the combination of our senses and various parts of our nervous systems/biology (e.g. our reflexes respond faster than our cognition due to the hierarchy of autonomic nervous system response). > While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Past reinforcement-based algorithms have worked well, but require thorough understanding of the problem being solved, or for the problem to be relatively simple and predictable. > Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. By combining advances in deep neural network training with reinforcement learning in a "novel artificial agent" ("a deep Q-network"), our agent can learn sophisticated problems through only reinforcement learning. > We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. Our new approach works across 49 games using the same approach for each game (where each game presumably has different rules and dynamics), and is able to perform at the same level as a professional human being. > This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem.
  • j_m_b 12y ago
    |> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. |We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem. Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies.
    • chriswarbo 12y ago
      > Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies. Unfortunately I can't access the paper, and the ReadCube link seems to require Flash, but I'm assuming a "Q-network" is based on Q-learning. Q-learning is essentially a lookup table of states to actions, where we guess what state we're in from our observations and look up which action to perform that will get us the most reward. In that sense, it's clear that a Q-learning approach would struggle with long-term planning, since its memory only goes as far as 1 action. Of course there are ways to extend Q-learning, but these tend to destroy it's best feature: implementation efficiency. One nice alternative I've seen in recent years is Gradient Temporal Difference, which allows linear functions to be learned rather than just single actions, and retains lots of the performance properties of Q-learning (O(n) in the number of functions learned, off-policy learning, etc.).
      • DavidSJ 12y ago
        Q-learning is not a lookup table from states to actions. It's a learned mapping (using any method you want, such as neural networks) from agent history (i.e. the agent is allowed to consider the past) & action pairs, to estimates of future reward, learned through temporal differences between the prediction at time step t, and the sum of new prediction plus immediate reward at time step t+1. In this way, information can flow back in time as the agent learns that some observation is predictive of reward, and then learns some observation is predictive of that observation, and so on until it connects the reward with some action K time steps ago. So long-term planning is definitely possible. This does require that some relevant information be available at each intermediate time step to connect the actions with the ultimate reward. The nice thing about these Atari games is you can usually judge value just by what's immediately on the screen, and in this paper they only use the last three frames' worth of state. In a game requiring a memory for past sensory information (e.g. where information appears on the screen then disappears), this might not do so well, but that's more a matter of working memory than long-term planning, and a different Q-learning system could contain a working memory (e.g. if it was RNN-based).