4 ms·

|> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to exce

by j_m_b 12y ago

|> This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. |We've created a universal learning algorithm that can take a multitude of inputs and consistently respond correctly without having to re-define the model for each game or problem. Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies.

chriswarbo 12y ago
> Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies. Unfortunately I can't access the paper, and the ReadCube link seems to require Flash, but I'm assuming a "Q-network" is based on Q-learning. Q-learning is essentially a lookup table of states to actions, where we guess what state we're in from our observations and look up which action to perform that will get us the most reward. In that sense, it's clear that a Q-learning approach would struggle with long-term planning, since its memory only goes as far as 1 action. Of course there are ways to extend Q-learning, but these tend to destroy it's best feature: implementation efficiency. One nice alternative I've seen in recent years is Gradient Temporal Difference, which allows linear functions to be learned rather than just single actions, and retains lots of the performance properties of Q-learning (O(n) in the number of functions learned, off-policy learning, etc.).
- DavidSJ 12y ago
  Q-learning is not a lookup table from states to actions. It's a learned mapping (using any method you want, such as neural networks) from agent history (i.e. the agent is allowed to consider the past) & action pairs, to estimates of future reward, learned through temporal differences between the prediction at time step t, and the sum of new prediction plus immediate reward at time step t+1. In this way, information can flow back in time as the agent learns that some observation is predictive of reward, and then learns some observation is predictive of that observation, and so on until it connects the reward with some action K time steps ago. So long-term planning is definitely possible. This does require that some relevant information be available at each intermediate time step to connect the actions with the ultimate reward. The nice thing about these Atari games is you can usually judge value just by what's immediately on the screen, and in this paper they only use the last three frames' worth of state. In a game requiring a memory for past sensory information (e.g. where information appears on the screen then disappears), this might not do so well, but that's more a matter of working memory than long-term planning, and a different Q-learning system could contain a working memory (e.g. if it was RNN-based).

by j_m_b 12y ago

chriswarbo 12y ago

> Further along in the article, it turns out it is actually a subset of games which don't involve long term planning strategies. Unfortunately I can't access the paper, and the ReadCube link seems to require Flash, but I'm assuming a "Q-network" is based on Q-learning. Q-learning is essentially a lookup table of states to actions, where we guess what state we're in from our observations and look up which action to perform that will get us the most reward. In that sense, it's clear that a Q-learning approach would struggle with long-term planning, since its memory only goes as far as 1 action. Of course there are ways to extend Q-learning, but these tend to destroy it's best feature: implementation efficiency. One nice alternative I've seen in recent years is Gradient Temporal Difference, which allows linear functions to be learned rather than just single actions, and retains lots of the performance properties of Q-learning (O(n) in the number of functions learned, off-policy learning, etc.).

DavidSJ 12y ago
Q-learning is not a lookup table from states to actions. It's a learned mapping (using any method you want, such as neural networks) from agent history (i.e. the agent is allowed to consider the past) & action pairs, to estimates of future reward, learned through temporal differences between the prediction at time step t, and the sum of new prediction plus immediate reward at time step t+1. In this way, information can flow back in time as the agent learns that some observation is predictive of reward, and then learns some observation is predictive of that observation, and so on until it connects the reward with some action K time steps ago. So long-term planning is definitely possible. This does require that some relevant information be available at each intermediate time step to connect the actions with the ultimate reward. The nice thing about these Atari games is you can usually judge value just by what's immediately on the screen, and in this paper they only use the last three frames' worth of state. In a game requiring a memory for past sensory information (e.g. where information appears on the screen then disappears), this might not do so well, but that's more a matter of working memory than long-term planning, and a different Q-learning system could contain a working memory (e.g. if it was RNN-based).