4 ms·

When it's playing a game (e.g. breakout) and it's being fed the pixels on the screen, how is the AI being told what the score/progress is? Does it have access t

When it's playing a game (e.g. breakout) and it's being fed the pixels on the screen, how is the AI being told what the score/progress is? Does it have access to some numeric metric that is chosen by the researchers for each game?

sanxiyn 10y ago
Yes. For example, Breakout saves score in address 76 and 77. Arcade Learning Environment has code to read the score, one per game. Code for Breakout is here: https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/src/games/supported/Breakout.cpp https://github.com/mgbellemare/Arcade-Learning-Environment/b...
- seanwilson 10y ago
  Thanks for that! So I was looking at the one for Montezuma's Revenge: https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/src/games/supported/MontezumaRevenge.cpp https://github.com/mgbellemare/Arcade-Learning-Environment/b... The reward seems to be only the game score which I believe is meant to be problematic for this game because your score doesn't go up very often (so you have to perform a lot of actions to get any feedback)? The lives are recorded but aren't part of the "get reward" method...are the lives factored into decision making somewhere else? Seems like knowing you just lost a life would really help decision making for such a game.
  - Phemist 10y ago
    I think novel game situations were counted into the reward function as well
    - gwern 10y ago
      Only in the specialty novelty-oriented DQN agents; the Montezuma's Revenge reward itself remains the same. The problem is defining 'novel' when every screen's pixels may be different (for example, imagine any game which has a timer ticking up).
      - Phemist 10y ago
        Not so much, a timer ticking up is only novel the first time round, and is unrelated to actions taken by the agent. Over multiple plays the agent will learn to ignore it. EDIT: It could be that the agent will just stand there the first few plays around, enjoying the novelty reward gained from simply watching the timer tick up. Haha
        gwern 10y ago
        The point is that every time the timer ticks, if you had defined 'novelty' as the bitstring representing the screen, you get a 'new' state. This multiplies against any blinking animations, any moving enemies, any of the agent's moves, any visible scores, etc. You get thousands or millions of unique framebuffer states before the agent has so much as left the first room in _Montezuma's Revenge_. And DQN already is RAM intensive for the experience replay buffer.
        Phemist 10y ago
        Thanks for the extra explanation. It seems I assumed too much about these Deep Q networks, due to some prior knowledge of the neuroscience related to RL. Although I do remember having seen a video about Montezuma's Revenge a week ago or so, where they talked about this exact problem. Anyway, it would seem to me that novelty functions that would allow the agent to ignore periodic changes in state such as timers going up, can be quite simple. A function that estimates novelty of individual bit values in the bitstring of the gamestate and then aggregates it, could quite easily account for timers, or generally elements changing periodically regardless of agent actions. A baseline novelty reward would seem relatively easy to predict by the agent and thus result in low prediction errors and low reinforcement of actions by agents. This function would have a linear space and time complexity to the length of the gamestate, and fairly naive & simple to use, but would get the job done I think? P.S Just wanted to thank you for the work you've put into your website, it's very informative and always a great starting point to dive deeper into topics you cover!
        gwern 10y ago
msohcw 10y ago
Yes, it's fed the score as the reward value used. If I'm not wrong, they didn't normalise it across games for the initial paper but normalised it to some range for some of the following research experiments.
- cosmoharrigan 10y ago
  The original papers [1][2] for the deep Q-network used reward clipping: "As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged". [1] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf [2] http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_et_al.pdf http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...
Dolores12 10y ago
They can measure time you played. More time in game mean higher score.

sanxiyn 10y ago

Yes. For example, Breakout saves score in address 76 and 77. Arcade Learning Environment has code to read the score, one per game. Code for Breakout is here: https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/src/games/supported/Breakout.cpp https://github.com/mgbellemare/Arcade-Learning-Environment/b...

seanwilson 10y ago
Thanks for that! So I was looking at the one for Montezuma's Revenge: https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/src/games/supported/MontezumaRevenge.cpp https://github.com/mgbellemare/Arcade-Learning-Environment/b... The reward seems to be only the game score which I believe is meant to be problematic for this game because your score doesn't go up very often (so you have to perform a lot of actions to get any feedback)? The lives are recorded but aren't part of the "get reward" method...are the lives factored into decision making somewhere else? Seems like knowing you just lost a life would really help decision making for such a game.
- Phemist 10y ago
  I think novel game situations were counted into the reward function as well
  - gwern 10y ago
    Only in the specialty novelty-oriented DQN agents; the Montezuma's Revenge reward itself remains the same. The problem is defining 'novel' when every screen's pixels may be different (for example, imagine any game which has a timer ticking up).
    - Phemist 10y ago
      Not so much, a timer ticking up is only novel the first time round, and is unrelated to actions taken by the agent. Over multiple plays the agent will learn to ignore it. EDIT: It could be that the agent will just stand there the first few plays around, enjoying the novelty reward gained from simply watching the timer tick up. Haha
      - gwern 10y ago
        The point is that every time the timer ticks, if you had defined 'novelty' as the bitstring representing the screen, you get a 'new' state. This multiplies against any blinking animations, any moving enemies, any of the agent's moves, any visible scores, etc. You get thousands or millions of unique framebuffer states before the agent has so much as left the first room in _Montezuma's Revenge_. And DQN already is RAM intensive for the experience replay buffer.
        Phemist 10y ago
        Thanks for the extra explanation. It seems I assumed too much about these Deep Q networks, due to some prior knowledge of the neuroscience related to RL. Although I do remember having seen a video about Montezuma's Revenge a week ago or so, where they talked about this exact problem. Anyway, it would seem to me that novelty functions that would allow the agent to ignore periodic changes in state such as timers going up, can be quite simple. A function that estimates novelty of individual bit values in the bitstring of the gamestate and then aggregates it, could quite easily account for timers, or generally elements changing periodically regardless of agent actions. A baseline novelty reward would seem relatively easy to predict by the agent and thus result in low prediction errors and low reinforcement of actions by agents. This function would have a linear space and time complexity to the length of the gamestate, and fairly naive & simple to use, but would get the job done I think? P.S Just wanted to thank you for the work you've put into your website, it's very informative and always a great starting point to dive deeper into topics you cover!
        gwern 10y ago

msohcw 10y ago

Yes, it's fed the score as the reward value used. If I'm not wrong, they didn't normalise it across games for the initial paper but normalised it to some range for some of the following research experiments.

cosmoharrigan 10y ago
The original papers [1][2] for the deep Q-network used reward clipping: "As the scale of scores varies greatly from game to game, we clipped all positive rewards at 1 and all negative rewards at -1, leaving 0 rewards unchanged". [1] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf [2] http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_et_al.pdf http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_...

Dolores12 10y ago

They can measure time you played. More time in game mean higher score.