3 ms·

In the video (at about 4:30), they identify an issue with novelty search - the agent is rewarded for finding areas of the map with irrelevant animations like wa

by Imnimo 3y ago

In the video (at about 4:30), they identify an issue with novelty search - the agent is rewarded for finding areas of the map with irrelevant animations like waves in water, swaying flowers or NPCs walking randomly, as each new frame will have lots of changed pixels. The author fixes the problem by just increasing the threshold for changed pixels needed to receive a reward. This is a good idea, but might run into issues in later parts of the game where the screen is mostly animated water. Interestingly, this problem has been studied previously - an OpenAI paper called it the "noisy TV problem", since an agent could in principle get maximum novelty reward for just staring at a TV screen showing random noise. The solution they proposed (https://openai.com/research/reinforcement-learning-with-prediction-based-rewards https://openai.com/research/reinforcement-learning-with-pred...) called Random Network Distillation, uses a pair of networks, one learned and the other randomly initialized. The goal of the learned network is to predict the output of the randomly initialized one on the newly observed frame. If the output is easily predicted, it means the frame is similar to what's already been seen before, even if it has different TV noise or water patterns. They got some pretty good results on Atari and other games, so it'd be interesting to see if it works well on Pokemon.

armada651 3y ago
> The author fixes the problem by just increasing the threshold for changed pixels needed to receive a reward. This is a good idea, but might run into issues in later parts of the game where the screen is mostly animated water. They could've also implemented this by reading the game memory. Pokemon Red internally keeps track of which areas the player has visited before. I think the reason they opted for the visual approach is because they didn't want to give the AI any knowledge that isn't readily accessible to the player. Or they simply wanted more granularity than what the game keeps track of.
- kevinmchugh 3y ago
  They read game state from memory for a number of things they might've done visually (health bars). Recognizing when the player has entered a new screen seems inbounds. Humans can recognize that easily.
cozis 3y ago
The problem is that when a map has animated tiles with different frequencies, the overall animation period of the map is the least common multiple of the periods of the single tiles. For instance, a map covered in tiles with 10 frames will also have 10 frames. But if the map has, say, both tiles with 10 and 11 frames, then the period of the map is 10*11=110. Basically, maps containing different animated tiles can potentially have lots of different screen states, while maps with the same tiles have a low number of screen states. I think maps full of the same tile repeated wouldn't be a problem so the threshold might work in pretty much the whole game.

by Imnimo 3y ago

armada651 3y ago

> The author fixes the problem by just increasing the threshold for changed pixels needed to receive a reward. This is a good idea, but might run into issues in later parts of the game where the screen is mostly animated water. They could've also implemented this by reading the game memory. Pokemon Red internally keeps track of which areas the player has visited before. I think the reason they opted for the visual approach is because they didn't want to give the AI any knowledge that isn't readily accessible to the player. Or they simply wanted more granularity than what the game keeps track of.

kevinmchugh 3y ago
They read game state from memory for a number of things they might've done visually (health bars). Recognizing when the player has entered a new screen seems inbounds. Humans can recognize that easily.

cozis 3y ago

The problem is that when a map has animated tiles with different frequencies, the overall animation period of the map is the least common multiple of the periods of the single tiles. For instance, a map covered in tiles with 10 frames will also have 10 frames. But if the map has, say, both tiles with 10 and 11 frames, then the period of the map is 10*11=110. Basically, maps containing different animated tiles can potentially have lots of different screen states, while maps with the same tiles have a low number of screen states. I think maps full of the same tile repeated wouldn't be a problem so the threshold might work in pretty much the whole game.