6 ms·
In the article, Uber says "Surprisingly, despite considerable research effort, so far no algorithm has obtained a score greater than 0 on Pitfall." I played pi
by degenerate 8y ago
In the article, Uber says "Surprisingly, despite considerable research effort, so far no algorithm has obtained a score greater than 0 on Pitfall."
I played pitfall as a kid and it seems quite straightforward for a computer to solve... jump over the puddle. I'd like if someone could talk more about this game in particular, specifically why it's so hard for AI to solve. Any interesting paper/link on the subject?
- Voloskaya 8y agoI don't know Pitfall, but generally speaking, the games that are hard to solve are the ones with sparse rewards. In the game, do you get points when you overcome some obstacles? If not and you only get points for successfully completing the entire level, then it becomes very hard for an agent to learn: How can it know if it is improving within the level itself?
- degenerate 8y agoAh, that would be it. There's no way to tell you are heading to victory.
- PretzelPirate 8y agoShouldn’t the AI be rewarded for exploring places it hasn’t explored before? It seems like that would help it progress in Pitfall.
- Voloskaya 8y agoYes it's an active topic of research: https://openreview.net/pdf?id=rJNwDjAqYX https://openreview.net/pdf?id=rJNwDjAqYX
- jaster 8y agoThis is glossed upon in the article: "AI researchers have typically tried to get around the issues posed by by Montezuma’s Revenge and Pitfall! by instructing reinforcement-learning algorithms to explore randomly at times, while adding rewards for exploration—what’s known as “intrinsic motivation.” But the Uber researchers believe this fails to capture an important aspect of human curiosity. “We hypothesize that a major weakness of current intrinsic motivation algorithms is detachment,” they write. “Wherein the algorithms forget about promising areas they have visited, meaning they do not return to them to see if they lead to new states."
- amelius 8y agoHere's a nice video about it: https://www.youtube.com/watch?v=0Ey02HT_1Ho https://www.youtube.com/watch?v=0Ey02HT_1Ho
- b_tterc_p 8y agoI would do the following. 1) Train a model to predict what happens next given an input 2) Each frame, predict the next frames given all possible inputs 3) Choose the input that maximizes uncertainty I would expect this to learn to avoid deaths relatively quickly. It doesn’t need to be good at knowing what will happen next, just better at recognizing specific dead ends (e.g. spikes or holes).
- pas 8y agohttps://blog.openai.com/reinforcement-learning-with-prediction-based-rewards/ https://blog.openai.com/reinforcement-learning-with-predicti...
- eksemplar 8y agoCouldn't you use negative rewards, by scoring deaths so that it's better to avoid it?
- rhcom2 8y agoWouldn't the optimal strategy be just to stand still then?
- luckydata 8y agoYou can reward moving as far to the right as possible or something like that.
- johnvanommen 8y agoThis sounds like the plot to War Games: An AI tries to figure out how to win at nuclear war, then concludes that: "Nuclear war is a strange game in which the only winning move is not to play."
- thedaemon 8y agoI too have never scored greater than 0 on Pitfall! Harder than Ghost n' Goblins.