4 ms·

Go-explore is weird for me. The main reasons for it's success are all based on very specific thing that barely transfer to anything such as the ability to save

Go-explore is weird for me. The main reasons for it's success are all based on very specific thing that barely transfer to anything such as the ability to save a state and restart from it at anytime (in this case using features from the emulator). It look almost like an engineering project. I don't like bad-mouthing the work of others, so I kind of feel bad for saying that and maybe someone can prove me wrong: But is this not borderline "cheating" (in quotes because obviously there is no agreed-upon rules)? The goal never was to solve Montezuma's revenge just for the sake of it. We could have done that way before by using hard engineering. The interesting thing about this game is that it reveals inherent flaws in our reinforcement learning approach. And so, if you manage to solve this game using a domain-transferable approach, surely that means you have made some significant progress in RL. That's doesn't seem to be the case here?

svantana 8y ago
I thought so too at first, but here's my new thinking: The most efficient way to learn real world tasks (for say a robot) is to learn in a realistic simulated environment, which ideally can be controlled by the agent, e.g. rewinding state after a failure to correct that particular behaviour. Ideally the simulation is regularly calibrated against real experiments, including confidence intervals on the real and simulated sensor inputs. Given that model of learning, Uber's approach here is very reasonable.
- Voloskaya 8y ago
  Good point! Uber definitly has simulated road environments, so it makes sense indeed. Thanks for sharing.
lostmsu 8y ago
I had the same though after reading the article, thanks for a good explanation of it. This solution does not seem to bring us much closer to the possibly unsolvable problem of finding an ultimate reward for RL, as it is very specific to the game. If they manage to get that behavior emerge from something not so explicit though, that would be a huge achievement.

svantana 8y ago

I thought so too at first, but here's my new thinking: The most efficient way to learn real world tasks (for say a robot) is to learn in a realistic simulated environment, which ideally can be controlled by the agent, e.g. rewinding state after a failure to correct that particular behaviour. Ideally the simulation is regularly calibrated against real experiments, including confidence intervals on the real and simulated sensor inputs. Given that model of learning, Uber's approach here is very reasonable.

Voloskaya 8y ago
Good point! Uber definitly has simulated road environments, so it makes sense indeed. Thanks for sharing.

lostmsu 8y ago

I had the same though after reading the article, thanks for a good explanation of it. This solution does not seem to bring us much closer to the possibly unsolvable problem of finding an ultimate reward for RL, as it is very specific to the game. If they manage to get that behavior emerge from something not so explicit though, that would be a huge achievement.