3 ms·

This is great. Using HTML5 games in a headless browser makes a lot of sense because the need for VNC is circumvented. However, I think that while OpenAI's imple

This is great. Using HTML5 games in a headless browser makes a lot of sense because the need for VNC is circumvented. However, I think that while OpenAI's implementation is certainly not the best, having access just the information on the screen is not a bad idea in itself as a (maybe optional) constraint. With access to the game's internal state we don't even need RL for solving a large number of games - algorithms like NEAT are sufficient.

Houshalter 9y ago
This project doesn't change that. The agents still only get screenshots of the game as far as I understand. However I think this approach is bad. Machine vision is a separate problem from reinforcement learning. You shouldn't need to be able to do both well. Machine vision consumes a ton of processing power and researcher time in figuring out the hyperparameters. And all it's doing is figuring out information that's already in memory like the location of various objects and the score. It really limits what can be done. E.g. the famous atari playing AIs by deepmind were limited to no memory and only knowing the last few frames, because backpropagating through thousands of frames was too expensive. Because of the way NNs work, it's trivial to separate out the machine vision into a separate module. So if you have a good RNN reinforcement learning system, you can easily add a machine vision learning system to it later if you need.
- unixpickle 9y ago
  In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401 https://arxiv.org/abs/1606.03401.

Houshalter 9y ago

This project doesn't change that. The agents still only get screenshots of the game as far as I understand. However I think this approach is bad. Machine vision is a separate problem from reinforcement learning. You shouldn't need to be able to do both well. Machine vision consumes a ton of processing power and researcher time in figuring out the hyperparameters. And all it's doing is figuring out information that's already in memory like the location of various objects and the score. It really limits what can be done. E.g. the famous atari playing AIs by deepmind were limited to no memory and only knowing the last few frames, because backpropagating through thousands of frames was too expensive. Because of the way NNs work, it's trivial to separate out the machine vision into a separate module. So if you have a good RNN reinforcement learning system, you can easily add a machine vision learning system to it later if you need.

unixpickle 9y ago
In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401 https://arxiv.org/abs/1606.03401.