3 ms·
What these guys doing is amazing. RL algos are so hard to get right and evaluate - they can be very unstable and depend on subtle details. I've tried it myself
by Dzugaru 9y ago
What these guys doing is amazing. RL algos are so hard to get right and evaluate - they can be very unstable and depend on subtle details. I've tried it myself (DQN and my version of actor critic) on simple tasks and still don't know if I had errors. Its like you're are chasing a moving target with a neural net - second-order instability :) With this code to pick apart and compare against (especially A2C) it'll be so much easier to hack RL.
- fmap 9y agoCan you expand a little bit on that? It seems like we are talking about very small programs here with simple specifications. What are some common problems with reinforcement learning implementations? Is it really just that you can't easily test your program?
- Dzugaru 9y ago1) You can't easily test it - bugs can lead to semi-working thing and you may need to run thousands of experiments to average 2) Many hyperparameters - it may be critical to get them right 3) No gradual implementation - it doesn't work until you get all the (often gimmicky) parts right. Take A3C for example - its paper version is parallel, may be hard to implement and debug in your language (and hardware) of choice. 4) Task and reward function choice may be hard - take a look on my question https://stackoverflow.com/questions/44781401/is-openai-gym-cartpole-v0-200th-step-interruption-unfair-to-agent https://stackoverflow.com/questions/44781401/is-openai-gym-c... Generally, RL algos performance is underwhelming in my experience, without heavy tuning (and so-called "domain expertise" in reward function), but I'm not an expert and OpenAI guys show that you can make the working thing like Dota2 bot, so they give me hope.