There was a good talk at NIPS this year about the difficulty of reproducing results and benchmarking in RL. In supervised learning you can easily compare result

There was a good talk at NIPS this year about the difficulty of reproducing results and benchmarking in RL. In supervised learning you can easily compare results on standard datasets. In RL though, once the actions of two policies diverge, you're at the whim of the PRNG. Even with the same random seeds, gradient descent hyperparameters, etc., it is not possible to meaningfully compare two policies with a single training run. Even if you hold everything else constant, random seed had a huge effect. Ideally papers would show aggregated data over 100 runs for each setup, but RL is too computationally expensive for that. It is frustrating, but also exciting because the field had so many open problems. On other hand, it really sucks that those with the 1000 machine cluster have such a huge advantage over smaller labs.