4 ms·

I do not have the experience to support or refute the author's claim, but he writes: "The paper does not clarify what “worker” means, but I assume it means 1 C

I do not have the experience to support or refute the author's claim, but he writes: "The paper does not clarify what “worker” means, but I assume it means 1 CPU." That seems like way under-powered to me. It's deepmind and so I would assume that 1 worker is 1 GPU/TPU node, meaning there are multiple GPUs for each worker. I could see how not having enough compute power could result in a poor solution

lacksconfidence 9y ago
The author also works for google brain though, so i doubt that they failed to consider GPU/TPU. Really "worker" is just too underspecified to say what it really means.
gwern 9y ago
DRL is different from regular DL in that it tends towards CPU-heavy, not GPU-heavy. It's hard to saturate a single GPU/TPU since you're using tiny little NNs and only once in a while updating them based on long episodes through the environment. It might not be using GPUs/TPUs at all! If you look at the algorithm which that DM paper is based on, PPO, the original OpenAI paper & implementation (https://blog.openai.com/openai-baselines-ppo/ https://blog.openai.com/openai-baselines-ppo/) doesn't use GPUs, it's pure-CPU. (They have a second version which adds GPU support.) Or in a DM vein, look at their latest IMPALA which you might've noticed on the front page a few days ago: https://arxiv.org/pdf/1802.01561.pdf https://arxiv.org/pdf/1802.01561.pdf Look at Table 1 pg5's computational resources for various agents: note how many of them have 0 GPUs whatsoever. Even the largest configuration, 500 CPUs, only saturates 1 Nvidia P100 GPU. (So, 'worker' could hypothetically refer to a server with X cores and 1 GPU processing them locally, but this is almost certainly not the case since it would imply scaling up to thousands of CPUs which is actually highly difficult and requires careful engineering like with IMPALA.)

lacksconfidence 9y ago

The author also works for google brain though, so i doubt that they failed to consider GPU/TPU. Really "worker" is just too underspecified to say what it really means.

gwern 9y ago

DRL is different from regular DL in that it tends towards CPU-heavy, not GPU-heavy. It's hard to saturate a single GPU/TPU since you're using tiny little NNs and only once in a while updating them based on long episodes through the environment. It might not be using GPUs/TPUs at all! If you look at the algorithm which that DM paper is based on, PPO, the original OpenAI paper & implementation (https://blog.openai.com/openai-baselines-ppo/ https://blog.openai.com/openai-baselines-ppo/) doesn't use GPUs, it's pure-CPU. (They have a second version which adds GPU support.) Or in a DM vein, look at their latest IMPALA which you might've noticed on the front page a few days ago: https://arxiv.org/pdf/1802.01561.pdf https://arxiv.org/pdf/1802.01561.pdf Look at Table 1 pg5's computational resources for various agents: note how many of them have 0 GPUs whatsoever. Even the largest configuration, 500 CPUs, only saturates 1 Nvidia P100 GPU. (So, 'worker' could hypothetically refer to a server with X cores and 1 GPU processing them locally, but this is almost certainly not the case since it would imply scaling up to thousands of CPUs which is actually highly difficult and requires careful engineering like with IMPALA.)