3 ms·
Very cool work, deep mind wows me once again. One thing I wish they would make more explicit (and in all their papers that I've read for that matter) is how mu
by sgillen 8y ago
Very cool work, deep mind wows me once again.
One thing I wish they would make more explicit (and in all their papers that I've read for that matter) is how much computational power it takes to train these networks and achieve these results.
I'm not sure if this is something they usually leave out because it's not interesting, or because it's something that people that work with deep networks all the time (I.E. not me) already have a feel for.
As someone in a related field (sometimes using deep networks but not researching them for their own sake). I certainly would like to know which of deep minds results would be feasible to replicate using my research groups resources, and it can be hard to do that without spending a lot of time actually trying to replicate the results and benchmarking them on your hardware.
- 21 8y ago> I'm not sure if this is something they usually leave out because it's not interesting, or because it's something that people that work with deep networks all the time (I.E. not me) already have a feel for. I think they leave it out because it's a trade secret. Like the way they waited a long time before announcing that they used specialized hardware (TPU) for AlphaGo.
- dzdt 8y agoRetracted! See below; compute is disclosed and is not that crazy. For this project their training hardware used 4 Nvidia K80's. Original comment: I think they leave it out because otherwise the standard response to their work would be "no surprise they get better results than anyone else, they are using two orders of magnitude more compute time than anyone else!" Not highlighting the computational expense makes their results look more impressive.
- zerostar07 8y agoAlso, because they are focusing heavily on the RL part of the modeling. They obviously have obscene amounts of available compute, but that is not their competitive advantage.
- sgillen 8y agowhat exactly do you mean? Are you saying that RL requires less compute? I would say having an obscene amount of compute is definitely a big competitive advantage, especially over a lot of small academic research labs.
- zerostar07 8y ago> We train each GQN model simultaneously on 4 NVidia K80 GPUs for 2 million gradient steps. The values of the hyper-parameters used for optimisation are detailed in Table S1, and we show the effect of model size on final performance in Fig. S4. > The values of all hyper-parameters were selected by performing informal search. We did not perform a systematic grid search owing to the high computational cost.