5 ms·
192 GPUs eh. Interesting to compare that with the numbers being dropped in some of the Google Brain and Deepmind papers like 800 GPUs...
by gwern 10y ago
192 GPUs eh. Interesting to compare that with the numbers being dropped in some of the Google Brain and Deepmind papers like 800 GPUs...
- deepnotderp 10y agoAFAICT, this is a single supercomputer, not a cluster like the Google systems.
- scq 10y agoIt's 24 Nvidia DGX-1 servers, which contain 8 GPUs each. It's worth noting that Nvidia already have their own 124-node DGX-1 installation, which would have 992 GPUs.
- p1esk 10y agoWhat's the difference?
- deepnotderp 10y agoParallelism of sgd across multiple machines is a,highly non-trivial task.
- gwern 10y agoI don't think parallelizing SGD is that important. There are steeply diminishing and even negative returns (see the sharp minima paper) to increasing minibatch size, so you wouldn't want to use hundreds of GPUs to run huge minibatches up to the size of the dataset; and you wouldn't want to split a model across multiple GPUs because no one has enough data+compute to train monstrous models which are dozens or hundreds of GB in size (if anything, one trend has been towards appreciating how powerful small well-trained NNs are and how grossly overparameterized many past NNs have been). What you would use that many GPUs for is hyperparameter optimization training many models in parallel using deep RL or evolutionary computation, asynchronous RL exploring many environments simultaneously, or supporting many researchers working on their own individual projects. None of which needs super networking parallelism stuff.
- p1esk 10y agoSorry, I still don't understand: what's the difference between implementing SGD on a "cluster" and implementing it on a "supercomputer"? Is a supercomputer not a cluster of multiple machines?
- deleted 10y ago[deleted]
- stuntprogrammer 10y agoYep, he said while eyeballing a dashboard, there are plenty of larger private machines..
- jacquesm 10y agoI know this is not that site but I'd love a picture.