3 ms·

That's just a combination of GPU model and data parallelism, which have been supported for about 2 years in Torch among others. You can add in the outer product

by jhj 10y ago

That's just a combination of GPU model and data parallelism, which have been supported for about 2 years in Torch among others. You can add in the outer product trick for fc layers as well. https://github.com/soumith/imagenet-multiGPU.torch/blob/master/models/alexnetowt.lua https://github.com/soumith/imagenet-multiGPU.torch/blob/mast... nccl is in Torch as well, but it doesn't always win, and has some weird interactions regarding streams and other such things with its use of persistent kernels. https://github.com/torch/cunn/blob/master/DataParallelTable.lua https://github.com/torch/cunn/blob/master/DataParallelTable.... However, this feels more of a benchmarking thing now. Networks tend to be over-parameterized and redundant in many ways; sending less data with much deeper / smaller parameter networks has been where the action is (e.g., residual networks, GoogLeNet, etc.), or to non-synchronous forms of communication among workers. Trying to squeeze out every last drop of GPU/GPU bandwidth is not as important as iterating on the architecture and learning algorithms themselves.

varelse 10y ago
I don't use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my balloon) to avoid working with a vendor that was trying to gouge us on servers: they beat NCCL by ~50%. That said, most people couldn't write them so I advise NCCL. You're the third person to tell me NCCL has ish(tm), fascinating. And sure, you could do the outer product trick. You could use non-deterministic ASGD. And you can do a lot of weird(tm) tricks. But why do these (IMO ad-hoc and potentially sketchy task-specific) things when there's an efficient way to parallelize the original network in a deterministic manner for training that allows performance on par with a DGX-1 server? Because for me, the action is in automagically discovering the optimal deterministic distribution of the computation so researchers don't need to worry about it. And IMO that's where the frameworks fail currently.

by jhj 10y ago

varelse 10y ago

I don't use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my balloon) to avoid working with a vendor that was trying to gouge us on servers: they beat NCCL by ~50%. That said, most people couldn't write them so I advise NCCL. You're the third person to tell me NCCL has ish(tm), fascinating. And sure, you could do the outer product trick. You could use non-deterministic ASGD. And you can do a lot of weird(tm) tricks. But why do these (IMO ad-hoc and potentially sketchy task-specific) things when there's an efficient way to parallelize the original network in a deterministic manner for training that allows performance on par with a DGX-1 server? Because for me, the action is in automagically discovering the optimal deterministic distribution of the computation so researchers don't need to worry about it. And IMO that's where the frameworks fail currently.