3 ms·
Fair enough, but when neither Intel nor AMD is bothering to put out competitive software/hardware solutions for deep learning, which is apparently going to be a
by varelse 10y ago
Fair enough, but when neither Intel nor AMD is bothering to put out competitive software/hardware solutions for deep learning, which is apparently going to be a ~$40B market by 2020, I predict continued dominance here. That said, since AMD is getting beaten by a factor of 2-3 in performance while Intel's Accelerator/FPGA solutions are getting beaten by a factor of 5-10, AMD has a real opportunity to be the competitor to NVIDIA here.
And despite questions about it, GTX 1080 will be a fine deep learning board, especially if the framework engineers get off their lazy butts and implement Alex Krizhevsky's one weird trick algorithm, an approach which allows an 8-GPU Big Sur with 12.5GB/s P2P bandwidth to go toe to toe with an 8-GPU DGX-1 on training AlexNet.
http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
- nl 10y agoPretty sure most frameworks support "one weird trick" style learning. Eg Caffe: https://github.com/BVLC/caffe/blob/master/docs/multigpu.md https://github.com/BVLC/caffe/blob/master/docs/multigpu.md
- varelse 10y ago"The current implementation uses a tree reduction strategy. e.g. if there are 4 GPUs in the system, 0:1, 2:3 will exchange gradients, then 0:2 (top of the tree) will exchange gradients, 0 will calculate updated model, 0->2, and then 0->1, 2->3." Nope... This is the stupid way... They should be using ring reductions, handily provided by the NCCL library: https://github.com/NVIDIA/nccl https://github.com/NVIDIA/nccl
- db1024 10y agoI wouldn't call it stupid. It was fine for when they wrote it. I would say their solution is a bit outdated though. I am a bit surprised that NVIDIA isn't allocating more engineering support to Caffe to make sure they support all of NVIDIA's latest features in a timely fashion. Torch at least tries to use nccl for nn.DataParallelTable by default.
- varelse 10y agoIMO no it wasn't. Ring reductions are the sort of things the MPI guys had a decade ago. This is (once again IMO) the sort of thing that happens when one doesn't do lit searches and then Juergen Schmidhuber has a temper tantrum when we all find out one of his students probably has a good claim to have previously invented something supposedly new. My heuristic whenever I think I've invented something is to aggressively exploit Google to prove me wrong. It usually does so, but sometimes in very unusual ways. Glad to hear about Torch though. Do they have automagic model parallelization (not data parallelization) as well?
- db1024 10y ago> Do they have automagic model parallelization (not data parallelization) as well? Not that I know of. Regarding claims of novelty, I don't think the Caffe maintainers are claiming that their multi-gpu update method is novel or even very good. I think it was just the easiest thing someone could think of. I think Flickr originally wrote the multi-gpu extensions and the maintainers simply accepted the pull request. If anything, I think the maintainers are more than willing to listen to people in the scientific computing community with experience. Even better if they have a pull request in hand. But otherwise, they probably won't know about better methods and won't care.
- varelse 10y agoSo from what I've heard unofficially from friends at NVIDIA, they've been pretty hard to work with such that NVIDIA just ended up making their own fork of Caffe for use within DIGITS. https://github.com/NVIDIA/caffe https://github.com/NVIDIA/caffe Am I missing something here?
- jhj 10y agoThat's just a combination of GPU model and data parallelism, which have been supported for about 2 years in Torch among others. You can add in the outer product trick for fc layers as well. https://github.com/soumith/imagenet-multiGPU.torch/blob/master/models/alexnetowt.lua https://github.com/soumith/imagenet-multiGPU.torch/blob/mast... nccl is in Torch as well, but it doesn't always win, and has some weird interactions regarding streams and other such things with its use of persistent kernels. https://github.com/torch/cunn/blob/master/DataParallelTable.lua https://github.com/torch/cunn/blob/master/DataParallelTable.... However, this feels more of a benchmarking thing now. Networks tend to be over-parameterized and redundant in many ways; sending less data with much deeper / smaller parameter networks has been where the action is (e.g., residual networks, GoogLeNet, etc.), or to non-synchronous forms of communication among workers. Trying to squeeze out every last drop of GPU/GPU bandwidth is not as important as iterating on the architecture and learning algorithms themselves.
- varelse 10y agoI don't use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my balloon) to avoid working with a vendor that was trying to gouge us on servers: they beat NCCL by ~50%. That said, most people couldn't write them so I advise NCCL. You're the third person to tell me NCCL has ish(tm), fascinating. And sure, you could do the outer product trick. You could use non-deterministic ASGD. And you can do a lot of weird(tm) tricks. But why do these (IMO ad-hoc and potentially sketchy task-specific) things when there's an efficient way to parallelize the original network in a deterministic manner for training that allows performance on par with a DGX-1 server? Because for me, the action is in automagically discovering the optimal deterministic distribution of the computation so researchers don't need to worry about it. And IMO that's where the frameworks fail currently.