3 ms·
No GPU support => not interesting. GPUs are an order of magnitude faster. If you're doing deep learning on CPUs you're wasting your time and/or money.
by modeless 11y ago
No GPU support => not interesting. GPUs are an order of magnitude faster. If you're doing deep learning on CPUs you're wasting your time and/or money.
- glxc 11y agoif your data exceeds memory then a GPU is worthless. distributed makes it scalable
- nightski 11y agoIt's not trivial to take any existing algorithm for training and scale it not only cross-GPU but also cross machine. Training tends to be very bandwith intensive. In fact that is the entire reason Nvidia is so heavily focused on this with their upcoming GPUs (Pascal). I am not saying it's not possible, just that it's a lot harder than "distributed makes it scalable".
- modeless 11y agoThis is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed. Furthermore, almost any technique you use to distribute and scale training will work just as well regardless of whether the computations are happening on CPUs or GPUs.
- gojomo 11y agoAt least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limits of GPUs: http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2...
- igul222 11y agoThis is also not quite right. Models whose parameters are too big to fit on one GPU can be trained by splitting them across multiple GPUs, as was done here, for example: http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf http://papers.nips.cc/paper/5346-sequence-to-sequence-learni...
- brianchu 11y agoAccording to the paper, the parameters fit on one GPU (or at least that one GPU was able to train the model). It was just too slow, so they trained on 8 GPUs in parallel. But those GPUs were still on the same machine (one node, multiple GPUs).
- limau 11y agoSINGA provides an abstraction for defining all known models,a neural net structure that is easy for model and data partitioning, and a parallelism model that supports both synchronous, asynchronous and hybrid training frameworks. Processing at each node could the be CPU, GPU, or CPU-GPU based.
- neel8986 11y agoIt is in their schedule for december http://singa.incubator.apache.org/develop/schedule.html http://singa.incubator.apache.org/develop/schedule.html
- brianchu 11y agoIt's only single node, multiple GPUs, though, which already exists in other frameworks.
- limau 11y agoWith the parallelism model and abstraction it has to support model/data or hybrid partitioning, and synchronous/asynchronous or hybrid training, it should be easy to extend to GPU cluster. However, training is required only periodically, and if it can be done on existing clusters as efficiently, why not?