3 ms·

w.r.t. deeplearning4j: Isn't it impractical to use multiple compute nodes for parallelization, rather than GPUs when it comes to training deep nets?

agibsonccc 12y ago
Distributed GPUs ;). I'll be adding GPU support here shortly. First I'm working on the core nets, since everything is vectorized, swapping out the matrix implementations will be easy. Mainly focusing on a point release first. Just need to add Recursive Neural Tensor Nets[1]. Afterwards, distributed GPUs will be cake with the support that both NVIDIA[2] and AMD[3] support.You will find distributed compute is what google does[4] Edit: I'd like to mention ersatz[5] here as well. Edit: Thanks for the correction David. [1]: http://nlp.stanford.edu/sentiment/ http://nlp.stanford.edu/sentiment/ [2]: http://www.jcuda.org/jcuda/jcublas/JCublas.html http://www.jcuda.org/jcuda/jcublas/JCublas.html [3]: http://developer.amd.com/tools-and-sdks/heterogeneous-computing/aparapi/ http://developer.amd.com/tools-and-sdks/heterogeneous-comput... [4]: http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2... [5]: http://ersatzlabs.com http://ersatzlabs.com
- deleted 12y ago
  [deleted]
  - dave_sullivan 12y ago
    Since Ersatz got mentioned, I'd like to point out that we don't do true distributed GPU yet (it's planned). Most of the models support a single GPU with the RNN models supporting multiple GPUs on the same machine. Of course, that's still way way faster than your laptop. Distributing computation across GPUs has been addressed here: http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html http://techblog.netflix.com/2014/02/distributed-neural-netwo... We looked at a similar scheme, but didn't see it as practical on something like AWS Also here: http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgCatanzaro_icml2013.pdf http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgC... In this case, they are using infiniband, a high speed ethernet card/bus/whatever-you-call-it, in order to share data very quickly between multiple systems. This is basically like google's experiment but w/ GPUs (and thus way cheaper?) I'll let Adam speak to his strategy. It's an open research question but that doesn't mean there aren't any options or that those options are even terribly difficult given the right resources. Of course, real world performance is another story... whether you need and/or can use all that capacity, etc.
    - kastnerkyle 12y ago
      A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate. [1] http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
  - deleted 12y ago
    [deleted]
  - agibsonccc 12y ago
    Sure! So my stack currently uses iterative reduce[1]. The algorithm can then be swapped out which can either be gradient descent or conjugate gradient. The default is conjugate gradient. I also have all of the knobs like adagrad, drop out,.... In general, the idea is that you split up the training in to mini batches and then average the parameters after all of them have been computed. I will be working on downpour SGD as well. In my case, I use akka underneath for all of the distributed computing. Everything is inherently parallelizable using an optimal batch size algorithm (very similar to hadoops). From there, it's usually just a case of figuring out what you want your matrix computations made of. I also built it from the ground up to speak pretty much any data format. The idea is using a DataSetIterator that speaks hdfs, local file systems,.... This allows you to handle data locality independent of the algorithms. If you want more on the architecture, here's a pretty broad overview[2] Admittedly, not optimal, this is a very early release of the lib. After RNTN have been impled, I will be focusing on optimizations and improvements. [1]: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-Reduce https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-Re... [2]: http://deeplearning4j.org/architecture.html http://deeplearning4j.org/architecture.html

agibsonccc 12y ago

Distributed GPUs ;). I'll be adding GPU support here shortly. First I'm working on the core nets, since everything is vectorized, swapping out the matrix implementations will be easy. Mainly focusing on a point release first. Just need to add Recursive Neural Tensor Nets[1]. Afterwards, distributed GPUs will be cake with the support that both NVIDIA[2] and AMD[3] support.You will find distributed compute is what google does[4] Edit: I'd like to mention ersatz[5] here as well. Edit: Thanks for the correction David. [1]: http://nlp.stanford.edu/sentiment/ http://nlp.stanford.edu/sentiment/ [2]: http://www.jcuda.org/jcuda/jcublas/JCublas.html http://www.jcuda.org/jcuda/jcublas/JCublas.html [3]: http://developer.amd.com/tools-and-sdks/heterogeneous-computing/aparapi/ http://developer.amd.com/tools-and-sdks/heterogeneous-comput... [4]: http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2... [5]: http://ersatzlabs.com http://ersatzlabs.com

deleted 12y ago
[deleted]
- dave_sullivan 12y ago
  Since Ersatz got mentioned, I'd like to point out that we don't do true distributed GPU yet (it's planned). Most of the models support a single GPU with the RNN models supporting multiple GPUs on the same machine. Of course, that's still way way faster than your laptop. Distributing computation across GPUs has been addressed here: http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html http://techblog.netflix.com/2014/02/distributed-neural-netwo... We looked at a similar scheme, but didn't see it as practical on something like AWS Also here: http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgCatanzaro_icml2013.pdf http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgC... In this case, they are using infiniband, a high speed ethernet card/bus/whatever-you-call-it, in order to share data very quickly between multiple systems. This is basically like google's experiment but w/ GPUs (and thus way cheaper?) I'll let Adam speak to his strategy. It's an open research question but that doesn't mean there aren't any options or that those options are even terribly difficult given the right resources. Of course, real world performance is another story... whether you need and/or can use all that capacity, etc.
  - kastnerkyle 12y ago
    A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate. [1] http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
- deleted 12y ago
  [deleted]
- agibsonccc 12y ago
  Sure! So my stack currently uses iterative reduce[1]. The algorithm can then be swapped out which can either be gradient descent or conjugate gradient. The default is conjugate gradient. I also have all of the knobs like adagrad, drop out,.... In general, the idea is that you split up the training in to mini batches and then average the parameters after all of them have been computed. I will be working on downpour SGD as well. In my case, I use akka underneath for all of the distributed computing. Everything is inherently parallelizable using an optimal batch size algorithm (very similar to hadoops). From there, it's usually just a case of figuring out what you want your matrix computations made of. I also built it from the ground up to speak pretty much any data format. The idea is using a DataSetIterator that speaks hdfs, local file systems,.... This allows you to handle data locality independent of the algorithms. If you want more on the architecture, here's a pretty broad overview[2] Admittedly, not optimal, this is a very early release of the lib. After RNTN have been impled, I will be focusing on optimizations and improvements. [1]: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-Reduce https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-Re... [2]: http://deeplearning4j.org/architecture.html http://deeplearning4j.org/architecture.html