We gave a talk on this at hadoop summit recently[1]. This is also what I've implemented in deeplearning4j [2]. The approach called iterative reduce is basicall

We gave a talk on this at hadoop summit recently[1]. This is also what I've implemented in deeplearning4j [2]. The approach called iterative reduce is basically training on mini batches in parallel and averaging the outcomes. The trick is to do it in multiple iterations on each mini batch averaging the parameters at runtime. It's a surprisingly simple technique that works really well. Unfortunately, it's not as async as sandblaster LBFGS by dean and co[3]. Andrew himself has told me that the gradient was unstable with this approach though. Right now, distributed training of neural nets is still like the wild west. I can see a few approaches coming up over time though. [1] https://www.youtube.com/watch?v=h2Ixpfn-DTg https://www.youtube.com/watch?v=h2Ixpfn-DTg [2] http://deeplearning4j.org/ http://deeplearning4j.org/ [3]: http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2...