I know distributed neural nets are 10000 miles out there for most, but I just want to add a few nuggets for those considering it. I work with distributed deep

I know distributed neural nets are 10000 miles out there for most, but I just want to add a few nuggets for those considering it. I work with distributed deep nets quite a bit. It's a different animal than training on a GPU. I am working on benchmarks with my framework deeplearning4j now. That aside, a few neat references/projects that will be digestible for people. For those of you already in neural net land, there's a few key takeaways when doing distributed neural nets: parameter averaging across mini batches (depending on the algorithm) adagrad momentum [1] Project Adam: http://www.wired.com/2014/07/microsoft-adam/ http://www.wired.com/2014/07/microsoft-adam/ [2]: Associated Paper: https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14... [3]: Hogwild algorithm: http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf [4]: A variation of this I use called Iterative Reduce done by my partner Josh Patterson: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-reduce https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-re... [5]: Sandblaster LBFGS by Dean and Co. http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2...