A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type o

A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate. [1] http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997