Pretty much any GPU workload will quickly collide with Amdahl's Law w/r to spreading the work across multiple GPUs in short order. That's because 13 GB/s inter

Pretty much any GPU workload will quickly collide with Amdahl's Law w/r to spreading the work across multiple GPUs in short order. That's because 13 GB/s interconnect (P2P transfers) can only take one so far w/r to a problem that is O(n log n) or less unless the constant factor is huge. And that's why NVIDIA itself is now developing its own internal networking protocol: http://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/ http://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/ As a concrete example, it's why in his NIPS 2014 talk, Jeffrey Dean goes on and on and on about the different ways to parallelize the different sorts of layers of a deep neural network across multiple processors because the convolution layers are O(n^2)ish* (a convolution across ~n examples and ~n feature detectors) and and the fully connected layers are O(n^3)ish* (a matrix matrix product)... And that's just one problem domain. There are many more... *Totally abusing the terminology