I don't think parallelizing SGD is that important. There are steeply diminishing and even negative returns (see the sharp minima paper) to increasing minibatch

I don't think parallelizing SGD is that important. There are steeply diminishing and even negative returns (see the sharp minima paper) to increasing minibatch size, so you wouldn't want to use hundreds of GPUs to run huge minibatches up to the size of the dataset; and you wouldn't want to split a model across multiple GPUs because no one has enough data+compute to train monstrous models which are dozens or hundreds of GB in size (if anything, one trend has been towards appreciating how powerful small well-trained NNs are and how grossly overparameterized many past NNs have been). What you would use that many GPUs for is hyperparameter optimization training many models in parallel using deep RL or evolutionary computation, asynchronous RL exploring many environments simultaneously, or supporting many researchers working on their own individual projects. None of which needs super networking parallelism stuff.