Batch Gradient descent can be parallelized the problem is that all the gradients need to be calculated for each step. So if one node is slow, the real time proc

Batch Gradient descent can be parallelized the problem is that all the gradients need to be calculated for each step. So if one node is slow, the real time processing will be as fast as the slowest node. Furthermore batchsize is limited to memory and training hyerparams are dependent to some extent on the hardware. So training in the real world is done on parallel identical hardware nodes. Farming it out with current algorithms does not make sense.