I'm not convinced that SGD is the answer for large scale distributed problems. It is certainly the best answer for single node applications, but it might not be

I'm not convinced that SGD is the answer for large scale distributed problems. It is certainly the best answer for single node applications, but it might not be the best for distributed training. The search for effective optimization algorithms in machine learning is ongoing. The big problems for SGD at massively parallel scale: * SGD does not parallelize well * SGD is a first order method affected by ill-conditioning * SGD is too noisy when high accuracy is desired