Stochastic gradient actually performs better in terms of generalization performance! https://arxiv.org/abs/1509.01240 https://arxiv.org/abs/1509.01240 The intu

Stochastic gradient actually performs better in terms of generalization performance! https://arxiv.org/abs/1509.01240 https://arxiv.org/abs/1509.01240 The intuition for it is that when optimizing a machine learning objective all the way to machine precision, you at some point cross an (unknown) threshold where you are over-optimizing the parameters to the particular model class you're using, but that's probably too much faith in your model specification. So stochastic optimization and early stopping (before gradient is zero) provides a form of regularization.