I thought it was uncontroversial that stochastic gradient descent can offer at least some protection against getting trapped in an undesirable local minima. Eve

I thought it was uncontroversial that stochastic gradient descent can offer at least some protection against getting trapped in an undesirable local minima. Even in 2011 when I took my first machine learning course we were taught that while SGD was first developed to make large problems more tractable, the noise introduced by evaluating the gradient on only a subset of the data can potentially help the solver escape local minima. I don't think the analogy with simulated annealing is that much of a stretch. The learning rate is like temperature and it's pattern of decrease is like an annealing schedule. Each step is decided from the sampling distribution for the full gradient and there is a nonzero probability of climbing uphill. As the learning rate decreases the solver becomes more likely to get stuck in a local minima. See here [1] for instance, and here [2] for a more detailed discussion of how SGD can help escape local minima that goes beyond this analogy. The analogy is incomplete and of course SGD can still get stuck in local minima, but my understanding is that there is a reasonable consensus that the noise introduced by SGD can help a solver find better local minima and the main area of contention is in trying to understand exactly why this is the case. [1] https://leon.bottou.org/publications/pdf/nimes-1991.pdf https://leon.bottou.org/publications/pdf/nimes-1991.pdf [2] https://proceedings.mlr.press/v80/kleinberg18a/kleinberg18a.pdf https://proceedings.mlr.press/v80/kleinberg18a/kleinberg18a....