3 ms·

It's always useful to see different SGD methods written with a consistent nomenclature. A few thoughts: 1. Is the 1999 Qian paper on momentum really the most a

It's always useful to see different SGD methods written with a consistent nomenclature. A few thoughts: 1. Is the 1999 Qian paper on momentum really the most appropriate one, given the comparison of the publication date to NAG? As a cursory examination of the paper reveals, momentum has been used for a long time before 1999! 2. Similarly, the original NAG paper isn't about stochastic gradient descent and doesn't really use the equation as written. A more appropriate reference is to the Sutskever, Martens, Dahl and Hinton paper of 2013 http://proceedings.mlr.press/v28/sutskever13.html http://proceedings.mlr.press/v28/sutskever13.html which is the publication that described/reworked NAG in this way. 3. It's worth noting the caveats about AMSGrad: https://www.fast.ai/2018/07/02/adam-weight-decay/ https://www.fast.ai/2018/07/02/adam-weight-decay/

raibosome 8y ago
Thank you for pointing these out! I have made the necessary edits to the citations for (1) and (2) and republished the article. For (1), the paper by Sutskever et al., 2013 (http://proceedings.mlr.press/v28/sutskever13.pdf http://proceedings.mlr.press/v28/sutskever13.pdf) attributed the classical momentum to Polyak, 1964 (https://www.researchgate.net/publication/243648538_Some_methods_of_speeding_up_the_convergence_of_iteration_methods https://www.researchgate.net/publication/243648538_Some_meth...). A Distil article on momentum (https://distill.pub/2017/momentum/ https://distill.pub/2017/momentum/) also cited Polyak's paper and also included a much earlier publication in 1959 by Ruttishauser (https://doi.org/10.1007/978-3-0348-7224-9_2 https://doi.org/10.1007/978-3-0348-7224-9_2), but I will just make reference to Polyak's.
- _jamesm_ 8y ago
  Cool, glad to have helped. It seems I have caused a further minor point of confusion though, so a correction to the correction: The original Nesterov Accelerated Gradient paper is about gradient descent, it's just not about stochastic gradient descent. It's useful to make the distinction between "traditional" optimization methods like Newton's method, Conjugate Gradient, BFGS and so on, which are all gradient descent methods in the sense they require at least a full gradient calculation per iteration, and a lot of the algorithms mentioned in the article, which are suitable for stochastic gradient descent and originate with the deep learning community (there's nothing to stop them being used elsewhere, it just doesn't seem that common). Some extra (unnecessary) detail on NAG to put things into a bit more context, if you are so inclined: Although NAG has received a fair amount of theoretical attention, as far as I know it isn't widely used practically because its convergence properties rely on an exact line search and a rather specific schedule for its momentum-like term. The Sutskever contribution is interesting because first, it expressed the NAG formula in a way that could be easily understood by machine learning practitioners. Then, by moving the procedure a half step, they showed you could think of it as momentum followed by gradient descent.

raibosome 8y ago

Thank you for pointing these out! I have made the necessary edits to the citations for (1) and (2) and republished the article. For (1), the paper by Sutskever et al., 2013 (http://proceedings.mlr.press/v28/sutskever13.pdf http://proceedings.mlr.press/v28/sutskever13.pdf) attributed the classical momentum to Polyak, 1964 (https://www.researchgate.net/publication/243648538_Some_methods_of_speeding_up_the_convergence_of_iteration_methods https://www.researchgate.net/publication/243648538_Some_meth...). A Distil article on momentum (https://distill.pub/2017/momentum/ https://distill.pub/2017/momentum/) also cited Polyak's paper and also included a much earlier publication in 1959 by Ruttishauser (https://doi.org/10.1007/978-3-0348-7224-9_2 https://doi.org/10.1007/978-3-0348-7224-9_2), but I will just make reference to Polyak's.

_jamesm_ 8y ago
Cool, glad to have helped. It seems I have caused a further minor point of confusion though, so a correction to the correction: The original Nesterov Accelerated Gradient paper is about gradient descent, it's just not about stochastic gradient descent. It's useful to make the distinction between "traditional" optimization methods like Newton's method, Conjugate Gradient, BFGS and so on, which are all gradient descent methods in the sense they require at least a full gradient calculation per iteration, and a lot of the algorithms mentioned in the article, which are suitable for stochastic gradient descent and originate with the deep learning community (there's nothing to stop them being used elsewhere, it just doesn't seem that common). Some extra (unnecessary) detail on NAG to put things into a bit more context, if you are so inclined: Although NAG has received a fair amount of theoretical attention, as far as I know it isn't widely used practically because its convergence properties rely on an exact line search and a rather specific schedule for its momentum-like term. The Sutskever contribution is interesting because first, it expressed the NAG formula in a way that could be easily understood by machine learning practitioners. Then, by moving the procedure a half step, they showed you could think of it as momentum followed by gradient descent.