3 ms·

Note that this is very much geared toward machine learning. In many other descent algorithm applications, it would be sensible to use Conjugate Gradients, which

• nimish 8y ago
  Of course, there's tons of theory and practice in nonlinear continuous optimization. It's interesting to see what machine learning practitioners will rediscover or reinvent next. Quasi-Newton methods appear to be surfacing again. But you do get some very interesting improvements to vanilla gradient descent!