3 ms·
Do machine learning cats use Nesterov methods? Add some inertia to gradient descent, and boom, kick butt theoretical benefits and super fast in practice.
by santaclaus 12y ago
Do machine learning cats use Nesterov methods? Add some inertia to gradient descent, and boom, kick butt theoretical benefits and super fast in practice.
- ajtulloch 12y agoYes - see e.g. "On the importance of initialization and momentum in deep learning" from ICML 2013 (http://www.cs.toronto.edu/~fritz/absps/momentum.pdf http://www.cs.toronto.edu/~fritz/absps/momentum.pdf) for an overview in the context of NNs.
- sjtrny 12y agoAccelerated gradient descent is pretty common everywhere.