I think there is room to exchange ideas between the nonlinear programming and ML communities for sure. Specifically for momentum, if I understand it right it's

I think there is room to exchange ideas between the nonlinear programming and ML communities for sure. Specifically for momentum, if I understand it right it's a particular way of perturbing step size and gradient steps to prevent oscillation. There are some other good examples of this used by many gradient-descent optimizers. For example: https://www.cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lecture5.pdf https://www.cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lect... There's part of me that wonders if one interesting way forward for deep learning is a minibatch form of BFGS or SNOPT.