Of course, there's tons of theory and practice in nonlinear continuous optimization. It's interesting to see what machine learning practitioners will rediscove

Of course, there's tons of theory and practice in nonlinear continuous optimization. It's interesting to see what machine learning practitioners will rediscover or reinvent next. Quasi-Newton methods appear to be surfacing again. But you do get some very interesting improvements to vanilla gradient descent!