Jorge Nocedal begs to differ: http://users.iems.northwestern.edu/~nocedal/publications.html http://users.iems.northwestern.edu/~nocedal/publications.htm... > T

Jorge Nocedal begs to differ: http://users.iems.northwestern.edu/~nocedal/publications.html http://users.iems.northwestern.edu/~nocedal/publications.htm... > This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations. Disclosure: I was his student for a while and in his lab for my whole PhD. My takeaway from studying under him was that even very very approximate hessian information (either the Hessian itself or the solves) is enough for pretty amazing convergence rates, in terms of flops / time.