4 ms·

Many nonconvex problems are solved with more sophisticated methods, like L-BFGS. Are perturbations still a good thing?

• andbberger 9y ago
  Such methods are usually not practical for deep learning
  • petters 9y ago
    That is true, but I got the impression from the article that they had a wider range of problems in mind.
  • chestervonwinch 9y ago
    There's quite a bit of literature on putting quasi-newton methods like BFGS in an online setting, but I've never actually seen anyone use them except in the papers that present them. Maybe a few theoreticals are still being settled or maybe no one has written a robust piece of software for the folks who aren't optimization experts?
  • traviscj 9y ago
    Jorge Nocedal begs to differ: http://users.iems.northwestern.edu/~nocedal/publications.html http://users.iems.northwestern.edu/~nocedal/publications.htm... > This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations. Disclosure: I was his student for a while and in his lab for my whole PhD. My takeaway from studying under him was that even very very approximate hessian information (either the Hessian itself or the solves) is enough for pretty amazing convergence rates, in terms of flops / time.