3 ms·
Stochastic gradient descent methods are the most-used for neural net fitting, so the most discussed. However, the methods you mention are sometimes used too. F
by imurray 10y ago
Stochastic gradient descent methods are the most-used for neural net fitting, so the most discussed.
However, the methods you mention are sometimes used too. For the closest thing to Newton's method applied to neural nets, search for "Hessian-free optimization". Hacky minibatch updates based on batch methods have been tried, and can work ok http://ai.stanford.edu/~quocle/LeNgiCoaLahProNg11.pdf http://ai.stanford.edu/~quocle/LeNgiCoaLahProNg11.pdf . Any method could also potentially be used with growing batch sizes, https://papers.nips.cc/paper/4308-statistical-tests-for-optimization-efficiency https://papers.nips.cc/paper/4308-statistical-tests-for-opti...
Minibatches are good for parallelizing computation, especially on GPUs. And batch methods can be less of a fiddle to tune. So the alternatives you mention are popular in some commercial settings, for cost functions with nasty curvature, or to make it easier to write fast code. I guess most people's experience is that the dead simple SGD methods often work well though, and that's why they keep returning to them.
- nimish 10y agoI wonder what the quasi Newton methods would look like if adapted to this framework
- imurray 10y agoBFGS is an example of a quasi-Newton method. L-BFGS is the popular "low-memory" variant. Any such method can be used with the two references I gave above. One could also attempt to approximate each of its internal computations to within some tolerance with minibatches. Work in that sort of direction here: https://ei.is.tuebingen.mpg.de/publications/mahhen2015 https://ei.is.tuebingen.mpg.de/publications/mahhen2015 Perhaps surprisingly, given how fundamental optimization is to so many fields, there isn't consensus on what's best. And as I previously hinted above, the answer may change with the hardware landscape.
- davidkretch 10y ago> Perhaps surprisingly, given how fundamental optimization is to so many fields, there isn't consensus on what's best. And as I previously hinted above, the answer may change with the hardware landscape. Not entirely to your point, but related, my understanding is that is no best at everything: methods that work well in one field won't necessarily work in others. See https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...
- imurray 10y agoI agree. But part of it is sociology: people use what their friends use.
- goerz 10y agoI always assumed that when people talk about gradient-descent, they actually mean L-BFGS, i.e. a quasi-Hessian. After all, it can be used as a black-box algorithm that only needs to be told the gradient. At least in quantum optimization, the "simple" (non-quasi-Newton) gradient approach almost never works, whereas L-BFGS does just fine. So, I'm confused why in machine learning, the situation would be different, or why one would choose not to use the "for free" enhancement that L-BFGS provides.
- imurray 10y agoIn machine learning "evaluating the gradient" means sweeping over the whole training set. For simple models, stochastic gradient descent will have found a good fit after one or two passes over the dataset. (For neural nets, tens to hundreds of passes over the dataset may be needed.) It's hard for batch L-BFGS to do much in the same number of gradient and function evaluations. No one uses batch steepest gradient descent, which would be the worst of all worlds. The discussion above was about making stochastic or mini-batch versions of algorithms like L-BFGS. It's possible, but not entirely straightforward, which is why "dumb" stochastic gradient descent is often used.