3 ms·
I may be misunderstanding something, but, from the Wikipedia article, checking that the Wolfe conditions hold requires calculation of the gradient at the propos
by dual_basis 7y ago
I may be misunderstanding something, but, from the Wikipedia article, checking that the Wolfe conditions hold requires calculation of the gradient at the proposed new point as well. If you're going to do that, you may as well move in the direction of the new gradient, no?
- kxyvr 7y agoNo, I don't think so. The derivative required by the Wolfe-conditions is a directional derivative. Even if we're using AD to calculate the derivative, this can be calculated using a forward-mode method, which is cheaper than a reverse mode calculation for the gradient. Of course, even if we wanted to spent the time to calculate the full gradient, if the step is satisfies the Wolfe condition, we can simply save this value and use it on the next iteration. Certainly, if we decide to use a stochastic method, we may not want this new gradient with the current randomization, but then we can always fall back to calculating the directional derivative, which is cheaper. Finally, even if we really, really don't want to do any additional derivatives, we could simply use a trust-region method, which doesn't require a derivative calculation to guarantee convergence. Yes, the model used in a trust-region method has derivatives, but all of these derivatives should be cached when computing the direction in the first place. But, really, if it's bad step because the Wolfe conditions say it's a bad step, then moving to that point is counter-productive, both in theory and practice. We might as well know that before we go there.
- dual_basis 7y agoYour suggestion is very interesting. Here is some recent related work which seems to have explored the idea in more depth: https://arxiv.org/pdf/1703.10034.pdf https://arxiv.org/pdf/1703.10034.pdf