4 ms·
Can you get second order with only the gradient? Even fancy accelerated gradient descents are still first order. I suppose there are BFGS-like quasi Newton meth
by electricslpnsld 8y ago
Can you get second order with only the gradient? Even fancy accelerated gradient descents are still first order. I suppose there are BFGS-like quasi Newton methods, but those are still building up estimates of the second order info over multiple iterations.
- deleted 8y ago[deleted]
- vbarrielle 8y agoYou can get closer to second order if your error is a sum of squares of the form 0.5 * f^T * f, in which case you case compute the jacobian J of the fitness f, and approximate your hessian as J^T * J. That's the Gauss-Newton method and it's very effective if a linear approximation of f holds locally. That's an approximation but that's arguably a second order method, since it acknowledges the nonlinearity coming from the squared norm of f.