4 ms·
This title is really confused - backpropagation is one implementation of automatic differentiation, and yields a gradient. What you do with that gradient next i
by brilee 8y ago
This title is really confused - backpropagation is one implementation of automatic differentiation, and yields a gradient. What you do with that gradient next is up to you - you can just follow it (steepest descent), or try other fancier second order techniques.
- electricslpnsld 8y agoCan you get second order with only the gradient? Even fancy accelerated gradient descents are still first order. I suppose there are BFGS-like quasi Newton methods, but those are still building up estimates of the second order info over multiple iterations.
- deleted 8y ago[deleted]
- vbarrielle 8y agoYou can get closer to second order if your error is a sum of squares of the form 0.5 * f^T * f, in which case you case compute the jacobian J of the fitness f, and approximate your hessian as J^T * J. That's the Gauss-Newton method and it's very effective if a linear approximation of f holds locally. That's an approximation but that's arguably a second order method, since it acknowledges the nonlinearity coming from the squared norm of f.