3 ms·
Copy of the abstract for quick local reference: > Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of
by mdp2021 5y ago
Copy of the abstract for quick local reference:
> Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic differentiation algorithms that also includes the forward mode. We present a method to compute gradients based solely on the directional derivative that one can compute exactly and efficiently via the forward mode. We call this formulation the forward gradient, an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent. We demonstrate forward gradient descent in a range of problems, showing substantial savings in computation and enabling training up to twice as fast in some cases.
--
They have an implementation of the "forward gradient" for PyTorch. Is any implementation in C available?
- GistNoesis 5y agoYou can use Tapenade in tangent mode if you want to generate forward derivative in the desired direction. http://www-tapenade.inria.fr:8080/tapenade/ http://www-tapenade.inria.fr:8080/tapenade/
- Fronzie 5y agohttp://ceres-solver.org/ http://ceres-solver.org/ works well, in my experience. It does require code to be templated, which might increase computation time. It is possible to explicitly define derivatives for individual functions, this can help preventing numerical instabilities in corner cases.
- cyber_kinetist 5y agoForward-mode differentiation is easy to implement in C++ with templates, operator overloading, and dual numbers (https://en.wikipedia.org/wiki/Automatic_differentiation#Automatic_differentiation_using_dual_numbers https://en.wikipedia.org/wiki/Automatic_differentiation#Auto...). Some libraries such as autodiff (https://github.com/autodiff/autodiff https://github.com/autodiff/autodiff) and CppAD (https://github.com/coin-or/CppAD https://github.com/coin-or/CppAD) use this method to calculate gradients and Jacobians.