4 ms·
Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at th
by Fede_V 10y ago
Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at the same speed as the forward pass - while derivative approximated using finite differences require 2 passes PER parameter, and run into numerous numerical issues with accuracy.
It's an absolute non-starter.
- azakai 10y agoYou don't actually need to calculate the derivative accurately. It's enough to sample several points near the current one and continue the search to the best of those. The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.
- nightski 10y agoExcept that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.