4 ms·
I thought automatic differentiation is a little different? Will have to look again. I thought the norm for a while was to manually enter the gradient, if poss
by taeric 4y ago
I thought automatic differentiation is a little different? Will have to look again. I thought the norm for a while was to manually enter the gradient, if possible, or to use a calculated one at each step.
- mirker 4y agoThe norm is to use library functions like addition, multiply, power, etc., which have hardcoded derivatives. Then the derivative is computed “symbolically” by composing these base derivatives, which is essentially an exercise of implementing calculator logic. I use scare quotes because there are numerical problems with some functions that have to be hacked around with more manual derivatives, like the cross-entropy/softmax function. In those cases, people suggest to use a different library function special-cased out for the situation, which is not really “symbolic” in the typical mathematical sense.