3 ms·

i wish people would just call autodiff (or at least describe it) as numerical chain rule, because that’s quite literally all it is (plus a few tricks for not ex

i wish people would just call autodiff (or at least describe it) as numerical chain rule, because that’s quite literally all it is (plus a few tricks for not explicitly computing the jacobian for certain operations), it’s a lot more clear that way

gipp 3y ago
More precise maybe, but I certainly wouldn't call that more clear.
- taminka 3y ago
  everyone studies chain rule in school, so it's definitely more clear, especially if you look at the amount of unnecessary complexity and convolutedness in many autodiff explanations (an substantial amount for such a simple subject)
evertedsphere 3y ago
The version described here and used most frequently in backpropagation implementations, "autodiff", is reverse-mode AD, but forward-mode exists, as does a spectrum of strategies between the two extremes. Sure, it all comes down to the chain rule, but at an algorithmic level the choice is not at all a trivial one. In fact, if asked to use the chain rule to propagate gradients through a computation graph, I suspect most people would intuitively default to the forward mode. (I would!) https://en.wikipedia.org/wiki/Automatic_differentiation#Beyond_forward_and_reverse_accumulation https://en.wikipedia.org/wiki/Automatic_differentiation#Beyo... Given this, it seems useful to use the term to denote a particular method of accumulating the gradients as one traverses the expressions provided by the chain rule.
sdenton4 3y ago
Technically incorrect! The numerical chain rule uses the method of finite difference, which will accumulate errors as you move through the computation. See the 'differences from other methods' section: https://en.m.wikipedia.org/wiki/Automatic_differentiation https://en.m.wikipedia.org/wiki/Automatic_differentiation The point, as the neighborhood comment says, is that the implementation really matters, and is worthy of study. It's fine to say that autodiff is a family of methods for implementing the chain rule, but incorrect to say that it's 'just' the numerical chain rule.
- taminka 3y ago
  by "numerical" i meant "operating on numbers as opposed to symbolic values", not "using the method of finite differences", apologies for the confusion

gipp 3y ago

More precise maybe, but I certainly wouldn't call that more clear.

taminka 3y ago
everyone studies chain rule in school, so it's definitely more clear, especially if you look at the amount of unnecessary complexity and convolutedness in many autodiff explanations (an substantial amount for such a simple subject)

evertedsphere 3y ago

The version described here and used most frequently in backpropagation implementations, "autodiff", is reverse-mode AD, but forward-mode exists, as does a spectrum of strategies between the two extremes. Sure, it all comes down to the chain rule, but at an algorithmic level the choice is not at all a trivial one. In fact, if asked to use the chain rule to propagate gradients through a computation graph, I suspect most people would intuitively default to the forward mode. (I would!) https://en.wikipedia.org/wiki/Automatic_differentiation#Beyond_forward_and_reverse_accumulation https://en.wikipedia.org/wiki/Automatic_differentiation#Beyo... Given this, it seems useful to use the term to denote a particular method of accumulating the gradients as one traverses the expressions provided by the chain rule.

sdenton4 3y ago

Technically incorrect! The numerical chain rule uses the method of finite difference, which will accumulate errors as you move through the computation. See the 'differences from other methods' section: https://en.m.wikipedia.org/wiki/Automatic_differentiation https://en.m.wikipedia.org/wiki/Automatic_differentiation The point, as the neighborhood comment says, is that the implementation really matters, and is worthy of study. It's fine to say that autodiff is a family of methods for implementing the chain rule, but incorrect to say that it's 'just' the numerical chain rule.

taminka 3y ago
by "numerical" i meant "operating on numbers as opposed to symbolic values", not "using the method of finite differences", apologies for the confusion