4 ms·
Nice introduction to backpropagation. The author understandably handles only a special case and avoids the case when an input comes in on many completely differ
by nadam 12y ago
Nice introduction to backpropagation. The author understandably handles only a special case and avoids the case when an input comes in on many completely different paths to the output. When this article will be extended, it would be interesting to read about the case when it is not that way. (e.g. convnets with parameter sharing).
- njohnson41 12y agoI also like how the backpropagation section starts out by immediately talking about how it is really just chain rule application. The backwards-moving pattern of "backpropagation" is really just a side-effect of the derivative chain rule application order, but a lot of intro materials treat backprop as if it is some fancy thing specially-designed for neural nets. I suppose "compute the gradient of this function using basic vector calculus" just isn't sexy enough. I complain mostly because it took me a while to figure out whether backprop was exactly the same as gradient descent, or if there were subtle differences.
- cafebeen 12y agoAnother interesting point--that chain-rule gradient evaluation is essentially something called automatic differentiation: https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation which is really cool stuff and should be included more often when talking about backprop.