3 ms·
You can also say backpropagation is the chain rule from centuries ago.
by 29athrowaway 2y ago
You can also say backpropagation is the chain rule from centuries ago.
- elijahbenizzy 2y agoBackpropogration is just an application of the chain rule -- cool that we all learned it in high school!
- telotortium 2y agoIt is a computationally clever application of the chain rule to minimize the amount of computation needed to compute gradients for all parameters in the network.
- om8 2y ago> to minimize the amount of computation IMO backprop is the most trivial implementation of differentiation in neural networks. Do you know an easier way to compute gradients with larger overhead? If so, please share it.
- QuadmasterXLII 2y agoMy first forays into making neural networks used replacement rules to modify an expression tree until all the “D” operators went away, but that takes exponential complexity in network depth if you aren’t careful. Finite differences is linear in number of parameters, as is differentiation by Dual Numbers
- eli_gottlieb 2y agoBackprop is the application of dynamic programming to the chain rule for total derivatives, which sounds trivial only in retrospect.
- Jensson 2y agoYou can do forward propagation. Humans typically finds forward easier than backwards.
- tripzilch 2y agosince you asked ... how about Monte Carlo with Gibbs sampling?
- pmayrgundter 2y agoI totally didn't realize this until these comments. Neat! I went digging in wikipedia.. the Backpropagation article was created in 2005 and yet the mention of association/derivation from the chain rule wasn't mentioned until 2014, through a borrow from the German article https://en.wikipedia.org/w/index.php?title=Backpropagation&oldid=618037812 https://en.wikipedia.org/w/index.php?title=Backpropagation&o...