4 ms·
Thid is really neat. Is this really used in practice? It seems to me that most of the AD frameworks used for deep learning implement the backward function tha
by elcomet 8y ago
Thid is really neat.
Is this really used in practice?
It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions
- cultus 8y agoI used it for non ML tasks in geophysics, which made my life a lot easier. However, I think most scientists and engineers aren't aware of it. It has been described as "criminally underused."
- IngoBlechschmid 8y agoYes, definitely, there are even battle-tested implementations for Fortran available. Though I have never seen AD frameworks used in the production contexts for neural networks/backpropagation. As you say, the code for this seems to be mostly handrolled. Please take this negative statement with a grain of salt, I don't actually work in machine learning.
- kxyvr 8y agoBackpropagation is literally the reverse mode AD applied to neural nets. That said, I wish had a paper that I could point to that shows explicitly that. From what I can tell, the AD community has known this basically since the start, but for whatever reason backpropogation is taught rather than the more general technique.
- aaaaaaaaaab 8y agoTo calculate the gradient of a function R^n -> R^m, forward-mode AD is preferable if n << m, while reverse-mode is faster for n >> m. From this it should be clear why machine learning uses reverse-mode. On the other hand, forward-mode is better for e.g. calculating the tangent of a high-dimensional curve (i.e. R -> R^n).