3 ms·
One of the innovations in the paper is they find a method for implementing reverse-mode autodiff in a stateless manner, making optimisations easier to verify.
by ogogmad 5y ago
One of the innovations in the paper is they find a method for implementing reverse-mode autodiff in a stateless manner, making optimisations easier to verify.
I might be wrong but they take the principle of dual numbers: a + b epsilon, and change the b component into its linear dual. Therefore, b is a function, and not a number. If b is a function instead of a number, then there's more flexibility regarding flow of control, and that flexibility allows you to express reverse-mode autodiff.
- brrrrrm 5y agoa pure functional language is trivially differentiated, no? View a program as a data flow graph and then append the differentiation data flow graph. How this is executed (reverse-mode vs forward-mode vs some exotic mix) is just a leaked abstraction. Ideally, it wouldn't need to be exposed to the user.
- taeric 5y agoMy understanding is that all trivial versions of anything are wonderful for the pedagogy, but far short of what is needed in implementation. Just consider garbage collection. Trivial reference counting is easy to explain and implement. Not going to win too many points in use, though.