4 ms·

> I think if you made up a DAG-adapted symbolic differentiator That's essentially what autodiff is. It works on the DAG, whereas "symbolic" diff insists that t

by klipt 11y ago

> I think if you made up a DAG-adapted symbolic differentiator That's essentially what autodiff is. It works on the DAG, whereas "symbolic" diff insists that the DAG is a tree (so reused components have to be duplicated, which leads to blowup).

abecedarius 11y ago
Yes, that's one way to look at it. However, here's a Python symbolic differentiator I just coded now to make a more direct comparison, since Norvig's Lisp is very different from my Python in lots of distracting ways: https://github.com/darius/sketchbook/blob/master/misc/symbdiff.py https://github.com/darius/sketchbook/blob/master/misc/symbdi... This symbolic differentiator could be made to not blow up, by memoizing, i.e. decorating with https://github.com/darius/sketchbook/blob/master/regex/memo.py https://github.com/darius/sketchbook/blob/master/regex/memo.... . It'd still have the difference that it constructs an explicit formula in x, as a data structure; the autodiff code instead takes a particular x as input and calculates a value for just that x, where the value includes both an 'actual value' and a 'derivative' component. (Your library and compiler could get fancy and inline that computation at compile time back into a pair of formulas, and then yes, they ought to be exactly equivalent.)
- throwaway39210 11y ago
  Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).

by klipt 11y ago

abecedarius 11y ago

Yes, that's one way to look at it. However, here's a Python symbolic differentiator I just coded now to make a more direct comparison, since Norvig's Lisp is very different from my Python in lots of distracting ways: https://github.com/darius/sketchbook/blob/master/misc/symbdiff.py https://github.com/darius/sketchbook/blob/master/misc/symbdi... This symbolic differentiator could be made to not blow up, by memoizing, i.e. decorating with https://github.com/darius/sketchbook/blob/master/regex/memo.py https://github.com/darius/sketchbook/blob/master/regex/memo.... . It'd still have the difference that it constructs an explicit formula in x, as a data structure; the autodiff code instead takes a particular x as input and calculates a value for just that x, where the value includes both an 'actual value' and a 'derivative' component. (Your library and compiler could get fancy and inline that computation at compile time back into a pair of formulas, and then yes, they ought to be exactly equivalent.)

throwaway39210 11y ago
Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).