3 ms·

Here's some code I just hacked up to show the idea: https://github.com/darius/sketchbook/blob/master/misc/autodiff.py https://github.com/darius/sketchbook/blob/

Here's some code I just hacked up to show the idea: https://github.com/darius/sketchbook/blob/master/misc/autodiff.py https://github.com/darius/sketchbook/blob/master/misc/autodi... At the bottom it computes both the value and the derivative of x^3 at x=2. x^3 is 8 and 3x^2 is 12. For comparison, symbolic differentiation is implemented halfway down http://norvig.com/paip/macsymar.lisp http://norvig.com/paip/macsymar.lisp where there are the rules you learn in calculus class, lines like (d (u + v) / d x = (d u / d x) + (d v / d x)) which means code that looks at the formula as source code and rewrites it. It blows up in the example the OP brings up, because it's working on an expression instead of on a DAG; I think if you made up a DAG-adapted symbolic differentiator it could output a compact sequence of formulas for the derivative, though I haven't tried it. It should be equivalent to symbolically evaluating the code from autodiff.py.

klipt 11y ago
> I think if you made up a DAG-adapted symbolic differentiator That's essentially what autodiff is. It works on the DAG, whereas "symbolic" diff insists that the DAG is a tree (so reused components have to be duplicated, which leads to blowup).
- abecedarius 11y ago
  Yes, that's one way to look at it. However, here's a Python symbolic differentiator I just coded now to make a more direct comparison, since Norvig's Lisp is very different from my Python in lots of distracting ways: https://github.com/darius/sketchbook/blob/master/misc/symbdiff.py https://github.com/darius/sketchbook/blob/master/misc/symbdi... This symbolic differentiator could be made to not blow up, by memoizing, i.e. decorating with https://github.com/darius/sketchbook/blob/master/regex/memo.py https://github.com/darius/sketchbook/blob/master/regex/memo.... . It'd still have the difference that it constructs an explicit formula in x, as a data structure; the autodiff code instead takes a particular x as input and calculates a value for just that x, where the value includes both an 'actual value' and a 'derivative' component. (Your library and compiler could get fancy and inline that computation at compile time back into a pair of formulas, and then yes, they ought to be exactly equivalent.)
  - throwaway39210 11y ago
    Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).
miltondts 11y ago
If this code is correct(and it appears to be), then it show to me that a small portion of code explains this idea much better than the dozens of explanations I found on the internet. The frustrating part is reading lots of words/math formulas explanations without understanding them and then seeing some diagram/executable code and thinking "that's it? so many hours wasted try to understand something so simple.. why didn't you draw a diagram? why didn't you show some executable code? WHY?!"
- abecedarius 11y ago
  Thanks for commenting -- I've had that feeling too, of frustration on struggling with some wodge of formulas bedecked with indices and weighted with implicit assumptions of unclear scope. Modern programming languages can usually express these things about as concisely, with the great advantage that you can engage with the code and it'll push back. You wonder "why isn't it this way?" and just try it.

klipt 11y ago

> I think if you made up a DAG-adapted symbolic differentiator That's essentially what autodiff is. It works on the DAG, whereas "symbolic" diff insists that the DAG is a tree (so reused components have to be duplicated, which leads to blowup).

abecedarius 11y ago
Yes, that's one way to look at it. However, here's a Python symbolic differentiator I just coded now to make a more direct comparison, since Norvig's Lisp is very different from my Python in lots of distracting ways: https://github.com/darius/sketchbook/blob/master/misc/symbdiff.py https://github.com/darius/sketchbook/blob/master/misc/symbdi... This symbolic differentiator could be made to not blow up, by memoizing, i.e. decorating with https://github.com/darius/sketchbook/blob/master/regex/memo.py https://github.com/darius/sketchbook/blob/master/regex/memo.... . It'd still have the difference that it constructs an explicit formula in x, as a data structure; the autodiff code instead takes a particular x as input and calculates a value for just that x, where the value includes both an 'actual value' and a 'derivative' component. (Your library and compiler could get fancy and inline that computation at compile time back into a pair of formulas, and then yes, they ought to be exactly equivalent.)
- throwaway39210 11y ago
  Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).

miltondts 11y ago

If this code is correct(and it appears to be), then it show to me that a small portion of code explains this idea much better than the dozens of explanations I found on the internet. The frustrating part is reading lots of words/math formulas explanations without understanding them and then seeing some diagram/executable code and thinking "that's it? so many hours wasted try to understand something so simple.. why didn't you draw a diagram? why didn't you show some executable code? WHY?!"

abecedarius 11y ago
Thanks for commenting -- I've had that feeling too, of frustration on struggling with some wodge of formulas bedecked with indices and weighted with implicit assumptions of unclear scope. Modern programming languages can usually express these things about as concisely, with the great advantage that you can engage with the code and it'll push back. You wonder "why isn't it this way?" and just try it.