6 ms·

To the best of my understanding: - with numerical differentiation, you take an algorithm to approximate a function, you repeatedly evaluate that, and you use t

To the best of my understanding: - with numerical differentiation, you take an algorithm to approximate a function, you repeatedly evaluate that, and you use the results to approximate the derivative - with symbolic differentiation, you take a symbolic, exact representation of a function (which you could evaluate with floating-point numbers to approximate its values if you wanted to use numerical differentiation instead), and you apply rules like linearity and the product rule to get a symbolic, exact representation of the derivative, which you evaluate with floating-point numbers to approximate the derivative - with automatic differentiation, you take an algorithm to approximate a function, which must be implemented using basic floating-point operations, and you apply rules like linearity and the product rule to the operations used to implement the approximation, and you get an algorithm that approximates the function's derivative Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different things: in the one, they're "calculating" the exact, symbolic expression for the derivative (which can optionally be evaluated); in the other, they're "calculating" the floating-point approximation of the values of the derivative. To me, automatic differentiation is GENIUS. My friend, who's written a Julia package [1] based on some of the ideas behind automatic differentiation, introduced me to them; unfortunately, according to him, numerical differentiation is usually more practical. I don't remember why but I'm going to ask him. [1]: https://github.com/jwmerrill/PowerSeries.jl https://github.com/jwmerrill/PowerSeries.jl

throwaway39210 11y ago
> Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different things: in the one, they're "calculating" the exact, symbolic expression for the derivative (which can optionally be evaluated); in the other, they're "calculating" the floating-point approximation of the values of the derivative. If this is correct, then both Theano and TensorFlow are definitely doing symbolic differentiation, no? The `grad` function in Theano (and the equivalent in TF, not sure what it's called) takes a computation graph (a symbolic expression of a mathematical function) as input and produces a computational graph expressing the gradient (which as you said, can then optionally be evaluated at specified points). So maybe the author wasn't referring to these types of frameworks when he said there have recently been an explosion of these autodiff tools? Or are Theano and TF and the like, in fact, autodiff tools as opposed to "symdiff" tools, but I'm still missing the point somehow?
- gh02t 11y ago
  It's not correct. They are doing AD, not symbolic differentiation. Think of AD as evaluating the derivative step-by-step as you proceed through the program (or nodes in the expression graph in Theano's model). In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.
  - throwaway39210 11y ago
    > In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in. This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into. edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top. edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100). [1] http://deeplearning.net/software/theano/library/gradient.html http://deeplearning.net/software/theano/library/gradient.htm... [2] import theano from theano import tensor as T def f(x): y = x for i in range(3): y = (x + y).sin() return y x = T.scalar(name='x') my_function = f(x) print "my_function =", theano.pprint(my_function) my_gradient = T.grad(my_function, x) print "my_gradient =", theano.pprint(my_gradient) -> my_function = sin((x + sin((x + sin((x + x)))))) my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]
gh02t 11y ago
You can think of AD as differentiating a function that is defined in terms of composition... e.g. f(g(h(x))), where (loosely) x is the initial state of your function and f, g, and h are differentiable mappings from one state to another. In this analogy, in a procedural language f, g, and h would be lines in your source code, which modify the overall program state to produce some final value. There are a few different ways to actually do it (including the rather tricky concept of "dual numbers"), but ultimately it amounts to differentiating the overall program f(g(h(x)) recursively using the chain rule. It's [often] more efficient than direct symbolic differentiation because of this recursive evaluation - think of it as something like tail-call optimization. It's not an approximate method, nor does it care [directly] about floating point. If you have implemented some f' that approximates a mathematical function f and you use AD to differentiate f', you are calculating the exact derivative of f', not an approximation of the derivative of f. It's an important distinction.
- laughinghan 11y ago
  I found dual numbers pretty intuitive, but I was a math major and studied abstract algebra and stuff, that way of thinking is definitely an acquired taste. To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? I don't really see why the distinction is particularly important. The reason the exact derivative of f' that you get from AD is useful is because it also approximates the derivative of f, right? If it were possible to make some pathological choice of f' (that is, algorithmic implementation of f) such that the result from AD is a poor approximation of the derivative of f, that's not really a successful use of AD on f, right? So when I say I'm gonna "use AD to get a derivative of f", I'm really saying I'm going to use AD on a good choice of an approximation of f to get a good approximation of the derivative of f, right? I suppose if infinite precision analog computing were possible, or in the context of theoretical real-valued register machines (does anyone still study that?), it matters that AD is exact on the algorithmic implementation, but we're not in or anywhere near those contexts, are we?
  - gh02t 11y ago

throwaway39210 11y ago

> Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different things: in the one, they're "calculating" the exact, symbolic expression for the derivative (which can optionally be evaluated); in the other, they're "calculating" the floating-point approximation of the values of the derivative. If this is correct, then both Theano and TensorFlow are definitely doing symbolic differentiation, no? The `grad` function in Theano (and the equivalent in TF, not sure what it's called) takes a computation graph (a symbolic expression of a mathematical function) as input and produces a computational graph expressing the gradient (which as you said, can then optionally be evaluated at specified points). So maybe the author wasn't referring to these types of frameworks when he said there have recently been an explosion of these autodiff tools? Or are Theano and TF and the like, in fact, autodiff tools as opposed to "symdiff" tools, but I'm still missing the point somehow?

gh02t 11y ago
It's not correct. They are doing AD, not symbolic differentiation. Think of AD as evaluating the derivative step-by-step as you proceed through the program (or nodes in the expression graph in Theano's model). In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.
- throwaway39210 11y ago
  > In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in. This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into. edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top. edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100). [1] http://deeplearning.net/software/theano/library/gradient.html http://deeplearning.net/software/theano/library/gradient.htm... [2] import theano from theano import tensor as T def f(x): y = x for i in range(3): y = (x + y).sin() return y x = T.scalar(name='x') my_function = f(x) print "my_function =", theano.pprint(my_function) my_gradient = T.grad(my_function, x) print "my_gradient =", theano.pprint(my_gradient) -> my_function = sin((x + sin((x + sin((x + x)))))) my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]

gh02t 11y ago

You can think of AD as differentiating a function that is defined in terms of composition... e.g. f(g(h(x))), where (loosely) x is the initial state of your function and f, g, and h are differentiable mappings from one state to another. In this analogy, in a procedural language f, g, and h would be lines in your source code, which modify the overall program state to produce some final value. There are a few different ways to actually do it (including the rather tricky concept of "dual numbers"), but ultimately it amounts to differentiating the overall program f(g(h(x)) recursively using the chain rule. It's [often] more efficient than direct symbolic differentiation because of this recursive evaluation - think of it as something like tail-call optimization. It's not an approximate method, nor does it care [directly] about floating point. If you have implemented some f' that approximates a mathematical function f and you use AD to differentiate f', you are calculating the exact derivative of f', not an approximation of the derivative of f. It's an important distinction.

laughinghan 11y ago
I found dual numbers pretty intuitive, but I was a math major and studied abstract algebra and stuff, that way of thinking is definitely an acquired taste. To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? I don't really see why the distinction is particularly important. The reason the exact derivative of f' that you get from AD is useful is because it also approximates the derivative of f, right? If it were possible to make some pathological choice of f' (that is, algorithmic implementation of f) such that the result from AD is a poor approximation of the derivative of f, that's not really a successful use of AD on f, right? So when I say I'm gonna "use AD to get a derivative of f", I'm really saying I'm going to use AD on a good choice of an approximation of f to get a good approximation of the derivative of f, right? I suppose if infinite precision analog computing were possible, or in the context of theoretical real-valued register machines (does anyone still study that?), it matters that AD is exact on the algorithmic implementation, but we're not in or anywhere near those contexts, are we?
- gh02t 11y ago