4 ms·

> Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different

> Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different things: in the one, they're "calculating" the exact, symbolic expression for the derivative (which can optionally be evaluated); in the other, they're "calculating" the floating-point approximation of the values of the derivative. If this is correct, then both Theano and TensorFlow are definitely doing symbolic differentiation, no? The `grad` function in Theano (and the equivalent in TF, not sure what it's called) takes a computation graph (a symbolic expression of a mathematical function) as input and produces a computational graph expressing the gradient (which as you said, can then optionally be evaluated at specified points). So maybe the author wasn't referring to these types of frameworks when he said there have recently been an explosion of these autodiff tools? Or are Theano and TF and the like, in fact, autodiff tools as opposed to "symdiff" tools, but I'm still missing the point somehow?

gh02t 11y ago
It's not correct. They are doing AD, not symbolic differentiation. Think of AD as evaluating the derivative step-by-step as you proceed through the program (or nodes in the expression graph in Theano's model). In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.
- throwaway39210 11y ago
  > In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in. This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into. edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top. edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100). [1] http://deeplearning.net/software/theano/library/gradient.html http://deeplearning.net/software/theano/library/gradient.htm... [2] import theano from theano import tensor as T def f(x): y = x for i in range(3): y = (x + y).sin() return y x = T.scalar(name='x') my_function = f(x) print "my_function =", theano.pprint(my_function) my_gradient = T.grad(my_function, x) print "my_gradient =", theano.pprint(my_gradient) -> my_function = sin((x + sin((x + sin((x + x)))))) my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]
  - abecedarius 11y ago
    Is that blowup perhaps an artifact of displaying the result as an expression, but where the result as an internal data structure is more compact, because of sharing? That's what I'd expect, although I haven't tried Theano yet. That is, I'd expect the sin((x + x)) which appears twice to be just one internal node which is referenced twice, and so on.
    - gh02t 11y ago
      That's my understanding, it does not compute the expression directly unless you ask it to. Could be wrong though, I've only used it a few times. AD does involve symbolic differentiation of each individual instruction, but instead of keeping the symbolic expression it is replaced with the numeric value, so the whole expression is never computed. It isn't magic and there are cases where it blows up. For instance, there are two basic ways to implement it, forward and backward propagation, each of which is good for a certain situation that the other is terrible at.
  - laughinghan 11y ago
    As pointed out by others, the blowup is an artifact of the visual representation, under-the-hood Theano does automatic differentiation (which they consider to be an optimization on symbolic differentiation, which is why they keep calling it that): > In this way Theano can be used for doing efficient symbolic differentiation (as the expression returned by T.grad will be optimized during compilation), even for function with many inputs. (see automatic differentiation for a description of symbolic differentiation). http://deeplearning.net/software/theano/tutorial/gradients.html http://deeplearning.net/software/theano/tutorial/gradients.h...

gh02t 11y ago

It's not correct. They are doing AD, not symbolic differentiation. Think of AD as evaluating the derivative step-by-step as you proceed through the program (or nodes in the expression graph in Theano's model). In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.

throwaway39210 11y ago
> In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in. This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into. edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top. edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100). [1] http://deeplearning.net/software/theano/library/gradient.html http://deeplearning.net/software/theano/library/gradient.htm... [2] import theano from theano import tensor as T def f(x): y = x for i in range(3): y = (x + y).sin() return y x = T.scalar(name='x') my_function = f(x) print "my_function =", theano.pprint(my_function) my_gradient = T.grad(my_function, x) print "my_gradient =", theano.pprint(my_gradient) -> my_function = sin((x + sin((x + sin((x + x)))))) my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]
- abecedarius 11y ago
  Is that blowup perhaps an artifact of displaying the result as an expression, but where the result as an internal data structure is more compact, because of sharing? That's what I'd expect, although I haven't tried Theano yet. That is, I'd expect the sin((x + x)) which appears twice to be just one internal node which is referenced twice, and so on.
  - gh02t 11y ago
    That's my understanding, it does not compute the expression directly unless you ask it to. Could be wrong though, I've only used it a few times. AD does involve symbolic differentiation of each individual instruction, but instead of keeping the symbolic expression it is replaced with the numeric value, so the whole expression is never computed. It isn't magic and there are cases where it blows up. For instance, there are two basic ways to implement it, forward and backward propagation, each of which is good for a certain situation that the other is terrible at.
- laughinghan 11y ago
  As pointed out by others, the blowup is an artifact of the visual representation, under-the-hood Theano does automatic differentiation (which they consider to be an optimization on symbolic differentiation, which is why they keep calling it that): > In this way Theano can be used for doing efficient symbolic differentiation (as the expression returned by T.grad will be optimized during compilation), even for function with many inputs. (see automatic differentiation for a description of symbolic differentiation). http://deeplearning.net/software/theano/tutorial/gradients.html http://deeplearning.net/software/theano/tutorial/gradients.h...