5 ms·

I'm seeing a lot of discussions around Automatic Differentiation, but I don't understand the purpose. Why would you take the derivative of a piece of code?

srl 6y ago
The most frequent application is machine learning. If my piece of code is implementing some function that I want to minimize, then taking the derivative (w.r.t. some vector of parameters) tells me in which direction I need to change the parameters in order to reduce the function: "gradient descent".
- dnautics 6y ago
  Note that the example here is forward differentiation and most machine learning these days is backward propagation
  - oxinabox 6y ago
    Note that this applies backwards also. Of the 7 ADs demoed, only 1 was forward. The other 6 were reverse mode. All gave same result
    - dnautics 6y ago
      Of course, sorry didn't mean to imply that AD was useful only in one direction. Haha I guess I was caught only reading 1/7 of the article!!
  - rockinghigh 6y ago
    Backward propagation computes the gradient of the loss function with respect to the weights. A gradient is a vector or tensor of derivatives. You need to differentiate the loss function somehow to evaluate this vector.
    - dnautics 6y ago
      That's not strictly speaking true. You also calculate the gradient of the input. Otherwise, you wouldn't be able to backpropagate over more than one layer. It's just useful because you can do both easily, where with forward mode getting derivatives with respect to weights is considerably more expensive.
- jpollock 6y ago
  So, we've got "reality", as represented by data. We've got a model, implemented in code. Since it's code can be differentiated - not sure how that works with branches, I guess that's the math. :) This is generated through a set of input parameters. We've got an error function, representing the difference between the model and reality. If we differentiate the error function, we can choose which set of parameter mutations are heading in the right direction to then generate a new model? We check each close point and find the max benefit? However, if everything is taking the parameters as input, is the derivative of the error function only generated once? Is it saying that the derivative of the error function is independent of the parameters, so it doesn't matter what the model is, they all have the same error function, and that error function can be found by generating a single model?
  - eigenspace 6y ago
    Dealing with branches is indeed and interesting problem. Many AD systems can't accomodate branches. I think most of the Julia ones do. The gist of it is that branches don't actually require that much fanciness, but they can introduce discontinuties, and AD systems will often do things like happily give you a finite derivative right at a discontinuity where a calculus student would normally complain.
jmeyer2k 6y ago
To expand, it tells you how you can change certain variables of a function to change the output of a function. In machine learning, you create a loss function that takes in modal parameters and returns how accurate the model is. Then, you can optimize the function inputs for accuracy. Automatic differentiation makes it much easier to code different types of models.
sesuximo 6y ago
It’s a tool to help solve inverse problems.
freemint 6y ago
I have used Automatic Differention used extensively to "optimize" simulations. This is problem is know under many names such parameter estimation, local sensitivity analysis, optimal control ... Having access to something the resembles the derivative of your code allows you to use optimization algorithms that converge faster (Newton or Gradient descent vs bisection methods) and you don't have to specify the derivatives by hand (which gets really bothersome when you have to specify a hessian matrix (2nd derivative information including all the mixed terms)).
- taeric 6y ago
  Would be cool to see what your code looked like. I have the same question as the start of the thread. That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Edit: note that I also see a difference in being able to get the derivative of a function, versus writing in a sense where any function can be moved to its derivative.
  - oxinabox 6y ago
    > That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Finding a derivative by hand gets tiring fast, and is error prone. Used to be a neural net paper doing anything novel spent like 1-2 pages deriving the derivative of its cool new thing. Not it's derivative isn't even mentioned.
    - taeric 6y ago
      But does that require auto derivative stuff? I'm assuming most could have simply mentioned the main equation, said "use Mathematica" and proceeded with the derivative?
      - Mehdi2277 6y ago
        I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determines the derivative for the whole function. AD focuses on running a function and including information to be able to find the derivative at one value. That value may involve tons of branching along the way. You could have a program that involves loops/conditionals that symbolic differentiation would have a painful time and unlikely to pull it off while AD works just fine. AD does not try to determine what the derivative looks everywhere but really traces the function with some extra stuff (depends on forward/backward for what stuff) to allow it to know what the derivative is. An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.
eigenspace 6y ago
You want to take the derivative of a piece of code so that you don't have to either hand-write the derivative yourself, or rely on finite differencing which can be slower in many circumstances and is almost universally less accurate.

srl 6y ago

The most frequent application is machine learning. If my piece of code is implementing some function that I want to minimize, then taking the derivative (w.r.t. some vector of parameters) tells me in which direction I need to change the parameters in order to reduce the function: "gradient descent".

dnautics 6y ago
Note that the example here is forward differentiation and most machine learning these days is backward propagation
- oxinabox 6y ago
  Note that this applies backwards also. Of the 7 ADs demoed, only 1 was forward. The other 6 were reverse mode. All gave same result
  - dnautics 6y ago
    Of course, sorry didn't mean to imply that AD was useful only in one direction. Haha I guess I was caught only reading 1/7 of the article!!
- rockinghigh 6y ago
  Backward propagation computes the gradient of the loss function with respect to the weights. A gradient is a vector or tensor of derivatives. You need to differentiate the loss function somehow to evaluate this vector.
  - dnautics 6y ago
    That's not strictly speaking true. You also calculate the gradient of the input. Otherwise, you wouldn't be able to backpropagate over more than one layer. It's just useful because you can do both easily, where with forward mode getting derivatives with respect to weights is considerably more expensive.
jpollock 6y ago
So, we've got "reality", as represented by data. We've got a model, implemented in code. Since it's code can be differentiated - not sure how that works with branches, I guess that's the math. :) This is generated through a set of input parameters. We've got an error function, representing the difference between the model and reality. If we differentiate the error function, we can choose which set of parameter mutations are heading in the right direction to then generate a new model? We check each close point and find the max benefit? However, if everything is taking the parameters as input, is the derivative of the error function only generated once? Is it saying that the derivative of the error function is independent of the parameters, so it doesn't matter what the model is, they all have the same error function, and that error function can be found by generating a single model?
- eigenspace 6y ago
  Dealing with branches is indeed and interesting problem. Many AD systems can't accomodate branches. I think most of the Julia ones do. The gist of it is that branches don't actually require that much fanciness, but they can introduce discontinuties, and AD systems will often do things like happily give you a finite derivative right at a discontinuity where a calculus student would normally complain.

jmeyer2k 6y ago

To expand, it tells you how you can change certain variables of a function to change the output of a function. In machine learning, you create a loss function that takes in modal parameters and returns how accurate the model is. Then, you can optimize the function inputs for accuracy. Automatic differentiation makes it much easier to code different types of models.

sesuximo 6y ago

It’s a tool to help solve inverse problems.

freemint 6y ago

I have used Automatic Differention used extensively to "optimize" simulations. This is problem is know under many names such parameter estimation, local sensitivity analysis, optimal control ... Having access to something the resembles the derivative of your code allows you to use optimization algorithms that converge faster (Newton or Gradient descent vs bisection methods) and you don't have to specify the derivatives by hand (which gets really bothersome when you have to specify a hessian matrix (2nd derivative information including all the mixed terms)).

taeric 6y ago
Would be cool to see what your code looked like. I have the same question as the start of the thread. That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Edit: note that I also see a difference in being able to get the derivative of a function, versus writing in a sense where any function can be moved to its derivative.
- oxinabox 6y ago
  > That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Finding a derivative by hand gets tiring fast, and is error prone. Used to be a neural net paper doing anything novel spent like 1-2 pages deriving the derivative of its cool new thing. Not it's derivative isn't even mentioned.
  - taeric 6y ago
    But does that require auto derivative stuff? I'm assuming most could have simply mentioned the main equation, said "use Mathematica" and proceeded with the derivative?
    - Mehdi2277 6y ago
      I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determines the derivative for the whole function. AD focuses on running a function and including information to be able to find the derivative at one value. That value may involve tons of branching along the way. You could have a program that involves loops/conditionals that symbolic differentiation would have a painful time and unlikely to pull it off while AD works just fine. AD does not try to determine what the derivative looks everywhere but really traces the function with some extra stuff (depends on forward/backward for what stuff) to allow it to know what the derivative is. An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.

eigenspace 6y ago

You want to take the derivative of a piece of code so that you don't have to either hand-write the derivative yourself, or rely on finite differencing which can be slower in many circumstances and is almost universally less accurate.