3 ms·

> That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Finding a derivative by

> That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps. Finding a derivative by hand gets tiring fast, and is error prone. Used to be a neural net paper doing anything novel spent like 1-2 pages deriving the derivative of its cool new thing. Not it's derivative isn't even mentioned.

taeric 6y ago
But does that require auto derivative stuff? I'm assuming most could have simply mentioned the main equation, said "use Mathematica" and proceeded with the derivative?
- Mehdi2277 6y ago
  I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determines the derivative for the whole function. AD focuses on running a function and including information to be able to find the derivative at one value. That value may involve tons of branching along the way. You could have a program that involves loops/conditionals that symbolic differentiation would have a painful time and unlikely to pull it off while AD works just fine. AD does not try to determine what the derivative looks everywhere but really traces the function with some extra stuff (depends on forward/backward for what stuff) to allow it to know what the derivative is. An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.
  - taeric 6y ago
    Ok, this I think finally clicks some to me. To try and rephrase, the techniques used are similar in both. In that it is all calculus. However, with AD, the focus is reducing all calculations down to what was performed in expressions that have duals, and getting the derivative of that, as calculated, to use in making a choice. Doing this symbolically would require the entire function space be solved for the problem in very difficult ways that are not easy to avoid. That is, with AD, you don't get the total derivative of a problem, per se. Instead, you get the problem reduced to a a method that can evaluate at a tuple, where you also have the derivative for that tuple? Extrapolating from this, even if a problem had piecewise spots where it will not differentiate, AD mostly works around this by focusing on the pieces?
- oxinabox 6y ago
  The main equation may not really be anywhere near so simple as that. E.g. for the kind of stuff I do for work the main equation is something like output a vector which is the instructions for every powerplant in Texas of how much power to produce, such that every person and business that needs power has power, subject to constraints about the capacity of each powerline Differentiate total emissions of this process with respect to demand. It's is an equation yes. But it's actually a multi-hundred line program that includes thousands more lines of constants and a step of running ab internal linear optimization solver inside the problem. Symbolic Differentiation, like Mathematica tends to start to slow down and generate massively amounts of code once functions get complicated. I am also not sure that it can handle dynamic length loops. Source to Source AD, like Zygote (Julia), Tapenade (Fortran), Jax (Python), Enzyme (LLVM) have a lot of what you might want though.
  - taeric 6y ago
    Thanks to both answers here! I think this is finally clicking. Probably many more ticks before I get it, but such is progress. Would love to see a blog or other post on solving this kind of problem. One with an inner LP would be amazing to see. I'm pretty sure some of the stuff I've looked at in the past on supply chain solutions could find relevance here.
- snicker7 6y ago
  Symbolic differentiation (e.g. "use Mathematica") is possible but is generally slow to generate and produces incredibly slow/inefficient code.
  - taeric 6y ago
    I think the part I had trouble with was seeing why AD would give better results. I still don't fully grok it, but that AD works not necessarily on the full equation, but the equation that was evaluated at a spot, I think finally clicks some with me. I don't get how it helps with some discontinuity problems, but I think I can get how those aren't as important in many contexts.

taeric 6y ago

But does that require auto derivative stuff? I'm assuming most could have simply mentioned the main equation, said "use Mathematica" and proceeded with the derivative?

Mehdi2277 6y ago
I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determines the derivative for the whole function. AD focuses on running a function and including information to be able to find the derivative at one value. That value may involve tons of branching along the way. You could have a program that involves loops/conditionals that symbolic differentiation would have a painful time and unlikely to pull it off while AD works just fine. AD does not try to determine what the derivative looks everywhere but really traces the function with some extra stuff (depends on forward/backward for what stuff) to allow it to know what the derivative is. An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.
- taeric 6y ago
  Ok, this I think finally clicks some to me. To try and rephrase, the techniques used are similar in both. In that it is all calculus. However, with AD, the focus is reducing all calculations down to what was performed in expressions that have duals, and getting the derivative of that, as calculated, to use in making a choice. Doing this symbolically would require the entire function space be solved for the problem in very difficult ways that are not easy to avoid. That is, with AD, you don't get the total derivative of a problem, per se. Instead, you get the problem reduced to a a method that can evaluate at a tuple, where you also have the derivative for that tuple? Extrapolating from this, even if a problem had piecewise spots where it will not differentiate, AD mostly works around this by focusing on the pieces?
oxinabox 6y ago
The main equation may not really be anywhere near so simple as that. E.g. for the kind of stuff I do for work the main equation is something like output a vector which is the instructions for every powerplant in Texas of how much power to produce, such that every person and business that needs power has power, subject to constraints about the capacity of each powerline Differentiate total emissions of this process with respect to demand. It's is an equation yes. But it's actually a multi-hundred line program that includes thousands more lines of constants and a step of running ab internal linear optimization solver inside the problem. Symbolic Differentiation, like Mathematica tends to start to slow down and generate massively amounts of code once functions get complicated. I am also not sure that it can handle dynamic length loops. Source to Source AD, like Zygote (Julia), Tapenade (Fortran), Jax (Python), Enzyme (LLVM) have a lot of what you might want though.
- taeric 6y ago
  Thanks to both answers here! I think this is finally clicking. Probably many more ticks before I get it, but such is progress. Would love to see a blog or other post on solving this kind of problem. One with an inner LP would be amazing to see. I'm pretty sure some of the stuff I've looked at in the past on supply chain solutions could find relevance here.
snicker7 6y ago
Symbolic differentiation (e.g. "use Mathematica") is possible but is generally slow to generate and produces incredibly slow/inefficient code.
- taeric 6y ago
  I think the part I had trouble with was seeing why AD would give better results. I still don't fully grok it, but that AD works not necessarily on the full equation, but the equation that was evaluated at a spot, I think finally clicks some with me. I don't get how it helps with some discontinuity problems, but I think I can get how those aren't as important in many contexts.