3 ms·

I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determine

I don't think mathematica supports AD. It supports symbolic differentiation which is similar, but different. Symbolic differentiation given a function determines the derivative for the whole function. AD focuses on running a function and including information to be able to find the derivative at one value. That value may involve tons of branching along the way. You could have a program that involves loops/conditionals that symbolic differentiation would have a painful time and unlikely to pull it off while AD works just fine. AD does not try to determine what the derivative looks everywhere but really traces the function with some extra stuff (depends on forward/backward for what stuff) to allow it to know what the derivative is. An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.

taeric 6y ago
Ok, this I think finally clicks some to me. To try and rephrase, the techniques used are similar in both. In that it is all calculus. However, with AD, the focus is reducing all calculations down to what was performed in expressions that have duals, and getting the derivative of that, as calculated, to use in making a choice. Doing this symbolically would require the entire function space be solved for the problem in very difficult ways that are not easy to avoid. That is, with AD, you don't get the total derivative of a problem, per se. Instead, you get the problem reduced to a a method that can evaluate at a tuple, where you also have the derivative for that tuple? Extrapolating from this, even if a problem had piecewise spots where it will not differentiate, AD mostly works around this by focusing on the pieces?

taeric 6y ago

Ok, this I think finally clicks some to me. To try and rephrase, the techniques used are similar in both. In that it is all calculus. However, with AD, the focus is reducing all calculations down to what was performed in expressions that have duals, and getting the derivative of that, as calculated, to use in making a choice. Doing this symbolically would require the entire function space be solved for the problem in very difficult ways that are not easy to avoid. That is, with AD, you don't get the total derivative of a problem, per se. Instead, you get the problem reduced to a a method that can evaluate at a tuple, where you also have the derivative for that tuple? Extrapolating from this, even if a problem had piecewise spots where it will not differentiate, AD mostly works around this by focusing on the pieces?