5 ms·

I found dual numbers pretty intuitive, but I was a math major and studied abstract algebra and stuff, that way of thinking is definitely an acquired taste. To

I found dual numbers pretty intuitive, but I was a math major and studied abstract algebra and stuff, that way of thinking is definitely an acquired taste. To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? I don't really see why the distinction is particularly important. The reason the exact derivative of f' that you get from AD is useful is because it also approximates the derivative of f, right? If it were possible to make some pathological choice of f' (that is, algorithmic implementation of f) such that the result from AD is a poor approximation of the derivative of f, that's not really a successful use of AD on f, right? So when I say I'm gonna "use AD to get a derivative of f", I'm really saying I'm going to use AD on a good choice of an approximation of f to get a good approximation of the derivative of f, right? I suppose if infinite precision analog computing were possible, or in the context of theoretical real-valued register machines (does anyone still study that?), it matters that AD is exact on the algorithmic implementation, but we're not in or anywhere near those contexts, are we?

gh02t 11y ago
> To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? Yep. I probably should have used f* instead of f' because of context. R->R isn't necessary however, it should generalize quite a bit. As to why the distinction is important, it's because of sensitivity. Approximating a function rarely also approximates its derivative well unless it is constructed to do so (e.g. k-th order Hermite polynomials). As an example, consider the simplest approximation scheme, basic polynomial fitting. They will guarantee that your fit matches the true function at all sample points, but the derivative might (and usually will) be wildly different. You have to be extremely careful differentiating approximate schemes and relating them to the true derivative. Usually what you're interested in for AD is the derivative of the approximate scheme and not an approximate derivative for the exact function (e.g., sensitivity analysis). And I agree that dual numbers are intuitive to me as well, but I think it requires a decent bit of background as to why they work.

gh02t 11y ago

> To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? Yep. I probably should have used f* instead of f' because of context. R->R isn't necessary however, it should generalize quite a bit. As to why the distinction is important, it's because of sensitivity. Approximating a function rarely also approximates its derivative well unless it is constructed to do so (e.g. k-th order Hermite polynomials). As an example, consider the simplest approximation scheme, basic polynomial fitting. They will guarantee that your fit matches the true function at all sample points, but the derivative might (and usually will) be wildly different. You have to be extremely careful differentiating approximate schemes and relating them to the true derivative. Usually what you're interested in for AD is the derivative of the approximate scheme and not an approximate derivative for the exact function (e.g., sensitivity analysis). And I agree that dual numbers are intuitive to me as well, but I think it requires a decent bit of background as to why they work.