> To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is

> To be clear, when you say f' you're not talking about Newton's notation for the derivative of f, you're talking about a function from floats to floats that is the approximation/algorithmic implementation of f (which is R->R), right? Yep. I probably should have used f* instead of f' because of context. R->R isn't necessary however, it should generalize quite a bit. As to why the distinction is important, it's because of sensitivity. Approximating a function rarely also approximates its derivative well unless it is constructed to do so (e.g. k-th order Hermite polynomials). As an example, consider the simplest approximation scheme, basic polynomial fitting. They will guarantee that your fit matches the true function at all sample points, but the derivative might (and usually will) be wildly different. You have to be extremely careful differentiating approximate schemes and relating them to the true derivative. Usually what you're interested in for AD is the derivative of the approximate scheme and not an approximate derivative for the exact function (e.g., sensitivity analysis). And I agree that dual numbers are intuitive to me as well, but I think it requires a decent bit of background as to why they work.