It's also known as "automatic differentiation" -- it's quite different from numerical/symbolic differentiation. More information here: - https://justindomke.w

It's also known as "automatic differentiation" -- it's quite different from numerical/symbolic differentiation. More information here: - https://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox/ https://justindomke.wordpress.com/2009/02/17/automatic-diffe... - https://wiki.haskell.org/Automatic_Differentiation https://wiki.haskell.org/Automatic_Differentiation The key idea is extending common operators (+, -, product, /, key mathematical functions) that usually operate on _real numbers_ to tuples of real numbers (x, dx) (the quantity and its derivative with respect to some variable) such that the operations preserve the properties of differentiation. For instance (with abuse of notation): - (x1, dx1) + (x2, dx2) = (x1 + x2, dx1 + dx2). - (x1, dx1) * (x2, dx2) = (x1 * y1, x1 * dx2 + x2 * dx1). - sin((x, dx)) = (sin(x), cos(x)). Note that the right element of the tuple can be computed precisely from quantities readily available from the inputs to the operator. It's also extensible to derivatives of scalars that are functions of many variables by a vector (of those variables) (common in machine learning). It's beautifully implemented in Google's Ceres optimisation package: https://ceres-solver.googlesource.com/ceres-solver/+/1.8.0/include/ceres/jet.h https://ceres-solver.googlesource.com/ceres-solver/+/1.8.0/i...