I forgot an important technique that is somewhere between numeric differentiation and automatic differentiation: (4) Complex-step differentiation http://citese

I forgot an important technique that is somewhere between numeric differentiation and automatic differentiation: (4) Complex-step differentiation http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.141.8002 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.141.... The technique is clever: they add an finite difference along the imaginary axis, and this turns out to give a more accurate derivative than normal forward differences. We don't support this in Ceres and suggest autodiff instead.