Thanks for the reply. Perlmutter and Siskind's paper doesn't really talk about the efficiency of higher-order derivatives either and only has a very simple run

Thanks for the reply. Perlmutter and Siskind's paper doesn't really talk about the efficiency of higher-order derivatives either and only has a very simple run time for a gradient calculation. I had a look at the code and I can't figure out how their abstractions would allow for efficient higher-order derivatives. Even on the front page, they list the examples // df/dxdx std::cout<<"df/dxdx = "<<f.d(&x).d(&x).id<<std::endl; // df/dxdy std::cout<<"df/dxdy = "<<f.d(&x).d(&y).id<<std::endl; Generally, I think those abstractions make it hard to compute things like Hessian-vector products because we really need a reverse pass to generate either a program or computational tree that we can fly through with a forward pass. That involves giving an explicit instruction to run a reverse pass since we don't do it automatically since it's expensive. Also, we generally give an entire vector to the forward pass at one time since we can overlap a lot of computation. Finding second order derivatives one at a time, like `df/dxdy` for an entire Hessian tends to be slow. It looks like most of the action is in var.cpp, but I can't see how this would run fast. Do you have any idea how this scales up for derivative calculations (gradient and Hess-vec) as we get at least into the thousands of variables in terms of derivative time vs evaluation time with no AD abstractions? Certainly, I may be misunderstanding something. That said, if the efficiency doesn't scale, it seems like the practical significance will be limited.