4 ms·

> What's not clear to me in this paper is the practical efficiency of their technique especially for high-order derivatives. On section "5.3.1 Order reduction

> What's not clear to me in this paper is the practical efficiency of their technique especially for high-order derivatives. On section "5.3.1 Order reduction for nested applications": > We gained the ability of writing a differentiable program acting on derivatives of another program, stressed as crucial (but lacking in most models) by other authors [18]. See http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1370&context=ecetr http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1370&... for [18]. --- > Right now, there exist good AD tools that work just fine with control structures, recursion, and composition. See their example implementation (https://github.com/ZigaSajovic/dCpp https://github.com/ZigaSajovic/dCpp) to understand what "control structures, recursion and composition" mean in this context.

kxyvr 10y ago
Thanks for the reply. Perlmutter and Siskind's paper doesn't really talk about the efficiency of higher-order derivatives either and only has a very simple run time for a gradient calculation. I had a look at the code and I can't figure out how their abstractions would allow for efficient higher-order derivatives. Even on the front page, they list the examples // df/dxdx std::cout<<"df/dxdx = "<<f.d(&x).d(&x).id<<std::endl; // df/dxdy std::cout<<"df/dxdy = "<<f.d(&x).d(&y).id<<std::endl; Generally, I think those abstractions make it hard to compute things like Hessian-vector products because we really need a reverse pass to generate either a program or computational tree that we can fly through with a forward pass. That involves giving an explicit instruction to run a reverse pass since we don't do it automatically since it's expensive. Also, we generally give an entire vector to the forward pass at one time since we can overlap a lot of computation. Finding second order derivatives one at a time, like `df/dxdy` for an entire Hessian tends to be slow. It looks like most of the action is in var.cpp, but I can't see how this would run fast. Do you have any idea how this scales up for derivative calculations (gradient and Hess-vec) as we get at least into the thousands of variables in terms of derivative time vs evaluation time with no AD abstractions? Certainly, I may be misunderstanding something. That said, if the efficiency doesn't scale, it seems like the practical significance will be limited.

kxyvr 10y ago

Thanks for the reply. Perlmutter and Siskind's paper doesn't really talk about the efficiency of higher-order derivatives either and only has a very simple run time for a gradient calculation. I had a look at the code and I can't figure out how their abstractions would allow for efficient higher-order derivatives. Even on the front page, they list the examples // df/dxdx std::cout<<"df/dxdx = "<<f.d(&x).d(&x).id<<std::endl; // df/dxdy std::cout<<"df/dxdy = "<<f.d(&x).d(&y).id<<std::endl; Generally, I think those abstractions make it hard to compute things like Hessian-vector products because we really need a reverse pass to generate either a program or computational tree that we can fly through with a forward pass. That involves giving an explicit instruction to run a reverse pass since we don't do it automatically since it's expensive. Also, we generally give an entire vector to the forward pass at one time since we can overlap a lot of computation. Finding second order derivatives one at a time, like `df/dxdy` for an entire Hessian tends to be slow. It looks like most of the action is in var.cpp, but I can't see how this would run fast. Do you have any idea how this scales up for derivative calculations (gradient and Hess-vec) as we get at least into the thousands of variables in terms of derivative time vs evaluation time with no AD abstractions? Certainly, I may be misunderstanding something. That said, if the efficiency doesn't scale, it seems like the practical significance will be limited.