4 ms·

> Isn't the theory (at least the mathematical, abstract version of it) already fully represented in books like Griewank that I mentioned in another comment here

by bmitc 4y ago

> Isn't the theory (at least the mathematical, abstract version of it) already fully represented in books like Griewank that I mentioned in another comment here in this post? I guess it's in there somewhere, but like many of the papers, I find that book to overcomplicate things and just not to my liking. I also don't see any mention of dual numbers, although it's probably somewhat implicit in the development (maybe). I'm mainly interested in the dual number approach. With the dual number approach and forward-mode automatic differentiation, there's really no reason to overcomplicate the developing by discuss "propagating tangents". The development can be more much intuitively done because using dual numbers, automatic differentiation just falls out without discussion of propagation.

data_maan 4y ago
Yes, the book is hard to read. I just went through a few chapters of it. > One of the papers posted led me to be more comfortable with the concept of differentiable programming as a distinct thing. I'd still be interested in knowing which of the papera that was ;)
kragen 4y ago
I think autodiff is mostly interesting because of optimization by gradient descent, and gradient descent is only efficient using reverse-mode autodiff, which can't be done with dual numbers. I think optimization by gradient descent (and its variants like Adam) is interesting because it's the only optimization algorithm family I know of whose runtime scales linearly oreven near-linearly with the number of dimensions, so for large-dimensionality problems it's the only computationally feasible algorithm. Also, though it's only guaranteed to find the optimum for differentiable convex problems, in practice it seems to work well enough to be interesting for nonconvex problems. I think optimization is interesting because it allows you to solve arbitrary inverse problems, or at least continuous relaxations of them.

by bmitc 4y ago

data_maan 4y ago

Yes, the book is hard to read. I just went through a few chapters of it. > One of the papers posted led me to be more comfortable with the concept of differentiable programming as a distinct thing. I'd still be interested in knowing which of the papera that was ;)

kragen 4y ago

I think autodiff is mostly interesting because of optimization by gradient descent, and gradient descent is only efficient using reverse-mode autodiff, which can't be done with dual numbers. I think optimization by gradient descent (and its variants like Adam) is interesting because it's the only optimization algorithm family I know of whose runtime scales linearly oreven near-linearly with the number of dimensions, so for large-dimensionality problems it's the only computationally feasible algorithm. Also, though it's only guaranteed to find the optimum for differentiable convex problems, in practice it seems to work well enough to be interesting for nonconvex problems. I think optimization is interesting because it allows you to solve arbitrary inverse problems, or at least continuous relaxations of them.