5 ms·

Introduction to Automatic Differentiation

25 points by noelwelsh 12y ago

TTPrograms 12y ago
So there's a new name for taylor expansions now?
- RBerenguel 12y ago
  It's a name that has been used for a long while in Numerical Analysis. After all, saying Taylor expansion doesn't help much when integrating DE's (in some sense even Runge-Kutta methods are a kind of Taylor expansion), for instance, whereas AD does give a lot more information.
- jwmerrill 12y ago
  AD is a way (actually ways) of organizing a computation. There are many different ways to organize the computation and evaluation of a Taylor series at a point. As an exercise, try to explain the difference between symbolic differentiation, forward mode AD, and reverse mode AD. You will find that there is more content here than just a new name.
noelwelsh 12y ago
Automatic differentiation computes with so-called dual numbers, which store an infinitesimal (e) alongside the result of the computation. The infinitesimal is the derivative. Working with infinitesimals is how calculus started out but was abandoned when calculus was formalized using epsilon-delta limits (http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition_of_limit http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition...) The is a newer formalization of calculus, called non-standard analysis, that works with infinitesimals, called the hyperreals. As you can imagine, there is a strong relationship between dual numbers and hyperreals. I found this exchange interesting: http://math.stackexchange.com/questions/341535/is-the-theory-of-dual-numbers-strong-enough-to-develop-real-analysis-and-does-i http://math.stackexchange.com/questions/341535/is-the-theory... It's kinda cool that non-standard analysis slips in through the back door when using dual numbers. I always found working (informally!) with infinitesimals quite simple.
scentoni 12y ago
Q: "What’s the derivative of a data structure anyway?" A: The Derivative of a Regular Type is its Type of One-Hole Contexts http://strictlypositive.org/diff.pdf http://strictlypositive.org/diff.pdf
- agumonkey 12y ago
  Amazing, I was looking for such a paper for a long time.

25 points by noelwelsh 12y ago

TTPrograms 12y ago

So there's a new name for taylor expansions now?

RBerenguel 12y ago
It's a name that has been used for a long while in Numerical Analysis. After all, saying Taylor expansion doesn't help much when integrating DE's (in some sense even Runge-Kutta methods are a kind of Taylor expansion), for instance, whereas AD does give a lot more information.
jwmerrill 12y ago
AD is a way (actually ways) of organizing a computation. There are many different ways to organize the computation and evaluation of a Taylor series at a point. As an exercise, try to explain the difference between symbolic differentiation, forward mode AD, and reverse mode AD. You will find that there is more content here than just a new name.

noelwelsh 12y ago

Automatic differentiation computes with so-called dual numbers, which store an infinitesimal (e) alongside the result of the computation. The infinitesimal is the derivative. Working with infinitesimals is how calculus started out but was abandoned when calculus was formalized using epsilon-delta limits (http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition_of_limit http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition...) The is a newer formalization of calculus, called non-standard analysis, that works with infinitesimals, called the hyperreals. As you can imagine, there is a strong relationship between dual numbers and hyperreals. I found this exchange interesting: http://math.stackexchange.com/questions/341535/is-the-theory-of-dual-numbers-strong-enough-to-develop-real-analysis-and-does-i http://math.stackexchange.com/questions/341535/is-the-theory... It's kinda cool that non-standard analysis slips in through the back door when using dual numbers. I always found working (informally!) with infinitesimals quite simple.

scentoni 12y ago

Q: "What’s the derivative of a data structure anyway?" A: The Derivative of a Regular Type is its Type of One-Hole Contexts http://strictlypositive.org/diff.pdf http://strictlypositive.org/diff.pdf

agumonkey 12y ago
Amazing, I was looking for such a paper for a long time.