3 ms·

> He basically notes that D had a slightly nicer chain rule if you write D'f(a)= (f(a), Df(a)) and use D' instead. That much I understood myself. > His descri

> He basically notes that D had a slightly nicer chain rule if you write D'f(a)= (f(a), Df(a)) and use D' instead. That much I understood myself. > His description of reverse mode AD is more complicated And that's where I got lost in the weeds. I'm sure if I could be bothered, I could sit down for a few weeks or months and gain a working understanding of Category Theory. My point is not whether I understand or don't understand CT, but... what's the point of spending effort in understanding it? The CT of forward and reverse mode Automatic Differentiation is cute, but it doesn't seem to have had much practical application in precisely the area where CT ought to be useful. I mean... both reverse and forward mode AD existed before it was labelled using CT terminology. What did CT add to the SOTA for AD?

ndriscoll 3y ago
If you already understand AD, maybe there's not much point for you. The point may be for others that have some background in linear algebra and maybe rings/modules (so they've seen a few non-contrived examples of categories and functors), and who haven't encountered AD. For them, Conal's description is concise and memorable. Again, it's like describing Fourier transforms as an orthogonal decomposition: for someone who hasn't studied linear algebra, functions being "infinite dimensional vectors" will sound scary, and eyes will glaze over. They might prefer to stick to just working at the level of manipulating integrals and using "clever tricks". For those with more linear algebra background, a bunch of the basic theorems around Fourier transforms become immediate, and you don't need to think about them. The "tricks" turn out to be geometrically motivated. So the thing it added is a compact and simple (and therefore pedagogically useful) description of AD for people with the target background.

ndriscoll 3y ago

If you already understand AD, maybe there's not much point for you. The point may be for others that have some background in linear algebra and maybe rings/modules (so they've seen a few non-contrived examples of categories and functors), and who haven't encountered AD. For them, Conal's description is concise and memorable. Again, it's like describing Fourier transforms as an orthogonal decomposition: for someone who hasn't studied linear algebra, functions being "infinite dimensional vectors" will sound scary, and eyes will glaze over. They might prefer to stick to just working at the level of manipulating integrals and using "clever tricks". For those with more linear algebra background, a bunch of the basic theorems around Fourier transforms become immediate, and you don't need to think about them. The "tricks" turn out to be geometrically motivated. So the thing it added is a compact and simple (and therefore pedagogically useful) description of AD for people with the target background.