To state it more clearly than some of the other comments, part of the code insight this paper provides is the following: Subject to transformation of your prog

To state it more clearly than some of the other comments, part of the code insight this paper provides is the following: Subject to transformation of your program into the first order / categorical representation, reverse mode auto differentiation is simply about 1) transforming the program into the continuation passing style category (this “flips” the shape of the compose operation in the category of our program) 2) do the same calculation / transform as you’d do for forward mode. Aka the really simple derivative calc we learn in high school. There’s an extra trick to be done to make things even better, but that’s the meat of it. That reverse mode becomes (almost) as simple as forward mode to implement. Subject to your language being in this sort of categorical first order representation :) Reverse mode being simple by any name to explain is a huge innovation alone.