I agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words,

I agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words, write that autodiff for yourself, and then use production grade ML solution where you won't ever write backward pass by hand. The APL paper doesn't have implementation of autodiff, rather 90s style hand derivations.