Yes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did tha

Yes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did that in 80s and 90s. Once we had autodiff, it became obvious that deriving backward pass by hand is no-go.