3 ms·
Nobody should write backward pass by hand.
by mlajtos 3y ago
Nobody should write backward pass by hand.
- dpflan 3y agoMaybe grad students...
- mlajtos 3y agoYes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did that in 80s and 90s. Once we had autodiff, it became obvious that deriving backward pass by hand is no-go.
- constantcrying 3y agoWhy? As a learning experience implementing backpropagation is extremely helpful, implementing an entire FNN/CNN from scratch is, to be honest. Also Implementing some basic automatic differentiation is something you should probably have done once in your life you are interested in Machine learning or numerical mathematics.
- mkaic 3y agoAgreed. Implementing backprop myself—even if it was a crappy, slow version in MatLab—is what finally got me to understand it. I've worked as an ML researcher for 2 years since then and I'm still routinely happy that I have that deeper understanding of what's going on under the hood of the models I'm training.
- mlajtos 3y agoI agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words, write that autodiff for yourself, and then use production grade ML solution where you won't ever write backward pass by hand. The APL paper doesn't have implementation of autodiff, rather 90s style hand derivations.
- xchip 3y agoI did it and it wasn't worth it.
- mcbuilder 3y agoTell that to any ML Compiler Engineer