4 ms·
U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning
- mlajtos 3y agoNobody should write backward pass by hand.
- dpflan 3y agoMaybe grad students...
- mlajtos 3y agoYes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did that in 80s and 90s. Once we had autodiff, it became obvious that deriving backward pass by hand is no-go.
- constantcrying 3y agoWhy? As a learning experience implementing backpropagation is extremely helpful, implementing an entire FNN/CNN from scratch is, to be honest. Also Implementing some basic automatic differentiation is something you should probably have done once in your life you are interested in Machine learning or numerical mathematics.
- mkaic 3y agoAgreed. Implementing backprop myself—even if it was a crappy, slow version in MatLab—is what finally got me to understand it. I've worked as an ML researcher for 2 years since then and I'm still routinely happy that I have that deeper understanding of what's going on under the hood of the models I'm training.
- mlajtos 3y agoI agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words, write that autodiff for yourself, and then use production grade ML solution where you won't ever write backward pass by hand. The APL paper doesn't have implementation of autodiff, rather 90s style hand derivations.
- xchip 3y agoI did it and it wasn't worth it.
- mcbuilder 3y agoTell that to any ML Compiler Engineer
- sctb 3y agoIt's neat to see ongoing Co-dfns work from Aaron and others! There are a number of YouTube videos online if anyone is interested in very cool and esoteric yet serious programming: https://www.youtube.com/playlist?list=PLDU0iEj6f8duXzmgnlGX4hMHJUMYh4rJq https://www.youtube.com/playlist?list=PLDU0iEj6f8duXzmgnlGX4....
- natas 3y agoI wonder what it would look like on kdb+?
- lopuhin 3y agoImpressively concise implementation, really interesting paper! Benchmark looks quite questionable though -- e.g. they use fp64 (while any sane person would use at least f32, if not f16), batch 1 (while normally one would try to get the max batch size which fits into memory, and it would reach 1 only for much bigger models or inputs), and also measure the time including transfer to/from GPU (while it would normally be interleaved with GPU operations). Not sure what results would look like in a more realistic setup, but still getting within 2x of PyTorch even in such a setting looks impressive!
- fulafel 3y agoInteresting Futhark mention as well in the Related work section: > Another approach to GPU-based array programming with an APL focus is the TAIL/Futhark system [8], which is a compiler chain taking APL to the TAIL (Typed Array Inter- mediate Language) and then compiling TAIL code using the Futhark GPU compiler backend.