3 ms·
While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms.
by claytonjy 4y ago
While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms.
Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.
- modeless 4y agoI disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.
- brrrrrm 4y agoI disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks. full stable diffusion in <800 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/examples/stable_diffusion.py#L532 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... autograd in <30 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/tensor.py#L123-L152 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... Adam in <20 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/nn/optim.py#L51 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...