3 ms·

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other kernels, and again times several different GPU architectures. If you don't care about getting peak utilization of your hardware you can run state of the art neural nets with a truly tiny amount of code. The algorithms are so simple you don't even need any libraries, it's easy enough to write everything from scratch even in low level languages. It's a fun exercise. But it will be many orders of magnitude less efficient so you'll have to wait a really long time for it to run.

claytonjy 4y ago
While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms. Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.
- modeless 4y ago
  I disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.
- brrrrrm 4y ago
  I disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks. full stable diffusion in <800 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/examples/stable_diffusion.py#L532 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... autograd in <30 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/tensor.py#L123-L152 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... Adam in <20 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/nn/optim.py#L51 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...
amelius 4y ago
Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine? I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need. Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of space.
- modeless 4y ago
  CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN for now. The real problem with deep learning on Nvidia is the Linux driver situation. Ugh. Hopefully one day they will come to their senses.
  - amelius 4y ago
    It's not just disk size. Also memory size, and loading speed. Yes, I agree about the driver situation.
    - modeless 4y ago
      The disk size of the shared library is not indicative of RAM usage or loading speed. Shared libraries are memory mapped with demand paging. Only the actually used parts of the library will be loaded into RAM.

claytonjy 4y ago

While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms. Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.

modeless 4y ago
I disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.
brrrrrm 4y ago
I disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks. full stable diffusion in <800 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/examples/stable_diffusion.py#L532 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... autograd in <30 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/tensor.py#L123-L152 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778... Adam in <20 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc37786ffdafc5dc0e990df593/tinygrad/nn/optim.py#L51 https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

amelius 4y ago

Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine? I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need. Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of space.

modeless 4y ago
CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN for now. The real problem with deep learning on Nvidia is the Linux driver situation. Ugh. Hopefully one day they will come to their senses.
- amelius 4y ago
  It's not just disk size. Also memory size, and loading speed. Yes, I agree about the driver situation.
  - modeless 4y ago
    The disk size of the shared library is not indicative of RAM usage or loading speed. Shared libraries are memory mapped with demand paging. Only the actually used parts of the library will be loaded into RAM.