3 ms·

> Hand-written assembly kernels don’t scale! I used to think this. And I think, in theory, it is true. But the fact of the matter is, modern ML just doesn't

> Hand-written assembly kernels don’t scale! I used to think this. And I think, in theory, it is true. But the fact of the matter is, modern ML just doesn't use that many kernels. Every framework uses the same libraries (BLAS) and every library uses the same basic idea (maximally saturate FMA-like units). Large language models are being run natively on commodity hardware with code written from scratch within days of their release (e.g. llama.cpp). From a conceptual standpoint, it's really easy to saturate hardware in this domain. It's been pretty easy since 2014 when convolutions were interpreted as matrix multiplications. Sure, the actual implementations can be tricky, but a single engineer (trained in it) can get that done for a specific hardware in a couple months. Of course, the interesting problem is how to generalize kernel generation. I spent years working with folks trying to do just that. But, in retrospect, the actual value add from a system that does all this for you is quite low. It's a realization I've been struggling to accept :'(

tysam_and 4y ago
We....do use kernels and kernel generation in the ML field. Every day. I'm confused as to a few of the points made here, unless the first sentence is meaning '...doesn't use that many hand-written kernels'. All of the convolutions we have run on kernels, some pre-built and customized/chosen from a list based on performance, and some dynamically generated. PyTorch 2.0 for example decomposes and fuses operations, then uses OpenAI's Triton to dynamically generate a custom fused kernel that tends to be very efficient. There are still hand-written kernels, even, Flash-Attention and the Memory-Efficient attention papers both caused huge leaps forward because they manually went through a lot of the inefficiencies of naive matrix multiplies for attention w.r.t. the hardware design and optimized it quite a lot. I think generalized kernel generation though may have more life in it than you might suspect! It is a fascinating field and I do not know nearly enough about it. I hope someday to be able to write my own Triton kernels/get to know how it integrates as a dynamic compiler for PyTorch code. We certainly live in wild times. Crazy indeed.
- brrrrrm 4y ago
  My claim is pretty subjective, but the idea is that there aren't many distinct kernels used in machine learning. It's all tensor contractions and element-wise operations. I'd argue that this can be maintained by hand without need for automation or high level abstraction. Triton is used in a templated way for a very specific albeit pervasive hardware (PTX compatible GPUs), which is why it works so well. Here's some of the code: https://github.com/pytorch/pytorch/blob/a66625da3bcdf1e262ddf7bc7fedaa070147c78e/torch/_inductor/kernel/mm.py#L16-L71 https://github.com/pytorch/pytorch/blob/a66625da3bcdf1e262dd... Generalized kernel generation (i.e. synthesis of optimal performance from non-expert user defined kernels and novel hardware) would be fantastic to have, but it just doesn't seem particularly necessary in the field.
tzhenghao 4y ago
Good points, but I'd push back just a little. > Sure, the actual implementations can be tricky, but a single engineer (trained in it) can get that done for a specific hardware in a couple months. I want to agree with you on this, but in practice, it's... 1. Hard to hire that engineer with deep expertise in handwritten kernels. CUDA engineers are still hard to come by and doesn't scale with productionized AI engineering demand. 2. "A few months" is a tough pill to swallow from an engineering roadmap POV, especially when models are deployed on a monthly basis. Most of the hand tuning efforts aren't scalable and will have to be done again on most iterations. This is especially true in reinforcement learning and robotics. > But, in retrospect, the actual value add from a system that does all this for you is quite low. It's a realization I've been struggling to accept. Yeah I remain neutral on this. On one hand, I can see that especially having to invest significant engineering effort (see point 2 above). On the other hand, you won't really know until you start benchmarking these models (and as you should).

tysam_and 4y ago

We....do use kernels and kernel generation in the ML field. Every day. I'm confused as to a few of the points made here, unless the first sentence is meaning '...doesn't use that many hand-written kernels'. All of the convolutions we have run on kernels, some pre-built and customized/chosen from a list based on performance, and some dynamically generated. PyTorch 2.0 for example decomposes and fuses operations, then uses OpenAI's Triton to dynamically generate a custom fused kernel that tends to be very efficient. There are still hand-written kernels, even, Flash-Attention and the Memory-Efficient attention papers both caused huge leaps forward because they manually went through a lot of the inefficiencies of naive matrix multiplies for attention w.r.t. the hardware design and optimized it quite a lot. I think generalized kernel generation though may have more life in it than you might suspect! It is a fascinating field and I do not know nearly enough about it. I hope someday to be able to write my own Triton kernels/get to know how it integrates as a dynamic compiler for PyTorch code. We certainly live in wild times. Crazy indeed.

brrrrrm 4y ago
My claim is pretty subjective, but the idea is that there aren't many distinct kernels used in machine learning. It's all tensor contractions and element-wise operations. I'd argue that this can be maintained by hand without need for automation or high level abstraction. Triton is used in a templated way for a very specific albeit pervasive hardware (PTX compatible GPUs), which is why it works so well. Here's some of the code: https://github.com/pytorch/pytorch/blob/a66625da3bcdf1e262ddf7bc7fedaa070147c78e/torch/_inductor/kernel/mm.py#L16-L71 https://github.com/pytorch/pytorch/blob/a66625da3bcdf1e262dd... Generalized kernel generation (i.e. synthesis of optimal performance from non-expert user defined kernels and novel hardware) would be fantastic to have, but it just doesn't seem particularly necessary in the field.

tzhenghao 4y ago

Good points, but I'd push back just a little. > Sure, the actual implementations can be tricky, but a single engineer (trained in it) can get that done for a specific hardware in a couple months. I want to agree with you on this, but in practice, it's... 1. Hard to hire that engineer with deep expertise in handwritten kernels. CUDA engineers are still hard to come by and doesn't scale with productionized AI engineering demand. 2. "A few months" is a tough pill to swallow from an engineering roadmap POV, especially when models are deployed on a monthly basis. Most of the hand tuning efforts aren't scalable and will have to be done again on most iterations. This is especially true in reinforcement learning and robotics. > But, in retrospect, the actual value add from a system that does all this for you is quite low. It's a realization I've been struggling to accept. Yeah I remain neutral on this. On one hand, I can see that especially having to invest significant engineering effort (see point 2 above). On the other hand, you won't really know until you start benchmarking these models (and as you should).