4 ms·

My take: optimizing matrix multiplication is not hard on modern architecture if you have the right abstraction. The code itself could be fragmented across diffe

by junrushao1994 4y ago

My take: optimizing matrix multiplication is not hard on modern architecture if you have the right abstraction. The code itself could be fragmented across different programming models, which is true, but the underlying techniques are not hard for a 2nd/3rd year undergrad to understand. There are only a few important ones on GPU: loop tiling, pipelining, shared memory swizzle, memory coalescing. A properly designed compiler can allow developers to optimize matmuls within 100 lines of code.

touisteur 4y ago
Looking at the effort plunked into things like cutlass and them still not reaching cuBLAS perf (which very few can beat - in the places where cuBLAS shines! which is... not that many...), and even in cuDNN and they're still eeking out single digit improvements regularly, I'd say this is probably harder than that. At least if you're reaching for the >50% use of the 37 TFLOPS of an A40. If you're fine throwing more GPUs at the problem, sure. Edit: I mean when you still see papers every year with large improvements in perf, and things like 'we used tensor cores and managed to get back fp32 accuracy with 3 rounds of the things' - what? - I can attest it doesn't take 2 weeks to get this kind of results. And it's just getting started on tensor cores! And when on the nvidia forums someone says 'nah probably no improvement to use tensor cores for fft' and you get a link with a paper with a significative improvement in perf using tensor cores, I say we're just starting.
- junrushao1994 4y ago
  This is definitely a great point! With the context of AI workloads, where critical matmuls are basically of regular large shapes, are there many cases where cutlass/Triton are worse than cuBLAS where we need to throw more GPUs at it?
  - touisteur 4y ago
    cuBLAS is very often too heavy (too much overhead, memory movement to fit the API, not optimized for small batches of small matrices) and you can get huge improvements while chaining cudnn/cutlass/autotuned kernels. Especially if you're still on GDDR6 every data movement is a killer so if you can put it all together and never go back to global memory, you get amazing improvements. And this is without tensor cores. Programming them by hand is a pain so here enters cutlass...
    - junrushao1994 4y ago
      Yeah cuBLAS is definitely not perfect in many cases :-(( Speaking of GEMM fusion that you mentioned, flash attention is basically GEMM fusion with online softmax right? This is something I believe really cool and can be made really easy wit a proper abstraction. Say, you may move a chunk of computation under a certain loop and instruct the compiler to optimize data movement or cache intermediate tiles somewhere on chip
      - touisteur 4y ago
        There's something of this in cutlass with prologues and epilogues, and in the 'backend mode' of cudnn, but overall breaking the 'cuBLAS takes your whole device and tries to saturate it for this one matmul' is going to require a huge lot of abstraction work. Cutlass is supposed to be the first step and to anyone who struggles to understand WTF you're doing when using it, you are not alone. I've seen literally amazing room-silencing stuff with it, but heavy template stuff is really not my thing.
        junrushao1994 4y ago
        I am personally a really huge fan of cutlass, and I've almost read every single file in their `include/cutlass/` folder (haven't followed up with the `cute` stuff yet). Just like you said, really appreciate that we could actually understand what is going on internally inside the kernel with cutlass, and customize it in a way that cuBLAS doesn't necessarily provide. Have to agree with you that the template stuff is really annoying. Well, even with some template tricks, the error messages are still less readable, and it is where I think a better abstraction could benefit. Imagine you have an abstraction that simplifies the those threadblocks/warps/etc in a unified way while generalizing it to more backends (AMDGPU, Vulkan, AVX512-VNNI, etc), providing more friendly error messages along compilation given the abstraction is almost certainly more structured than pure c++ code.
        touisteur 4y ago
        I know what you mean. I guess Triton might be one way people are actually trying that, and further than that there's a lot of people putting years of work into MLIR-based tech, trying to abstract in a retargetable way the algorithm from its 'scheduling'. Might be worth a look if you're into this :-)
- junrushao1994 4y ago
  > we used tensor cores and managed to get back fp32 accuracy with 3 rounds of the things Hey are you referring to 3xTF32 (https://github.com/NVIDIA/cutlass/tree/master/examples/28_ampere_3xtf32_fast_accurate_tensorop_fprop https://github.com/NVIDIA/cutlass/tree/master/examples/28_am...)? IMO this is a perfect example where proper abstraction could save engineers non-trivial amount of time - imagine a compiler stack which allows 3xTF32 as a normal dtype and subsequent analysis compatible with this special dtype :-)
mathisfun123 4y ago
> A properly designed compiler can allow developers to optimize matmuls within 100 lines of code. man this is such a funny closing comment - what exactly do you think is involved in designing a compiler that enables devs to optimize matmuls if not 1000s of person hours/years/etc of very "fine-grained" perf research? what the "abstraction" people don't understand (because they only deal in abstractions) is that achieving performance involves literally the antithesis of abstraction - you need to understand your hardware down to the gate level (sometimes). > loop tiling, pipelining, shared memory swizzle, memory coalescing have you ever applied any of these? the only way you could apply these as a generic (without consideration of your particular hardware) algo is using a tuner; this is of course widely the route taken but that's not an "understanding" of anything except guess and check.
- pokeypokes 4y ago
  Yeah the parent comment kinda reads like "optimizing matrix multiplication isn't hard if someone abstracts away all the hard parts for me"
  - mathisfun123 4y ago
    yup exactly; it's like other comments on hn about nn frameworks: "abstraction is the most important thing - look at pytorch it's the best framework because of the perfect/beautiful/brilliant abstractions" (re functorch or fx or dynamo). ignoring entirely how much tedious and grueling bookkeeping/corner-casing/kernel-tuning (by a perpetual 100s of fulltime engineers) presenting such an "abstract" interface to the user requires.
    - junrushao1994 4y ago
      Hey I am the first author of one of the "abstractions", so I guess my words would more or less reflect my personal daily experience dealing with those lovely kernels. Well, I don't have 100 engineers working for me, unfortunately :-( Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton (which is also a small team of < 5 core contributors), what's this abstraction and their key to high performance? It's a tile-based programming model, where a tile could be conveniently lowered to vector instructions, coalesced memory access, and transformed to permuted layout. Its `dot` on tiles can be directly lowed to TensorCore-specific instructions. With those in design, without a huge team painfully maintaining the system, critical kernels like FlashAttention could be quickly developed within say 30 lines of code.

touisteur 4y ago

Looking at the effort plunked into things like cutlass and them still not reaching cuBLAS perf (which very few can beat - in the places where cuBLAS shines! which is... not that many...), and even in cuDNN and they're still eeking out single digit improvements regularly, I'd say this is probably harder than that. At least if you're reaching for the >50% use of the 37 TFLOPS of an A40. If you're fine throwing more GPUs at the problem, sure. Edit: I mean when you still see papers every year with large improvements in perf, and things like 'we used tensor cores and managed to get back fp32 accuracy with 3 rounds of the things' - what? - I can attest it doesn't take 2 weeks to get this kind of results. And it's just getting started on tensor cores! And when on the nvidia forums someone says 'nah probably no improvement to use tensor cores for fft' and you get a link with a paper with a significative improvement in perf using tensor cores, I say we're just starting.

junrushao1994 4y ago
This is definitely a great point! With the context of AI workloads, where critical matmuls are basically of regular large shapes, are there many cases where cutlass/Triton are worse than cuBLAS where we need to throw more GPUs at it?
- touisteur 4y ago
  cuBLAS is very often too heavy (too much overhead, memory movement to fit the API, not optimized for small batches of small matrices) and you can get huge improvements while chaining cudnn/cutlass/autotuned kernels. Especially if you're still on GDDR6 every data movement is a killer so if you can put it all together and never go back to global memory, you get amazing improvements. And this is without tensor cores. Programming them by hand is a pain so here enters cutlass...
  - junrushao1994 4y ago
    Yeah cuBLAS is definitely not perfect in many cases :-(( Speaking of GEMM fusion that you mentioned, flash attention is basically GEMM fusion with online softmax right? This is something I believe really cool and can be made really easy wit a proper abstraction. Say, you may move a chunk of computation under a certain loop and instruct the compiler to optimize data movement or cache intermediate tiles somewhere on chip
    - touisteur 4y ago
      There's something of this in cutlass with prologues and epilogues, and in the 'backend mode' of cudnn, but overall breaking the 'cuBLAS takes your whole device and tries to saturate it for this one matmul' is going to require a huge lot of abstraction work. Cutlass is supposed to be the first step and to anyone who struggles to understand WTF you're doing when using it, you are not alone. I've seen literally amazing room-silencing stuff with it, but heavy template stuff is really not my thing.
      - junrushao1994 4y ago
        I am personally a really huge fan of cutlass, and I've almost read every single file in their `include/cutlass/` folder (haven't followed up with the `cute` stuff yet). Just like you said, really appreciate that we could actually understand what is going on internally inside the kernel with cutlass, and customize it in a way that cuBLAS doesn't necessarily provide. Have to agree with you that the template stuff is really annoying. Well, even with some template tricks, the error messages are still less readable, and it is where I think a better abstraction could benefit. Imagine you have an abstraction that simplifies the those threadblocks/warps/etc in a unified way while generalizing it to more backends (AMDGPU, Vulkan, AVX512-VNNI, etc), providing more friendly error messages along compilation given the abstraction is almost certainly more structured than pure c++ code.
        touisteur 4y ago
        I know what you mean. I guess Triton might be one way people are actually trying that, and further than that there's a lot of people putting years of work into MLIR-based tech, trying to abstract in a retargetable way the algorithm from its 'scheduling'. Might be worth a look if you're into this :-)
junrushao1994 4y ago
> we used tensor cores and managed to get back fp32 accuracy with 3 rounds of the things Hey are you referring to 3xTF32 (https://github.com/NVIDIA/cutlass/tree/master/examples/28_ampere_3xtf32_fast_accurate_tensorop_fprop https://github.com/NVIDIA/cutlass/tree/master/examples/28_am...)? IMO this is a perfect example where proper abstraction could save engineers non-trivial amount of time - imagine a compiler stack which allows 3xTF32 as a normal dtype and subsequent analysis compatible with this special dtype :-)

mathisfun123 4y ago

> A properly designed compiler can allow developers to optimize matmuls within 100 lines of code. man this is such a funny closing comment - what exactly do you think is involved in designing a compiler that enables devs to optimize matmuls if not 1000s of person hours/years/etc of very "fine-grained" perf research? what the "abstraction" people don't understand (because they only deal in abstractions) is that achieving performance involves literally the antithesis of abstraction - you need to understand your hardware down to the gate level (sometimes). > loop tiling, pipelining, shared memory swizzle, memory coalescing have you ever applied any of these? the only way you could apply these as a generic (without consideration of your particular hardware) algo is using a tuner; this is of course widely the route taken but that's not an "understanding" of anything except guess and check.

pokeypokes 4y ago
Yeah the parent comment kinda reads like "optimizing matrix multiplication isn't hard if someone abstracts away all the hard parts for me"
- mathisfun123 4y ago
  yup exactly; it's like other comments on hn about nn frameworks: "abstraction is the most important thing - look at pytorch it's the best framework because of the perfect/beautiful/brilliant abstractions" (re functorch or fx or dynamo). ignoring entirely how much tedious and grueling bookkeeping/corner-casing/kernel-tuning (by a perpetual 100s of fulltime engineers) presenting such an "abstract" interface to the user requires.
  - junrushao1994 4y ago
    Hey I am the first author of one of the "abstractions", so I guess my words would more or less reflect my personal daily experience dealing with those lovely kernels. Well, I don't have 100 engineers working for me, unfortunately :-( Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton (which is also a small team of < 5 core contributors), what's this abstraction and their key to high performance? It's a tile-based programming model, where a tile could be conveniently lowered to vector instructions, coalesced memory access, and transformed to permuted layout. Its `dot` on tiles can be directly lowed to TensorCore-specific instructions. With those in design, without a huge team painfully maintaining the system, critical kernels like FlashAttention could be quickly developed within say 30 lines of code.