4 ms·

> A properly designed compiler can allow developers to optimize matmuls within 100 lines of code. man this is such a funny closing comment - what exactly do yo

> A properly designed compiler can allow developers to optimize matmuls within 100 lines of code. man this is such a funny closing comment - what exactly do you think is involved in designing a compiler that enables devs to optimize matmuls if not 1000s of person hours/years/etc of very "fine-grained" perf research? what the "abstraction" people don't understand (because they only deal in abstractions) is that achieving performance involves literally the antithesis of abstraction - you need to understand your hardware down to the gate level (sometimes). > loop tiling, pipelining, shared memory swizzle, memory coalescing have you ever applied any of these? the only way you could apply these as a generic (without consideration of your particular hardware) algo is using a tuner; this is of course widely the route taken but that's not an "understanding" of anything except guess and check.

pokeypokes 4y ago
Yeah the parent comment kinda reads like "optimizing matrix multiplication isn't hard if someone abstracts away all the hard parts for me"
- mathisfun123 4y ago
  yup exactly; it's like other comments on hn about nn frameworks: "abstraction is the most important thing - look at pytorch it's the best framework because of the perfect/beautiful/brilliant abstractions" (re functorch or fx or dynamo). ignoring entirely how much tedious and grueling bookkeeping/corner-casing/kernel-tuning (by a perpetual 100s of fulltime engineers) presenting such an "abstract" interface to the user requires.
  - junrushao1994 4y ago
    Hey I am the first author of one of the "abstractions", so I guess my words would more or less reflect my personal daily experience dealing with those lovely kernels. Well, I don't have 100 engineers working for me, unfortunately :-( Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton (which is also a small team of < 5 core contributors), what's this abstraction and their key to high performance? It's a tile-based programming model, where a tile could be conveniently lowered to vector instructions, coalesced memory access, and transformed to permuted layout. Its `dot` on tiles can be directly lowed to TensorCore-specific instructions. With those in design, without a huge team painfully maintaining the system, critical kernels like FlashAttention could be quickly developed within say 30 lines of code.
    - mathisfun123 4y ago
      >Hey I am the first author of one of the "abstractions" I know who you are and you should probably be out in the open with the fact that you have a conflict of interest in working at octo, a company that sells a very specific type of ML compiler. >Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton Pretty ironic you would call out Triton is being the right abstraction because while it is true philippe did a very good thing by moving things from warp level to block level, there is absolutely no one that thinks (myself included) that Triton is an abstraction.
      - junrushao1994 4y ago
        I’m using my real name, so its not hard to know who I am. Unfortunately, I don’t know much about you, and actually I don’t really think there is conflict of interest if you work in Modular, because Modular is also developing compiler abstractions, which is something I like and agree with, isn’t it? Let’s discuss about techniques, and it doesn’t have to be that heated :-) To clarify, my point is matmuls can be solved with proper compiler abstractions, and it’s not that hard, and if you are working on a compiler, I believe you would more or less agree with that point, do you? Liking Triton or not is a personal preference, and I use this as an example only because it’s gaining a lot of momentum at the moment, not saying it’s a perfect abstraction. If you personally don’t like it, I could also discuss about exo-compilation, tensor comprehension, but let’s always focus on concrete technical items :-)
mirker 4y ago
There are domain specific languages that use these primitives/abstractions e.g., Halide https://halide-lang.org/tutorials/tutorial_lesson_05_scheduling_1.html https://halide-lang.org/tutorials/tutorial_lesson_05_schedul...
- junrushao1994 4y ago
  Halide is cool and has inspired many brilliant works like Tiramisu and Exocompilation. Love the idea a lot :-) We recently have some follow-ups on this idea as well. Happy to discuss about it and I don't want to distract from this thread...
junrushao1994 4y ago
Yes. I am the first author of the latest generation auto-tuner in an open source deep learning compiler :-) My comment is based on my personal experience: I did lead a 2nd/3rd grade undergrad to add software pipelining support and it worked within 1 month; we did get cutlass-level performance within 100 lines of code specifying the design space.
deleted 4y ago
[deleted]

pokeypokes 4y ago

Yeah the parent comment kinda reads like "optimizing matrix multiplication isn't hard if someone abstracts away all the hard parts for me"

mathisfun123 4y ago
yup exactly; it's like other comments on hn about nn frameworks: "abstraction is the most important thing - look at pytorch it's the best framework because of the perfect/beautiful/brilliant abstractions" (re functorch or fx or dynamo). ignoring entirely how much tedious and grueling bookkeeping/corner-casing/kernel-tuning (by a perpetual 100s of fulltime engineers) presenting such an "abstract" interface to the user requires.
- junrushao1994 4y ago
  Hey I am the first author of one of the "abstractions", so I guess my words would more or less reflect my personal daily experience dealing with those lovely kernels. Well, I don't have 100 engineers working for me, unfortunately :-( Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton (which is also a small team of < 5 core contributors), what's this abstraction and their key to high performance? It's a tile-based programming model, where a tile could be conveniently lowered to vector instructions, coalesced memory access, and transformed to permuted layout. Its `dot` on tiles can be directly lowed to TensorCore-specific instructions. With those in design, without a huge team painfully maintaining the system, critical kernels like FlashAttention could be quickly developed within say 30 lines of code.
  - mathisfun123 4y ago
    >Hey I am the first author of one of the "abstractions" I know who you are and you should probably be out in the open with the fact that you have a conflict of interest in working at octo, a company that sells a very specific type of ML compiler. >Let's instead constructively talk about techniques in concrete items. If you look at OpenAI's Triton Pretty ironic you would call out Triton is being the right abstraction because while it is true philippe did a very good thing by moving things from warp level to block level, there is absolutely no one that thinks (myself included) that Triton is an abstraction.
    - junrushao1994 4y ago
      I’m using my real name, so its not hard to know who I am. Unfortunately, I don’t know much about you, and actually I don’t really think there is conflict of interest if you work in Modular, because Modular is also developing compiler abstractions, which is something I like and agree with, isn’t it? Let’s discuss about techniques, and it doesn’t have to be that heated :-) To clarify, my point is matmuls can be solved with proper compiler abstractions, and it’s not that hard, and if you are working on a compiler, I believe you would more or less agree with that point, do you? Liking Triton or not is a personal preference, and I use this as an example only because it’s gaining a lot of momentum at the moment, not saying it’s a perfect abstraction. If you personally don’t like it, I could also discuss about exo-compilation, tensor comprehension, but let’s always focus on concrete technical items :-)

mirker 4y ago

There are domain specific languages that use these primitives/abstractions e.g., Halide https://halide-lang.org/tutorials/tutorial_lesson_05_scheduling_1.html https://halide-lang.org/tutorials/tutorial_lesson_05_schedul...

junrushao1994 4y ago
Halide is cool and has inspired many brilliant works like Tiramisu and Exocompilation. Love the idea a lot :-) We recently have some follow-ups on this idea as well. Happy to discuss about it and I don't want to distract from this thread...

junrushao1994 4y ago

Yes. I am the first author of the latest generation auto-tuner in an open source deep learning compiler :-) My comment is based on my personal experience: I did lead a 2nd/3rd grade undergrad to add software pipelining support and it worked within 1 month; we did get cutlass-level performance within 100 lines of code specifying the design space.

deleted 4y ago

[deleted]