3 ms·

Modern GPU Programming for MLSys

80 points by crowwork 3mo ago

mathisfun123 3mo ago
"Modern [NVIDIA GPU] Programming for ..." Everything after "Pipelining GEMM with TMA" (inclusive) is specific to NVIDIA. Which is fine but the title (of the guide itself) is clearly misleading.
- nh23423fefe 3mo ago
  > Our main target is the Blackwell generation, misleading?
  - mathisfun123 3mo ago
    what is it with hn people where they willfully misinterpret the simplest observations; > the title (of the guide itself) is clearly misleading. ... > title: the distinguishing name of a written, printed, or filmed production do you understand now? or do i need to also define for you the word misleading?
    - nh23423fefe 3mo ago
      nah talking to you sucks.
hazard 3mo ago
This looks great, but I'd really like to see associated exercises (and solutions) to make it useful for self-study
throwaw12 3mo ago
So many frameworks are being built. What are the state of the art frameworks in ML programming area? Similar to what React is for web and tailwind for CSS Triton, ONNX, JAX, PyTorch, cublass, ..... I know they might be for different purposes, but having some idea what is for what and when to use would be helpful
- mathisfun123 3mo ago
  > ONNX, JAX, PyTorch these are model-level frameworks > Triton this is a kernel DSL > [cublas] this is a BLAS library built atop CUDA > I know they might be for different purposes, but having some idea what is for what and when to use would be helpful when people ask this question i always ask: who are you and what is your job? if you're not an ML/DL/AI person then you knowing the specifics is about as useful as me knowing the specifics of react/express/angular/tailwind/django/whatever as an ML person. this is not meant to be condescending, this is meant to allay your anxiety, ie that if you ever find yourself in the position where you have to know these things for your job, it won't be that hard to figure out (just like it isn't that hard to figure out the difference between react and express and django if you're a webdev).
  - throwaw12 3mo ago
    I am a product engineer in yet another enterprise SaaS CRUD shop, who wants to learn more about the landscape and find the way to enter it eventually.
    - mathisfun123 3mo ago
      > wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here's your moment of zen: so does ML/DL/AI. it really really does suck. it's basically just as bad as webdev in terms of tedium/boredom/incidental complexity/etc. it's not fun, interesting, exciting, whatever else you're projecting based on an outside-looking-in-perspective. now i'll acknowledge that there's one big difference: the pay is way better at the far end of the distribution - meaning if you can get to a FAANG ML team then you'll get more money than you're probably getting now (and a ton more stress too) and it's even more than the CRUD devs in FAANG. fine. but ask yourself if it's really worth learning a whole heap of new bullshit just for a chance at more money (no guarantee). okay now a useful/practical answer: i went back to school for a PhD but i should've just dropped out with the MS. do that. even better do Georgia Tech's online MS.
reinitctxoffset 3mo ago
I can't signal boost this enough. I spent months, months of late nights watching commits to nvfuser and shit, I wrote a SASS decompiler instrumented everything trying to learn Blackwell. This is the first time I've seen something so clean, just a real work of scholarship on it. My hat is off to the authors and the contribution it represents. If I would caution a reader anything it's that the 2CTA (sm_100 sm_110) patterns here are different on 1CTA in important ways and it's not a better / worse thing, they are good for different workloads. Really outstanding work. I proves q lot of this in lean4 and published but I got lazy short of really doing the pedagogical work. This is what you should be starting with if you want to max out 2CTA gear, it's immaculate.
zander_jiang 3mo ago
The book is really well written.

80 points by crowwork 3mo ago

"Modern [NVIDIA GPU] Programming for ..." Everything after "Pipelining GEMM with TMA" (inclusive) is specific to NVIDIA. Which is fine but the title (of the guide itself) is clearly misleading.

nh23423fefe 3mo ago
> Our main target is the Blackwell generation, misleading?
- mathisfun123 3mo ago
  what is it with hn people where they willfully misinterpret the simplest observations; > the title (of the guide itself) is clearly misleading. ... > title: the distinguishing name of a written, printed, or filmed production do you understand now? or do i need to also define for you the word misleading?
  - nh23423fefe 3mo ago
    nah talking to you sucks.

hazard 3mo ago

This looks great, but I'd really like to see associated exercises (and solutions) to make it useful for self-study

throwaw12 3mo ago

So many frameworks are being built. What are the state of the art frameworks in ML programming area? Similar to what React is for web and tailwind for CSS Triton, ONNX, JAX, PyTorch, cublass, ..... I know they might be for different purposes, but having some idea what is for what and when to use would be helpful

mathisfun123 3mo ago
> ONNX, JAX, PyTorch these are model-level frameworks > Triton this is a kernel DSL > [cublas] this is a BLAS library built atop CUDA > I know they might be for different purposes, but having some idea what is for what and when to use would be helpful when people ask this question i always ask: who are you and what is your job? if you're not an ML/DL/AI person then you knowing the specifics is about as useful as me knowing the specifics of react/express/angular/tailwind/django/whatever as an ML person. this is not meant to be condescending, this is meant to allay your anxiety, ie that if you ever find yourself in the position where you have to know these things for your job, it won't be that hard to figure out (just like it isn't that hard to figure out the difference between react and express and django if you're a webdev).
- throwaw12 3mo ago
  I am a product engineer in yet another enterprise SaaS CRUD shop, who wants to learn more about the landscape and find the way to enter it eventually.
  - mathisfun123 3mo ago
    > wants to learn more about the landscape and find the way to enter it eventually let's swap roles and let's pretend i'm an ML engineer asking you how to enter CRUD. what would you tell me? my strong suspicion (if i caught you in an honest, frank, moment) is you would say to me "why the fuck would you want to do that - it sucks". i have this suspicion because i did actually used to do CRUD and it does suck! but here's your moment of zen: so does ML/DL/AI. it really really does suck. it's basically just as bad as webdev in terms of tedium/boredom/incidental complexity/etc. it's not fun, interesting, exciting, whatever else you're projecting based on an outside-looking-in-perspective. now i'll acknowledge that there's one big difference: the pay is way better at the far end of the distribution - meaning if you can get to a FAANG ML team then you'll get more money than you're probably getting now (and a ton more stress too) and it's even more than the CRUD devs in FAANG. fine. but ask yourself if it's really worth learning a whole heap of new bullshit just for a chance at more money (no guarantee). okay now a useful/practical answer: i went back to school for a PhD but i should've just dropped out with the MS. do that. even better do Georgia Tech's online MS.

reinitctxoffset 3mo ago

I can't signal boost this enough. I spent months, months of late nights watching commits to nvfuser and shit, I wrote a SASS decompiler instrumented everything trying to learn Blackwell. This is the first time I've seen something so clean, just a real work of scholarship on it. My hat is off to the authors and the contribution it represents. If I would caution a reader anything it's that the 2CTA (sm_100 sm_110) patterns here are different on 1CTA in important ways and it's not a better / worse thing, they are good for different workloads. Really outstanding work. I proves q lot of this in lean4 and published but I got lazy short of really doing the pedagogical work. This is what you should be starting with if you want to max out 2CTA gear, it's immaculate.

zander_jiang 3mo ago

The book is really well written.