6 ms·

This guy is worried about GPU kernels, which are never, ever written in Python. As you point out, Python is a glue language for ML. > There is no single langua

This guy is worried about GPU kernels, which are never, ever written in Python. As you point out, Python is a glue language for ML. > There is no single language with a rich enough ecosystem that can provide literally all of the aforementioned functionality besides Python. That may be true, but some of us are still bitter that all that grew up around an at-least-averagely-annoying language rather than something nicer.

MontyCarloHall 1y ago
>This guy is worried about GPU kernels Then the title should be "why GPU kernel programming needs a new programming language." I can get behind that; I've written CUDA C and it was not fun (though this was over a decade ago and things may have since improved, not to mention that the code I wrote then could today be replaced by a couple lines of PyTorch). That said, GPU kernel programming is fairly niche: for the vast majority of ML applications, the high-level API functions in PyTorch/TensorFlow/JAX/etc. provide optimal GPU performance. It's pretty rare that one would need to implement custom kernels. >which are never, ever written in Python. Not true! Triton is a Python API for writing kernels, which are JIT compiled.
- catgary 1y ago
  I agree with you that writing kernels isn’t necessarily the most important thing for most ML devs. I think an MLIR-first workflow with robust support for the StableHLO and LinAlg dialects is the best path forward for ML/array programming, so on one hand I do applaud what Mojo is doing. But I’m much more interested in how MLIR opens the door to “JAX in <x>”. I think Julia is moving in that direction with Reactant.jl, and I think there’s a Rust project doing something similar (I think burn.dev may be using ONNX has an even higher-level IR). In my ideal world, I would be able to write an ML model and training loop in some highly verified language and call it from Python/Rust for training.
- pama 1y ago
  Although they are incredibly useful, the ML applications, or MLops, or ML Devops, or whatever other production-related application tech stack terminology may come to mind, are providing critical scaffolding infrastructure and glue but are not strictly what comes to mind when you only use the term “Machine Learning”. The key to machine learning is the massively parallel ability to efficiently train large neural network models, and the key to using the benefits of these trained models is the ability to rapidly evaluate them. Yes you need to get data for the training and you need complex infrastructure for the applications but the conferences, papers, studies on machine learning dont refer to these (other than in passing), in part because they are solvable, and in part because they are largely orthogonal to ML. Another way to think about it is that the nvidia price is what goes to infinity in this recent run and not the database or disk drive providers. So if someone finds a way to make the core ML part better with a programming language solution that is certainly very welcome and the title is appropriate. (The fact that GPU programming is considered niche in the current state of ML is a strong argument for keeping the title as is.)
- seanmcdirmid 1y ago
  There are a lot of tricks for writing GPU code in a high level language and using some sort of meta programming to make it work out (I think Conal Elliott first did this in Haskell, where he also does symbolic differentiation in the same paper!).
ModernMech 1y ago
> This guy is worried about GPU kernels, which are never, ever written in Python. As you point out, Python is a glue language for ML. That's kind of the point of Mojo, they're trying to solve the so-called "two language problem" in this space. Why should you need two languages to write your glue code and kernel code? Why can't there be a language which is both as easy to write as Python, but can still express GPU kernels for ML applications? That's what Mojo is trying to be through clever use of LLVM MLIR.
- nostrademons 1y ago
  It's interesting, people have been trying to solve the "two language problem" since before I started professionally programming 25 years ago, and in that time period two-language solutions have just gotten even more common. Back in the 90s they were usually spoken about only in reference to games and shell programming; now the pattern of "scripting language calls out to highly-optimized C or CUDA for compute-intensive tasks" is common for webapps, ML, cryptocurrency, drones, embedded, robotics, etc. I think this is because many, many problem domains have a structure that lends themselves well to two-language solutions. They have a small homogenous computation structure on lots of data that needs to run extremely fast. And they also have a lot of configuration and data-munging that is basically quick one-time setup but has to be specified somewhere, and the more concisely you can specify it, the less human time development takes. The requirements on a language designed to run extremely fast are going to be very different from one that is designed to be as flexible and easy to write as possible. You usually achieve quick execution by eschewing flexibility and picking a programming model that is fairly close to the machine model, but you achieve flexibility by having lots of convenience features built into the language, most of which will have some cost in memory or indirections. There've been a number of attempts at "one language to rule them all", notably PL/1, C++, Julia (in the mathematical programming subdomain), and Common Lisp, but it often feels like the "flexible" subset is shoehorned in to fit the need for zero-cost abstractions, and/or the "compute-optimized" subset is almost a whole separate language that is bolted on with similar but more verbose syntax.
  - soVeryTired 1y ago
jimbokun 1y ago
> That may be true, but some of us are still bitter that all that grew up around an at-least-averagely-annoying language rather than something nicer. Don't worry. If you stick around this industry long enough you'll see this happen several more times.
- Hizonner 1y ago
  I'm basically retired. But I'm still bitter about each of the times...
  - anakaine 1y ago
    Im not. Move on with life and be happy. What we have is functional, easy to develop, and well supported.
    - Hizonner 1y ago
      My bitterness is the only thing that keeps me happy.
almostgotcaught 1y ago
> which are never, ever written in Python nah never ever ever ever ever ... except https://github.com/FlagOpen/FlagGems https://github.com/FlagOpen/FlagGems https://github.com/linkedin/Liger-Kernel https://github.com/linkedin/Liger-Kernel https://github.com/meta-pytorch/applied-ai https://github.com/meta-pytorch/applied-ai https://github.com/gpu-mode/triton-index https://github.com/gpu-mode/triton-index https://github.com/AlibabaPAI/FLASHNN https://github.com/AlibabaPAI/FLASHNN https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth the number of people commenting on this stuff that don't know what they're actually talking about grows by leaps and bounds every day...
- Hizonner 1y ago
  I stand corrected. I should have known people would be doing that in Python. How many of the world's total FLOPs are going through those?
  - almostgotcaught 1y ago
    Triton is a backend for PyTorch. Lately it is the backend. So it's definitely double digits percentage if not over 50%.
    - lairv 1y ago
      It's the backend for torch.compile, pytorch eager mode will still use cuBLAS/cuDNN/custom CUDA kernels, not sure what's the usage of torch.compile
      - almostgotcaught 1y ago
        > not sure what's the usage of torch.compile consider that at minimum both FB and OAI themselves definitely make heavy use of the Triton backend in PyTorch.
    - cavisne 1y ago
      Doesn’t triton write its own intermediate language that then compiles to PTX?
anothernewdude 1y ago
> rather than something nicer. Python was the something nicer. A lot of the other options were so much worse.

MontyCarloHall 1y ago

>This guy is worried about GPU kernels Then the title should be "why GPU kernel programming needs a new programming language." I can get behind that; I've written CUDA C and it was not fun (though this was over a decade ago and things may have since improved, not to mention that the code I wrote then could today be replaced by a couple lines of PyTorch). That said, GPU kernel programming is fairly niche: for the vast majority of ML applications, the high-level API functions in PyTorch/TensorFlow/JAX/etc. provide optimal GPU performance. It's pretty rare that one would need to implement custom kernels. >which are never, ever written in Python. Not true! Triton is a Python API for writing kernels, which are JIT compiled.

catgary 1y ago
I agree with you that writing kernels isn’t necessarily the most important thing for most ML devs. I think an MLIR-first workflow with robust support for the StableHLO and LinAlg dialects is the best path forward for ML/array programming, so on one hand I do applaud what Mojo is doing. But I’m much more interested in how MLIR opens the door to “JAX in <x>”. I think Julia is moving in that direction with Reactant.jl, and I think there’s a Rust project doing something similar (I think burn.dev may be using ONNX has an even higher-level IR). In my ideal world, I would be able to write an ML model and training loop in some highly verified language and call it from Python/Rust for training.
pama 1y ago
Although they are incredibly useful, the ML applications, or MLops, or ML Devops, or whatever other production-related application tech stack terminology may come to mind, are providing critical scaffolding infrastructure and glue but are not strictly what comes to mind when you only use the term “Machine Learning”. The key to machine learning is the massively parallel ability to efficiently train large neural network models, and the key to using the benefits of these trained models is the ability to rapidly evaluate them. Yes you need to get data for the training and you need complex infrastructure for the applications but the conferences, papers, studies on machine learning dont refer to these (other than in passing), in part because they are solvable, and in part because they are largely orthogonal to ML. Another way to think about it is that the nvidia price is what goes to infinity in this recent run and not the database or disk drive providers. So if someone finds a way to make the core ML part better with a programming language solution that is certainly very welcome and the title is appropriate. (The fact that GPU programming is considered niche in the current state of ML is a strong argument for keeping the title as is.)
seanmcdirmid 1y ago
There are a lot of tricks for writing GPU code in a high level language and using some sort of meta programming to make it work out (I think Conal Elliott first did this in Haskell, where he also does symbolic differentiation in the same paper!).

ModernMech 1y ago

> This guy is worried about GPU kernels, which are never, ever written in Python. As you point out, Python is a glue language for ML. That's kind of the point of Mojo, they're trying to solve the so-called "two language problem" in this space. Why should you need two languages to write your glue code and kernel code? Why can't there be a language which is both as easy to write as Python, but can still express GPU kernels for ML applications? That's what Mojo is trying to be through clever use of LLVM MLIR.

nostrademons 1y ago
It's interesting, people have been trying to solve the "two language problem" since before I started professionally programming 25 years ago, and in that time period two-language solutions have just gotten even more common. Back in the 90s they were usually spoken about only in reference to games and shell programming; now the pattern of "scripting language calls out to highly-optimized C or CUDA for compute-intensive tasks" is common for webapps, ML, cryptocurrency, drones, embedded, robotics, etc. I think this is because many, many problem domains have a structure that lends themselves well to two-language solutions. They have a small homogenous computation structure on lots of data that needs to run extremely fast. And they also have a lot of configuration and data-munging that is basically quick one-time setup but has to be specified somewhere, and the more concisely you can specify it, the less human time development takes. The requirements on a language designed to run extremely fast are going to be very different from one that is designed to be as flexible and easy to write as possible. You usually achieve quick execution by eschewing flexibility and picking a programming model that is fairly close to the machine model, but you achieve flexibility by having lots of convenience features built into the language, most of which will have some cost in memory or indirections. There've been a number of attempts at "one language to rule them all", notably PL/1, C++, Julia (in the mathematical programming subdomain), and Common Lisp, but it often feels like the "flexible" subset is shoehorned in to fit the need for zero-cost abstractions, and/or the "compute-optimized" subset is almost a whole separate language that is bolted on with similar but more verbose syntax.
- soVeryTired 1y ago

jimbokun 1y ago

> That may be true, but some of us are still bitter that all that grew up around an at-least-averagely-annoying language rather than something nicer. Don't worry. If you stick around this industry long enough you'll see this happen several more times.

Hizonner 1y ago
I'm basically retired. But I'm still bitter about each of the times...
- anakaine 1y ago
  Im not. Move on with life and be happy. What we have is functional, easy to develop, and well supported.
  - Hizonner 1y ago
    My bitterness is the only thing that keeps me happy.

almostgotcaught 1y ago

> which are never, ever written in Python nah never ever ever ever ever ... except https://github.com/FlagOpen/FlagGems https://github.com/FlagOpen/FlagGems https://github.com/linkedin/Liger-Kernel https://github.com/linkedin/Liger-Kernel https://github.com/meta-pytorch/applied-ai https://github.com/meta-pytorch/applied-ai https://github.com/gpu-mode/triton-index https://github.com/gpu-mode/triton-index https://github.com/AlibabaPAI/FLASHNN https://github.com/AlibabaPAI/FLASHNN https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth the number of people commenting on this stuff that don't know what they're actually talking about grows by leaps and bounds every day...

Hizonner 1y ago
I stand corrected. I should have known people would be doing that in Python. How many of the world's total FLOPs are going through those?
- almostgotcaught 1y ago
  Triton is a backend for PyTorch. Lately it is the backend. So it's definitely double digits percentage if not over 50%.
  - lairv 1y ago
    It's the backend for torch.compile, pytorch eager mode will still use cuBLAS/cuDNN/custom CUDA kernels, not sure what's the usage of torch.compile
    - almostgotcaught 1y ago
      > not sure what's the usage of torch.compile consider that at minimum both FB and OAI themselves definitely make heavy use of the Triton backend in PyTorch.
  - cavisne 1y ago
    Doesn’t triton write its own intermediate language that then compiles to PTX?

anothernewdude 1y ago

> rather than something nicer. Python was the something nicer. A lot of the other options were so much worse.