4 ms·

I don't see how we're this far in and so many competitors are still trying to ignore the massive market penetration of CUDA. I don't care how fast your chip is

I don't see how we're this far in and so many competitors are still trying to ignore the massive market penetration of CUDA. I don't care how fast your chip is if I can only run 20% of the already tiny amount of software capable of leveraging this acceleration.

collegeburner 3y ago
some stuff is still CUDA-dependent (a lot of scientific code). other stuff is not (major ML frameworks and many math libraries). nobody said CUDA was obsolete, just that there are great options for many use cases that no longer rely on it.
vessenes 3y ago
I think this was a more accurate assessment a couple of years ago. AMD historically under invested in their software stack, as is well documented. But, they are catching up. The prevalence of pytorch makes it easier for hardware vendors to target a single set of libraries for implementation and get broad distribution. Even Apple has made major progress getting mps support in more broadly, first directly into pytorch, and they are now charting their own course with MPX with some early interesting successes. Zuck recently used the word "H100 equivalent" in compute, and my memory is he indicated roughly 35% of that compute was not from NVIDIA -- that will all be AMD. There's still work to do -- lots of repositories still contain `if device=="cuda"` type language, but my own experience is that manging code around to use apple gpus has gotten vastly easier this year, and I see more and more AMD GPU owners floating around github issues with resolvable problems. A year ago they were barely present. All that said - people aren't ignoring it - and entrepreneurs and cloud monopolies are putting real resources into opening things up. I think the playing field will continue to level / get back to competitive over the next two to three years.
- ShamelessC 3y ago
  The main issue is until you solve 99% of all the problems, you’re still introducing substantial friction to every potential user who may have had to go from zero coding knowledge needed to very specialized coding knowledge needed. And while many problems are trivial to fix at eg the PyTorch layer, lots of stuff like flash attention, DeepSpeed, etc. are coded directly in CUDA kernels.
glalonde 3y ago
Is that true and is that sustainable? My understanding has been that only the relatively low level libraries/kernels are written in cuda and all the magical algorithms are in python using various ML libaries. It's like how intel BLAS isn't much of a moat -- there are several open source implementations and you can mix and match. How is CUDA so sticky when most ML devs aren't writing CUDA but something several layers of abstraction above it? Why can't intel, AMD, google w/e come along and write an adapter for that lowest level to TF, pytorch or whatever is the framework of the day?
- 1024core 3y ago
  > Why can't intel, AMD, google w/e come along and write an adapter for that lowest level to TF, pytorch or whatever is the framework of the day? A long long time ago, i.e. the last time AMD was competing with Intel (before this time, that is), we used to use Intel's icc in our lab to optimize for the Intel CPUs and squeeze as much as possible out of them. Then AMD came out with their "Athlon"(?) and it was an Intel-beater at that time. But AMD never released a compiler for it; I bet they had one internally, but we had to rely on plain old GCC. These hardware companies don't seem to get that a kick-ass software can really add wings to their hardware sales. If I were a hardware vendor, I would, if nothing else, make my hardware's software open so the community can run with it and create better software; which will result in more hardware sales!
- cavisne 3y ago
  It's a good question. I think fundamentally its because no wants to/can compete with Nvidia when making a general purpose parallel processor. They all want to make something a bit more specialized, so they need to guess what functionality is needed and not needed. This is a really tricky guess, case in point that AMD's latest chip cant compete on training because they could not get Flash Attention 2 working on the backward pass because of their hardware architecture. [1] Attempts to abstract at a higher layer have failed so far because that lower layer is really valuable, again Flash Attention is a good example. [1] https://www.semianalysis.com/p/amd-mi300-performance-faster-than https://www.semianalysis.com/p/amd-mi300-performance-faster-...
benreesman 3y ago
They’re not ignoring it, they’re eroding it. The API that matters is Torch, and only the API. Letting NVIDIA charge famine prices is both a bad idea and a huge incentive to write software bridging the gap.
- kkielhofner 3y ago
  I don’t know if you do work in this space but this wasn’t accurate several years ago and it certainly isn’t accurate now. The amount of extremely CUDA specific handwritten compute architecture code, custom kernels, etc has exploded and other than a few things here and there (yes, like torch SDPA) we’re waaaay past vanilla torch for anything beyond a toy. The pricing and theoretical hardware specs of these novelties doesn’t matter when an inferior on paper Nvidia product will wipe the floor with the other in the real world. People have spent 15 years wringing every last penny out of CUDA on Nvidia hardware. There is some light shining through but you’ll still see things like “Woo-hoo FlashAttention finally supports ROCm! Oh wait, what’s that, FlashAttention2 has been running on CUDA for six months?” Don’t even get me started on the “alternative” software stacks and drivers.
  - pjmlp 3y ago
    Also the anti-CUDA folks keep forgetting the polyglot nature of the ecosystem, made possible since PTX was introduced on CUDA 3.0. The graphical debugging tools that allows stepping through GPU code just like with on the CPU.
- pjmlp 3y ago
  For those like myself Torch means nothing, I don't use GPU for machine learning rather graphics programming and general purpose compute.
indymike 3y ago
> already tiny amount of software capable of leveraging this acceleration. This is why everyone is trying to compete with CUDA. Everyone wants a slice of the pie as we go from tiny amount to everything.
postalrat 3y ago
You aren't the customer they want.

collegeburner 3y ago

some stuff is still CUDA-dependent (a lot of scientific code). other stuff is not (major ML frameworks and many math libraries). nobody said CUDA was obsolete, just that there are great options for many use cases that no longer rely on it.

vessenes 3y ago

I think this was a more accurate assessment a couple of years ago. AMD historically under invested in their software stack, as is well documented. But, they are catching up. The prevalence of pytorch makes it easier for hardware vendors to target a single set of libraries for implementation and get broad distribution. Even Apple has made major progress getting mps support in more broadly, first directly into pytorch, and they are now charting their own course with MPX with some early interesting successes. Zuck recently used the word "H100 equivalent" in compute, and my memory is he indicated roughly 35% of that compute was not from NVIDIA -- that will all be AMD. There's still work to do -- lots of repositories still contain `if device=="cuda"` type language, but my own experience is that manging code around to use apple gpus has gotten vastly easier this year, and I see more and more AMD GPU owners floating around github issues with resolvable problems. A year ago they were barely present. All that said - people aren't ignoring it - and entrepreneurs and cloud monopolies are putting real resources into opening things up. I think the playing field will continue to level / get back to competitive over the next two to three years.

ShamelessC 3y ago
The main issue is until you solve 99% of all the problems, you’re still introducing substantial friction to every potential user who may have had to go from zero coding knowledge needed to very specialized coding knowledge needed. And while many problems are trivial to fix at eg the PyTorch layer, lots of stuff like flash attention, DeepSpeed, etc. are coded directly in CUDA kernels.

glalonde 3y ago

Is that true and is that sustainable? My understanding has been that only the relatively low level libraries/kernels are written in cuda and all the magical algorithms are in python using various ML libaries. It's like how intel BLAS isn't much of a moat -- there are several open source implementations and you can mix and match. How is CUDA so sticky when most ML devs aren't writing CUDA but something several layers of abstraction above it? Why can't intel, AMD, google w/e come along and write an adapter for that lowest level to TF, pytorch or whatever is the framework of the day?

1024core 3y ago
> Why can't intel, AMD, google w/e come along and write an adapter for that lowest level to TF, pytorch or whatever is the framework of the day? A long long time ago, i.e. the last time AMD was competing with Intel (before this time, that is), we used to use Intel's icc in our lab to optimize for the Intel CPUs and squeeze as much as possible out of them. Then AMD came out with their "Athlon"(?) and it was an Intel-beater at that time. But AMD never released a compiler for it; I bet they had one internally, but we had to rely on plain old GCC. These hardware companies don't seem to get that a kick-ass software can really add wings to their hardware sales. If I were a hardware vendor, I would, if nothing else, make my hardware's software open so the community can run with it and create better software; which will result in more hardware sales!
cavisne 3y ago
It's a good question. I think fundamentally its because no wants to/can compete with Nvidia when making a general purpose parallel processor. They all want to make something a bit more specialized, so they need to guess what functionality is needed and not needed. This is a really tricky guess, case in point that AMD's latest chip cant compete on training because they could not get Flash Attention 2 working on the backward pass because of their hardware architecture. [1] Attempts to abstract at a higher layer have failed so far because that lower layer is really valuable, again Flash Attention is a good example. [1] https://www.semianalysis.com/p/amd-mi300-performance-faster-than https://www.semianalysis.com/p/amd-mi300-performance-faster-...

benreesman 3y ago

They’re not ignoring it, they’re eroding it. The API that matters is Torch, and only the API. Letting NVIDIA charge famine prices is both a bad idea and a huge incentive to write software bridging the gap.

kkielhofner 3y ago
I don’t know if you do work in this space but this wasn’t accurate several years ago and it certainly isn’t accurate now. The amount of extremely CUDA specific handwritten compute architecture code, custom kernels, etc has exploded and other than a few things here and there (yes, like torch SDPA) we’re waaaay past vanilla torch for anything beyond a toy. The pricing and theoretical hardware specs of these novelties doesn’t matter when an inferior on paper Nvidia product will wipe the floor with the other in the real world. People have spent 15 years wringing every last penny out of CUDA on Nvidia hardware. There is some light shining through but you’ll still see things like “Woo-hoo FlashAttention finally supports ROCm! Oh wait, what’s that, FlashAttention2 has been running on CUDA for six months?” Don’t even get me started on the “alternative” software stacks and drivers.
- pjmlp 3y ago
  Also the anti-CUDA folks keep forgetting the polyglot nature of the ecosystem, made possible since PTX was introduced on CUDA 3.0. The graphical debugging tools that allows stepping through GPU code just like with on the CPU.
pjmlp 3y ago
For those like myself Torch means nothing, I don't use GPU for machine learning rather graphics programming and general purpose compute.

indymike 3y ago

> already tiny amount of software capable of leveraging this acceleration. This is why everyone is trying to compete with CUDA. Everyone wants a slice of the pie as we go from tiny amount to everything.

postalrat 3y ago

You aren't the customer they want.