3 ms·

> I really fail to see what people see even in Nvidia/Cuda Cloud providers are choosing nvidia today as the gpgpu platform on the cloud, driven by neural netwo

> I really fail to see what people see even in Nvidia/Cuda Cloud providers are choosing nvidia today as the gpgpu platform on the cloud, driven by neural network needs. So lots of software will be written with the platform in mind, aka. for nvidia arch / in cuda. This will add friction preventing users to easily change platform (see how long it took for people to switch their workloads to arm for example). It will drives more cloud providers to use nvidia, more software to be written for cuda, and the moat will build itself.

l33tman 3y ago
But there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). Sure some hardcore people write some of their own cuda extensions but the vast majority don't. If there was a competing pytorch/tensorflow-supported backend at half the price, AI companies would quickly fix the quirks and run on that instead. It will be a race to the bottom eventually. Even more so on the cloud where you have a higher tendency to run value added services on top, that in turn are price sensitive.
- brucethemoose2 3y ago
  Some popular ML projects ship with hand optimized CUDA kernels for performance. It is, unfortunately, the standard. If you get around that, sometimes projects will use dependencies that include an odd CUDA-only library. I ran into this a lot with GANs (which often use some weird loss metric with a random hard CUDA dependency) or media libs. Of course there are plenty of solutions, and some are theoretically transparent. But the issue is getting people to use them. Academic ML researchers in particular seem incredibly impatient, looking for the absolute easiest way to prototype something (which usually means coding what they already know). Ease of installation on a wide range of HW is about their last priority.
  - l33tman 3y ago
    Yeah I write CUDA kernels for my own projects as well. There won't be a shift if someone launches something that is 10% better, but if someone launches something that is twice the performance for the same price the companies with huge GPU-bills will pay to port those kernels pretty fast. The kernels I wrote I hand-optimized using Nsight for a specific usecase, and sure they saved me like 200% compared to pure pyTorch and it took a few days, but when your company spends $10M for a single training run, you have quite many of "my days" to spend if another HW platform would save you $5M (but you need to have the calendar time as well to spend, which is another factor in this space that moves crazily fast). If/when the field stabilizes a bit people will start looking at that bottom line I think but right now everybody just "has" to get those H100 cards before someone else gets them I guess...
- Roark66 3y ago
  >But there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). That's true, but the current bottleneck in AI/ML is memory size+ bandwidth. NVidia with their cuda really is the most economic game in town right now on both the Low "hobbyist level" end (nothing beats 2x used rtx3090 setup in value for money), up to "high end" (h100 is still the best generally available card). Some interesting "unified memory" competition appears on the horizon(Apple, AMD), but both are still far from where nvidia is.
  - thiago_fm 3y ago
    Cost vs. benefit here is the most important factor. If somebody would come up with a product Tomorrow that isn't as performant as H100s, but price-wise is cheaper, they could make NVidia miserable, as NVidia is selling those products with a huge margin, they'd have a hard time keeping those margins and a price race to the bottom would happen.
- godelski 3y ago
  > there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). There are a lot of optimizations that CUDA has that are nowhere near supported in other software or even hardware. Custom cuda kernels also aren't as rare as one might think, they will often just be hidden unless you're looking at libraries. Our more well known example is going to be StyleGAN[0] but it isn't uncommon to see elsewhere, even in research code. Swin even has a cuda kernel[1]. Or find torch here[1] (which github reports that 4% of the code is cuda (and 42% C++ and 2% C)). These things are everywhere. I don't think pytorch and tensorflow could ever be agnostic, there will always be a difference just because you have to spend resources differently (developing kernels is time resource). We can draw evidence by looking at Intel MKL, which is still better than open source libraries and has been so for a long time. I really do want AMD to compete in this space. I'd even love a third player like Intel. We really do need competition here, but it would be naive to think that there's going to be a quick catchup here. AMD has a lot of work to do and posting a few bounties and starting a company (idk, called "micro grad"?) isn't going to solve the problem anytime soon. And fwiw, I'm willing to bet that most AI companies would rather run in house servers than from cloud service providers. The truth is that right now just publishing is extremely correlated to compute infrastructure (doesn't need to be but with all the noise we've just said "fuck the poor" because rejecting is easy) and anyone building products has costly infrastructure. [0] https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d041b42ec8e806021a205ed9349f87c6a4/torch_utils/ops/upfirdn2d.cu https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d... [1] https://github.com/microsoft/Swin-Transformer/blob/2cb103f2de145ff43bb9f6fc2ae8800c24ad04c6/kernels/window_process/swin_window_process_kernel.cu#L4 https://github.com/microsoft/Swin-Transformer/blob/2cb103f2d... [2] https://github.com/pytorch/pytorch/tree/main/aten/src https://github.com/pytorch/pytorch/tree/main/aten/src

l33tman 3y ago

But there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). Sure some hardcore people write some of their own cuda extensions but the vast majority don't. If there was a competing pytorch/tensorflow-supported backend at half the price, AI companies would quickly fix the quirks and run on that instead. It will be a race to the bottom eventually. Even more so on the cloud where you have a higher tendency to run value added services on top, that in turn are price sensitive.

brucethemoose2 3y ago
Some popular ML projects ship with hand optimized CUDA kernels for performance. It is, unfortunately, the standard. If you get around that, sometimes projects will use dependencies that include an odd CUDA-only library. I ran into this a lot with GANs (which often use some weird loss metric with a random hard CUDA dependency) or media libs. Of course there are plenty of solutions, and some are theoretically transparent. But the issue is getting people to use them. Academic ML researchers in particular seem incredibly impatient, looking for the absolute easiest way to prototype something (which usually means coding what they already know). Ease of installation on a wide range of HW is about their last priority.
- l33tman 3y ago
  Yeah I write CUDA kernels for my own projects as well. There won't be a shift if someone launches something that is 10% better, but if someone launches something that is twice the performance for the same price the companies with huge GPU-bills will pay to port those kernels pretty fast. The kernels I wrote I hand-optimized using Nsight for a specific usecase, and sure they saved me like 200% compared to pure pyTorch and it took a few days, but when your company spends $10M for a single training run, you have quite many of "my days" to spend if another HW platform would save you $5M (but you need to have the calendar time as well to spend, which is another factor in this space that moves crazily fast). If/when the field stabilizes a bit people will start looking at that bottom line I think but right now everybody just "has" to get those H100 cards before someone else gets them I guess...
Roark66 3y ago
>But there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). That's true, but the current bottleneck in AI/ML is memory size+ bandwidth. NVidia with their cuda really is the most economic game in town right now on both the Low "hobbyist level" end (nothing beats 2x used rtx3090 setup in value for money), up to "high end" (h100 is still the best generally available card). Some interesting "unified memory" competition appears on the horizon(Apple, AMD), but both are still far from where nvidia is.
- thiago_fm 3y ago
  Cost vs. benefit here is the most important factor. If somebody would come up with a product Tomorrow that isn't as performant as H100s, but price-wise is cheaper, they could make NVidia miserable, as NVidia is selling those products with a huge margin, they'd have a hard time keeping those margins and a price race to the bottom would happen.
godelski 3y ago
> there is really nothing that "normal" AI requires that is bound to CUDA. pyTorch and Tensorflow are backend agnostic (ideally...). There are a lot of optimizations that CUDA has that are nowhere near supported in other software or even hardware. Custom cuda kernels also aren't as rare as one might think, they will often just be hidden unless you're looking at libraries. Our more well known example is going to be StyleGAN[0] but it isn't uncommon to see elsewhere, even in research code. Swin even has a cuda kernel[1]. Or find torch here[1] (which github reports that 4% of the code is cuda (and 42% C++ and 2% C)). These things are everywhere. I don't think pytorch and tensorflow could ever be agnostic, there will always be a difference just because you have to spend resources differently (developing kernels is time resource). We can draw evidence by looking at Intel MKL, which is still better than open source libraries and has been so for a long time. I really do want AMD to compete in this space. I'd even love a third player like Intel. We really do need competition here, but it would be naive to think that there's going to be a quick catchup here. AMD has a lot of work to do and posting a few bounties and starting a company (idk, called "micro grad"?) isn't going to solve the problem anytime soon. And fwiw, I'm willing to bet that most AI companies would rather run in house servers than from cloud service providers. The truth is that right now just publishing is extremely correlated to compute infrastructure (doesn't need to be but with all the noise we've just said "fuck the poor" because rejecting is easy) and anyone building products has costly infrastructure. [0] https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d041b42ec8e806021a205ed9349f87c6a4/torch_utils/ops/upfirdn2d.cu https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d... [1] https://github.com/microsoft/Swin-Transformer/blob/2cb103f2de145ff43bb9f6fc2ae8800c24ad04c6/kernels/window_process/swin_window_process_kernel.cu#L4 https://github.com/microsoft/Swin-Transformer/blob/2cb103f2d... [2] https://github.com/pytorch/pytorch/tree/main/aten/src https://github.com/pytorch/pytorch/tree/main/aten/src