10 ms·
> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. I am a big fan of Jim Keller, but this is s
by belval 3y ago
> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast.
I am a big fan of Jim Keller, but this is semantics that argues CUDA the language is not a moat, when most people refer to CUDA the libraries.
CUDA has first party support in most of the libraries in use today, if you were to stray from that happy path you must have deep pocket to work through the hurdles of working with something like XLA.
There's a reason why Microsoft, Google, Amazon and Meta are still buying Nvidia accelerators even when they have their own in-house accelerators.
- dist-epoch 3y agoThe impossibility of buying NVIDIA GPUs in quantity today is a giant incentive to port the most important libraries to AMD/....
- photonbeam 3y agoIm surprised we dont hear more about this, it seems like a huge motivator
- cjbgkagh 3y agoI think you’re assuming these big companies are serious about AI itself and not just joining in on a trend. They would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money.
- hackerlight 3y agoAndrew Ng says ROCm is a lot better today than early 2023
- pjmlp 3y agoThe computing shops on my city shopping malls have enough laptops with NVIDIA GPUs, while they aren't at the same level as desktop or server GPUs, there are available and usable for people to learn CUDA, quite the contrary with most AMD offerings.
- dist-epoch 3y agoNot sure what you mean, both NVIDIA and AMD desktop GPUs are available, there is no particular shortage. Prices can be high for top ones, like NVIDIA 4090, but you can buy it. Server GPUs are the ones difficult to buy. People don't really learn CUDA, they use it mostly through another library like PyTorch.
- pjmlp 3y agoMost AMD stuff doesn't work on consumer GPUs. Try to make ROCm work on a random AMD card.
- westhanover 3y agoWell he is obviously just talking his book. He is selling a Nvidia competitor. If CUDA isn’t beautiful and the future deserves to be beautiful, why don’t you make it and show us, Jim? All we are getting is more hot air about CUDA. That is all AMD and Intel have as well, bold statements about the future and no code.
- neural_thing 3y agoHe isn't though. He hasn't been at AMD or Intel for quite some time.
- westhanover 3y agoHe sells AI chips that don’t run CUDA.
- fisf 3y agoJim is at tenstorrent now.
- HarHarVeryFunny 3y agoCUDA/etc is certainly a pain. A good number of years ago I wrote my own Torch-like C++ NN framework using CUDA (cuDNN, cuBLAS) for the GPU, and one of the annoyances is that cuDNN has incomplete coverage of even the basic operators needed. Add/Sub/Min/Max/Sqrt/Negate are all provided as part of cuDNN, but if you want other common NN building blocks like Div/Exp/Log/Pow/Inv/InvSqrt then you have to write them yourself in CUDA and either forgo cuDNNs tensor-descriptor layout flexibility or re-implement that yourself too. Of course frameworks like PyTorch support all the operators you'd expect, since they've written their own kernels in CUDA where the functionality is missing from cuDNN. It's not clear exactly what Keller is referring to there. When people say CUDA they might be referring to the entire ecosystem (nvcc CUDA C/C++ compiler, CUDA API's, higher level cuBLAS, cuDNN, etc), or maybe just the base compiler (which lets you write your own kernels) and API for allocating memory, queueing kernels, etc. The cuDNN kernels (convolution, etc) are highly optimized, as is cuBLAS (e.g. matmul), and I doubt anyone is going to do better writing these themselves. Does Keller consider using cuDNN as "writing CUDA" ? As far as I'm aware the higher level, performant, CUDA libraries, as well as specialized components like TensorRT are written in CUDA, although that could mean a combination of C/C++ & PTX pseudo-assembler (ptxas is really a compiler, not an assembler). The alternative would be they they were written in hand optimized SASS assembler which afaik is only available outside of NVDIA via the Open Source CuAssembler. I believe Mojo support for NVIDA is based on PTX. I'm not sure if that would be really be considered as "CUDA" or not if they are not using nvcc at all. Most people, outside of framework vendors, would have no reason to use CUDA anyway, since it's just too low level. The only sane use case would be where writing a custom kernel in (e.g.) PyTorch or Mojo doesn't get the performance you want and you write than one kernel in CUDA. The hope would be that the Mojo compiler is good enough that this would not be necessary.
- m463 3y ago> "Basically nobody writes CUDA," Can't help but think of that old yogi berra quote: "Nobody goes there anymore, it's too crowded" (but yeah, I suspect a lot of people use cuda via accelerated libraries/apps like opencv)