4 ms·

> CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes Interesting take, but why

> CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes Interesting take, but why would demand for training and inference wade? This seems like a very contrarian take.

talldayo 2y ago
Maybe it won't - I say "time will tell" because we really do not know how much LLMs will be demanded in 10 years. Nvidia's stock skyrocketed because they were incidentally prepared for an enormous increase in demand the moment it happened. Now that expectations are cooling down and Sam Altman is signalling that AGI is a long ways off, the math that justified designing NPU/TPU hardware in-house might not add up anymore. Even if you believe in the tech itself, the hype is cooling and the do-or-die moment is rapidly approaching. My overall point is that I think Nvidia played smartly from the start. They could derive profit from any sufficiently large niche their competitors were too afraid to exploit, and general purpose GPU compute was the perfect investment. With AMD, Apple and the rest of the industry focusing on simpler GPUs, Nvidia was given an empty soapbox to market CUDA with. The big question is whether demand for CUDA can be supplanted with application-specific accelerators.
- felarof 2y ago
  > The big question is whether demand for CUDA can be supplanted with application-specific accelerators. At least for AI workloads, Google's XLA compiler and the JAX ML framework have reduced the need for something like CUDA. There are two main ways to train ML models today: 1) Kernel-heavy approach: This is where frameworks like PyTorch are used, and developers write custom kernels (using Triton or CUDA) to speed up certain ops. 2) Compiler-heavy approach: This uses tools like XLA, which apply techniques like op fusion and compiler optimizations to automatically generate fast, low-level code. NVIDIA's CUDA is a major strength in the first approach. But if the second approach gains more traction, NVIDIA’s advantage might not be as important. And I think the second approach has a strong chance of succeeding, given that two massive companies—Google (TPUs) and Amazon (Trainium)—are heavily investing in it. (PS: I'm also bit biased towards approach 2), we build llama3 fine-tuning on TPU https://github.com/felafax/felafax https://github.com/felafax/felafax)
  - 01100011 2y ago
    It's weird to me that folks think NVDA is just sitting there, waiting for everyone to take their lunch. Yes, I'm totally sure NVDA is completely blind to competition and has chosen to sit on their cash rather than develop alternatives...</s>
  - ein0p 2y ago
    Not really, no. Over the past several years, JAX was used in only 3% of top publications. PyTorch in 60%. There's no trend to suggest that JAX has "reduced the need" for anything, except for Google itself. https://paperswithcode.com/trends https://paperswithcode.com/trends
    - orf 2y ago
      Not sure if publications are a good proxy for this - there’s definitely a selection bias there, and inertia.
      - ein0p 2y ago
        Feel free to suggest a more reliable proxy if you don't like this one. What this means is your chance to find a person who knows/has experience with JAX are very low.
QuadmasterXLII 2y ago
Nvidia’s hardware optimized for gaming was the best crypto miner out of the gate, although it was eventually surpassed for crypto mining by asics. NVIDIA’s cards which (by market share) were mostly sold as crypto miners, were nonetheless the best llm accelerators out of the gate. When in three years the best selling compute task is suddenly fleeple inverting, 900 lines of CUDA will turn Nvidia’s LLM cards into the worlds fastest fleeple inverters, while TPUs will continue to be very good token predictors

talldayo 2y ago

Maybe it won't - I say "time will tell" because we really do not know how much LLMs will be demanded in 10 years. Nvidia's stock skyrocketed because they were incidentally prepared for an enormous increase in demand the moment it happened. Now that expectations are cooling down and Sam Altman is signalling that AGI is a long ways off, the math that justified designing NPU/TPU hardware in-house might not add up anymore. Even if you believe in the tech itself, the hype is cooling and the do-or-die moment is rapidly approaching. My overall point is that I think Nvidia played smartly from the start. They could derive profit from any sufficiently large niche their competitors were too afraid to exploit, and general purpose GPU compute was the perfect investment. With AMD, Apple and the rest of the industry focusing on simpler GPUs, Nvidia was given an empty soapbox to market CUDA with. The big question is whether demand for CUDA can be supplanted with application-specific accelerators.

felarof 2y ago
> The big question is whether demand for CUDA can be supplanted with application-specific accelerators. At least for AI workloads, Google's XLA compiler and the JAX ML framework have reduced the need for something like CUDA. There are two main ways to train ML models today: 1) Kernel-heavy approach: This is where frameworks like PyTorch are used, and developers write custom kernels (using Triton or CUDA) to speed up certain ops. 2) Compiler-heavy approach: This uses tools like XLA, which apply techniques like op fusion and compiler optimizations to automatically generate fast, low-level code. NVIDIA's CUDA is a major strength in the first approach. But if the second approach gains more traction, NVIDIA’s advantage might not be as important. And I think the second approach has a strong chance of succeeding, given that two massive companies—Google (TPUs) and Amazon (Trainium)—are heavily investing in it. (PS: I'm also bit biased towards approach 2), we build llama3 fine-tuning on TPU https://github.com/felafax/felafax https://github.com/felafax/felafax)
- 01100011 2y ago
  It's weird to me that folks think NVDA is just sitting there, waiting for everyone to take their lunch. Yes, I'm totally sure NVDA is completely blind to competition and has chosen to sit on their cash rather than develop alternatives...</s>
- ein0p 2y ago
  Not really, no. Over the past several years, JAX was used in only 3% of top publications. PyTorch in 60%. There's no trend to suggest that JAX has "reduced the need" for anything, except for Google itself. https://paperswithcode.com/trends https://paperswithcode.com/trends
  - orf 2y ago
    Not sure if publications are a good proxy for this - there’s definitely a selection bias there, and inertia.
    - ein0p 2y ago
      Feel free to suggest a more reliable proxy if you don't like this one. What this means is your chance to find a person who knows/has experience with JAX are very low.

QuadmasterXLII 2y ago

Nvidia’s hardware optimized for gaming was the best crypto miner out of the gate, although it was eventually surpassed for crypto mining by asics. NVIDIA’s cards which (by market share) were mostly sold as crypto miners, were nonetheless the best llm accelerators out of the gate. When in three years the best selling compute task is suddenly fleeple inverting, 900 lines of CUDA will turn Nvidia’s LLM cards into the worlds fastest fleeple inverters, while TPUs will continue to be very good token predictors