3 ms·

> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternative

> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed. Your margin is my opportunity - Jeff Bezos

bigyabai 2mo ago
But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today. The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware. I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.
- coredog64 2mo ago
  > you can't even reuse the same architecture for training and inference AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`
  - bigyabai 2mo ago
    I stand corrected, only Google's post-Ironwood TPUs have the split as well. Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
- acdha 2mo ago
  > But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today. How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized. My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
  - bigyabai 2mo ago
    Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won. Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.

bigyabai 2mo ago

But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today. The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware. I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.

coredog64 2mo ago
> you can't even reuse the same architecture for training and inference AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`
- bigyabai 2mo ago
  I stand corrected, only Google's post-Ironwood TPUs have the split as well. Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
acdha 2mo ago
> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today. How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized. My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
- bigyabai 2mo ago
  Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won. Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.