4 ms·

Nvidia can actually charge larger margins if inference compute goes down. It would enable them to manufacture more units of smaller GPUs using inferior and chea

Nvidia can actually charge larger margins if inference compute goes down. It would enable them to manufacture more units of smaller GPUs using inferior and cheaper silicon, all of which would increase the profits per unit sold as well as the number of units they can manufacture. The industry has to find a way to separate itself from Nvidia's GPGPU technology if they want to stop being gouged. The issue is that nobody, not Apple, not AMD, not Intel, has been treating Nvidia's hardware as a serious threat.

logicchains 2y ago
>The issue is that nobody, not Apple, not AMD, not Intel, has been treating Nvidia's hardware as a serious threat Google has and they've built a much more cost efficient (for them) system: the TPU. They even rent them out, and in terms of cost per unit compute TPUs are significantly cheaper than renting GPUs from the big cloud providers. Amazon's also tried to do something similar with Trainium chips, however they're usefulness is more limited due to software issues (Amazon's much weaker at compiler development than Google, so Trainium software is quite slow and buggy).
vidarh 2y ago
For inference Nvidia has more significant competition than for training. See Groq, Google's TPU's etc.
- pants2 2y ago
  People talk about Groq and Cerberus as competitors but it seems to me their manufacturing process makes the availability of those chips extremely limited. You can call up Nvidia and order $10B worth of GPUs and have them delivered the next week. Can't say the same for these specialty competitors.
  - moralestapia 2y ago
    You mean Cerebras. >call up Nvidia and order $10B worth of GPUs Doubt it. No idea about Groq, but Cerebras might give you a similar timeline than nVidia. Each of their wafers are 50x-100x H100s so they need to make less of them, in absolute units. But cooling, power, etc... nVidia might have an advantage as their ecosystem is huge and more "liquid" in a sense.
  - vidarh 2y ago
    > You can call up Nvidia and order $10B worth of GPUs and have them delivered the next week Nvidia sold $14.5 billion of datacenter hardware in the third quarter of their fiscal 2024 and that led to severe supply constraints, with estimate lead times for H100's up to 52 weeks some places, so no you can't, as that $14.5 billion was clearly capped by their ability to supply, not demand. You're right, though, that Groq etc. can't deliver anywhere near the same volume now, but there's little reason to believe that will continue. There's no need for full GPU's for inference only workloads, so competitors can enter the space with a tiny proportion of functionality.
    - boroboro4 2y ago
      Groq chips have 230 mb of sram memory. Good luck running 670B model on those chips, even without supply constraints.
      - vidarh 2y ago
        Nvidia's H100 has 80GB. As long the interconnect is fast enough you don't need everything to fit on one.
      - baq 2y ago
        Their architecture means you buy them by the rack. Individual chips are useless, the magic happens when you set them up so each chip handles a subset of the model. IOW, do you think groq’s 70B models run on 230MB of sram?
        boroboro4 2y ago
        I didn’t say the model gonna run on one chip of course. 70B needs ~300 chips (only for weights, fp8, just like they do, key value cache not included), 670B would need ~3000 chips, and in racks or not it’s very hard to set up such cluster for one model. There are reasons they still don’t have Llama 405B model.
        vidarh 2y ago
- panabee 2y ago
  Nvidia (NVDA) generates revenue with hardware, but digs moats with software. The CUDA moat is widely unappreciated and misunderstood. Dethroning Nvidia demands more than SOTA hardware. OpenAI, Meta, Google, AWS, AMD, and others have long failed to eliminate the Nvidia tax. Without diving into the gory details, the simple proof is that billions were spent on inference last year by some of the most sophisticated technology companies in the world. They had the talent and the incentive to migrate, but didn't. In particular, OpenAI spent $4 billion, 33% more than on training, yet still ran on NVDA. Google owns leading chips and leading models, and could offer the tech talent to facilitate migrations, yet still cannot cross the CUDA moat and convince many inference customers to switch. People are desperate to quit their NVDA-tine addiction, but they can't for now. [Edited to include Google, even though Google owns the chips and the models; h/t @onlyrealcuzzo]
  - vidarh 2y ago
    The CUDA moat is largely irrelevant for inference. The code needed for inference is small enough that there are e.g. bare-metal CPU only implementations. That isn't what's limiting people from moving fully off Nvidia for inference. And you'll note almost "everyone" in this game are in the process of developing their own chips.
  - onlyrealcuzzo 2y ago
    > OpenAI, Meta, AWS, AMD, and others have long attempted to eliminate the Nvidia tax, yet failed. Gemini / Google runs and trains on TPUs. You have no incentive to infer on AMD if you need to buy a massive Nvidia cluster to train.
    - panabee 2y ago
      Google was omitted because they own the hardware and the models, but in retrospect, they represent a proof point nearly as compelling as OpenAI. Thanks for the comment. Google has leading models operating on leading hardware, backed by sophisticated tech talent who could facilitate migrations, yet Google still cannot leap over the CUDA moat and capture meaningful inference market share. Yes, training plays a crucial role. This is where companies get shoehorned into the CUDA ecosystem, but if CUDA were not so intertwined with performance and reliability, customers could theoretically switch after training.
ClumsyPilot 2y ago
I think future of inference is on the client side You can do inference on almost any hardware, I do not see any edge for NVIDIA here I can download DeepSeek 30b model and run inference at good speed on AMD GPU ms and even on CPU. Apple silicon works fine too. I get >50 tokens/s on £300 AMD GPUs. The main bottleneck appears to be memory, not processing power.
- mrbungie 2y ago
  In fact, I'm not sure how the "we will need tons of centralized inference infrastructure" argument works when Apple with +50% smartphone market share in the USA has a totally opposite strategy focused on privacy: on-device inference.
  - gajjanag 2y ago
    This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to process a batch of tokens from multiple users/requests than processing them from a single user's request on device.
  - talldayo 2y ago
    Apple's strategy already failed. Their big bet on NPU hardware did not pay off at all, and right now it's effectively wasted silicon on every iDevice while the GPU does all the heavy inference work. Now they partner with OpenAI to handle their inference (and even that's not good enough in many cases[0]). The "centralized compute" lobby is being paid by Apple to do the work their devices cannot. Until Apple or AMD unifies their GPU architectures and implements complex streaming multiprocessors, Nvidia will remain in a class of their own. Apple used to lead the charge on the foremost CUDA alternative too, but then they abandoned it to focus on proprietary standards instead. It's pretty easy to argue that Apple shot themselves in the foot with every opportunity they had to compete on good faith. And make no mistake: Apple could have competed with Nvidia if they weren't so stubborn about Linux support and putting smartphone GPUs in laptops and desktops. [0] https://apnews.com/article/apple-ai-news-hallucinations-iphone-6b37a11b9cdd0e100c299e922d58b530 https://apnews.com/article/apple-ai-news-hallucinations-ipho...
buyucu 2y ago
both llama.cpp and vllm support inference with rocm or vulkan. inference is the easiest thing to decouple from nvidia.

logicchains 2y ago

>The issue is that nobody, not Apple, not AMD, not Intel, has been treating Nvidia's hardware as a serious threat Google has and they've built a much more cost efficient (for them) system: the TPU. They even rent them out, and in terms of cost per unit compute TPUs are significantly cheaper than renting GPUs from the big cloud providers. Amazon's also tried to do something similar with Trainium chips, however they're usefulness is more limited due to software issues (Amazon's much weaker at compiler development than Google, so Trainium software is quite slow and buggy).

vidarh 2y ago

For inference Nvidia has more significant competition than for training. See Groq, Google's TPU's etc.

pants2 2y ago
People talk about Groq and Cerberus as competitors but it seems to me their manufacturing process makes the availability of those chips extremely limited. You can call up Nvidia and order $10B worth of GPUs and have them delivered the next week. Can't say the same for these specialty competitors.
- moralestapia 2y ago
  You mean Cerebras. >call up Nvidia and order $10B worth of GPUs Doubt it. No idea about Groq, but Cerebras might give you a similar timeline than nVidia. Each of their wafers are 50x-100x H100s so they need to make less of them, in absolute units. But cooling, power, etc... nVidia might have an advantage as their ecosystem is huge and more "liquid" in a sense.
- vidarh 2y ago
  > You can call up Nvidia and order $10B worth of GPUs and have them delivered the next week Nvidia sold $14.5 billion of datacenter hardware in the third quarter of their fiscal 2024 and that led to severe supply constraints, with estimate lead times for H100's up to 52 weeks some places, so no you can't, as that $14.5 billion was clearly capped by their ability to supply, not demand. You're right, though, that Groq etc. can't deliver anywhere near the same volume now, but there's little reason to believe that will continue. There's no need for full GPU's for inference only workloads, so competitors can enter the space with a tiny proportion of functionality.
  - boroboro4 2y ago
    Groq chips have 230 mb of sram memory. Good luck running 670B model on those chips, even without supply constraints.
    - vidarh 2y ago
      Nvidia's H100 has 80GB. As long the interconnect is fast enough you don't need everything to fit on one.
    - baq 2y ago
      Their architecture means you buy them by the rack. Individual chips are useless, the magic happens when you set them up so each chip handles a subset of the model. IOW, do you think groq’s 70B models run on 230MB of sram?
      - boroboro4 2y ago
        I didn’t say the model gonna run on one chip of course. 70B needs ~300 chips (only for weights, fp8, just like they do, key value cache not included), 670B would need ~3000 chips, and in racks or not it’s very hard to set up such cluster for one model. There are reasons they still don’t have Llama 405B model.
        vidarh 2y ago
panabee 2y ago
Nvidia (NVDA) generates revenue with hardware, but digs moats with software. The CUDA moat is widely unappreciated and misunderstood. Dethroning Nvidia demands more than SOTA hardware. OpenAI, Meta, Google, AWS, AMD, and others have long failed to eliminate the Nvidia tax. Without diving into the gory details, the simple proof is that billions were spent on inference last year by some of the most sophisticated technology companies in the world. They had the talent and the incentive to migrate, but didn't. In particular, OpenAI spent $4 billion, 33% more than on training, yet still ran on NVDA. Google owns leading chips and leading models, and could offer the tech talent to facilitate migrations, yet still cannot cross the CUDA moat and convince many inference customers to switch. People are desperate to quit their NVDA-tine addiction, but they can't for now. [Edited to include Google, even though Google owns the chips and the models; h/t @onlyrealcuzzo]
- vidarh 2y ago
  The CUDA moat is largely irrelevant for inference. The code needed for inference is small enough that there are e.g. bare-metal CPU only implementations. That isn't what's limiting people from moving fully off Nvidia for inference. And you'll note almost "everyone" in this game are in the process of developing their own chips.
- onlyrealcuzzo 2y ago
  > OpenAI, Meta, AWS, AMD, and others have long attempted to eliminate the Nvidia tax, yet failed. Gemini / Google runs and trains on TPUs. You have no incentive to infer on AMD if you need to buy a massive Nvidia cluster to train.
  - panabee 2y ago
    Google was omitted because they own the hardware and the models, but in retrospect, they represent a proof point nearly as compelling as OpenAI. Thanks for the comment. Google has leading models operating on leading hardware, backed by sophisticated tech talent who could facilitate migrations, yet Google still cannot leap over the CUDA moat and capture meaningful inference market share. Yes, training plays a crucial role. This is where companies get shoehorned into the CUDA ecosystem, but if CUDA were not so intertwined with performance and reliability, customers could theoretically switch after training.

ClumsyPilot 2y ago

I think future of inference is on the client side You can do inference on almost any hardware, I do not see any edge for NVIDIA here I can download DeepSeek 30b model and run inference at good speed on AMD GPU ms and even on CPU. Apple silicon works fine too. I get >50 tokens/s on £300 AMD GPUs. The main bottleneck appears to be memory, not processing power.

mrbungie 2y ago
In fact, I'm not sure how the "we will need tons of centralized inference infrastructure" argument works when Apple with +50% smartphone market share in the USA has a totally opposite strategy focused on privacy: on-device inference.
- gajjanag 2y ago
  This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to process a batch of tokens from multiple users/requests than processing them from a single user's request on device.
- talldayo 2y ago
  Apple's strategy already failed. Their big bet on NPU hardware did not pay off at all, and right now it's effectively wasted silicon on every iDevice while the GPU does all the heavy inference work. Now they partner with OpenAI to handle their inference (and even that's not good enough in many cases[0]). The "centralized compute" lobby is being paid by Apple to do the work their devices cannot. Until Apple or AMD unifies their GPU architectures and implements complex streaming multiprocessors, Nvidia will remain in a class of their own. Apple used to lead the charge on the foremost CUDA alternative too, but then they abandoned it to focus on proprietary standards instead. It's pretty easy to argue that Apple shot themselves in the foot with every opportunity they had to compete on good faith. And make no mistake: Apple could have competed with Nvidia if they weren't so stubborn about Linux support and putting smartphone GPUs in laptops and desktops. [0] https://apnews.com/article/apple-ai-news-hallucinations-iphone-6b37a11b9cdd0e100c299e922d58b530 https://apnews.com/article/apple-ai-news-hallucinations-ipho...

buyucu 2y ago

both llama.cpp and vllm support inference with rocm or vulkan. inference is the easiest thing to decouple from nvidia.