3 ms·
Nvidia is the only one that does this on the sort of scale these AI companies need. They do boxes packed full of AI accelerators with custom high speed bus conn
by PaulKeeble 2y ago
Nvidia is the only one that does this on the sort of scale these AI companies need. They do boxes packed full of AI accelerators with custom high speed bus connectors and high end interconnectivity to make super computers full of GPUs very easy to purchase combined with the software to make it all work. No one else has that sort of integration to the very large that is necessary for the sort of training the top firms are doing.
- ArtTimeInvestor 2y agoYes, that's why I asked about inference. For inference, Nvidia's strengths seem to be not very important. You can do inference on a single GPU. And AFAIK the software stack is not important for inference either. Because you don't have to experiment with the sofware. You just need to get it to run and then you will run it for a long time unchanged. Groq for example runs LLAMA on their custom hardware, correct? And I expect hardware for inference to become a bigger market than hardware for training.
- basiccalendar74 2y agoDeepseek v3 needs 16 H100s or 8 H200 GPUs for inference.
- daveguy 2y agoOr a single 2 processor AMD EPYC rig, for less than $6k. https://xcancel.com/carrigmat/status/1884244369907278106 https://xcancel.com/carrigmat/status/1884244369907278106 The only reason you need all those GPUs is because they only have a fraction of the ram you can cram in a server. With AMD focusing on ram channels and cores the above rig can do 6-8 tokens per second inference. The GPUs will be faster, but the point is inference on the top deepseek model is possible for $6k with an AMD server rig. 8 H200's alone would cost $256,000 and gobble up way more power than the 400 watt envelope of that EPYC rig.
- ArtTimeInvestor 2y agoI thought the whole story about Deepseek was that Deepseek does not have H100s?