16 ms·
AMD MI300X 30% higher performance than Nvidia H100, even with optimized stack
- superkuh 3y agoNot supporting 8 bit quantizations is not just some minor optimization missing. An 8 bit quantization of a model is half the size of a 16 bit model. You can load a model with twice the number of parameters using 8 bit quantizations as compared to loading a 16 bit model. GPU ram is always in demand and since 8 bit quants give you <1% performance degredation re: perplexity there's no reason not to. That's before getting to any potential 8 vs 16 bit processing speed differences the two companies are arguing over here.
- andersa 3y agoBoth GPUs support both formats, so comparing an 8 bit quantized model on one to a 16 bit model on another is nonsense. That'll massively affect the result.
- brucethemoose2 3y agoIts more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retrieval. Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.
- Tuna-Fish 3y agoThe hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inference library. So for now at least the market disagrees with you on the relative merits of 8 vs 16 bit processing. In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.
- dhruvdh 3y agoIt should also be noted that H100 has 80 GB VRAM and MI300X has 192 GB. So, it can happily load the largest 8 bit model H100 can in 16 bit. Also, like others noted, vLLM is behind on FP8 support, its not a hardware issue.
- whalesalad 3y agonvidia is winning with cuda, not hardware. as soon as AMD can crack the rocm/cross compile nut they will be on top. dominating server market already. and in pure raster perf, they’re dominating the graphics space as well.
- gumby 3y agoI though usage of PyTorch >> cuda directly, which case cuda is less crucial.
- godelski 3y agoPyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.
- anvuong 3y agoIt's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
- godelski 3y agoFor sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...) Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software. I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
- mugivarra69 3y agogreat stuff, just need to beat h200
- ChrisArchitect 3y agoAMD's Response to Nvidia regarding H100 vs. MI300X on real world workloads https://news.ycombinator.com/item?id=38669440 https://news.ycombinator.com/item?id=38669440
- gdiamos 3y agoIf anyone wants access to mi250 or mi300 today, contact us here: https://www.lamini.ai/contact https://www.lamini.ai/contact If you are a LLM startup blocked on compute, send me a dm https://www.linkedin.com/in/gregory-diamos-1a8b9083 https://www.linkedin.com/in/gregory-diamos-1a8b9083 https://twitter.com/gregorydiamos https://twitter.com/gregorydiamos
- pavelstoev 3y agoLooking forward to MLPerf results https://mlcommons.org/benchmarks/ https://mlcommons.org/benchmarks/
- adricl 3y ago^This
- st380752143 3y agoWhen is it updated?
- duchenne 3y agoIs this released yet? Where can I buy or rent some? Even the previous version?
- beebeepka 3y agoThey announced availability last week.
- pk-protect-ai 3y agoCan I run torch nightly builds on it?
- netika 3y agoWhat about multi-gpu setups? Nvidia has nvlink, effectively combining any amount of GPUs into one speedy network, which can benefit both training and inference speed for bigger model sizes. AFAIK, AMD doesn't have anything like that, thus, limiting the usefullness of these chips. Also, compute stack is too rough right now. Not sure ROCm will be comparable with CUDA in terms of usability in the next couple of years at least.
- genAIdea 3y agoCould someone explain why ROCm is so behind Cuda and why wouldnt big tech and startups adopt MI300X instead of waiting in line begging Jensen for H100?
- rreichman 3y agoNvidia has been working on CUDA for 16 years and for the past few years building ML use cases on CUDA. AMD is late in the game with ROCm. That said, I think the future is bright for AMD. The MI300X looks great and they're spending a lot on making ROCm better and fully PyTorch compatible, which will automatically make all of the PyTorch-based software AMD ready.