2 ms·

Not supporting 8 bit quantizations is not just some minor optimization missing. An 8 bit quantization of a model is half the size of a 16 bit model. You can loa

Not supporting 8 bit quantizations is not just some minor optimization missing. An 8 bit quantization of a model is half the size of a 16 bit model. You can load a model with twice the number of parameters using 8 bit quantizations as compared to loading a 16 bit model. GPU ram is always in demand and since 8 bit quants give you <1% performance degredation re: perplexity there's no reason not to. That's before getting to any potential 8 vs 16 bit processing speed differences the two companies are arguing over here.

andersa 3y ago
Both GPUs support both formats, so comparing an 8 bit quantized model on one to a 16 bit model on another is nonsense. That'll massively affect the result.
brucethemoose2 3y ago
Its more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retrieval. Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.
Tuna-Fish 3y ago
The hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inference library. So for now at least the market disagrees with you on the relative merits of 8 vs 16 bit processing. In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.
dhruvdh 3y ago
It should also be noted that H100 has 80 GB VRAM and MI300X has 192 GB. So, it can happily load the largest 8 bit model H100 can in 16 bit. Also, like others noted, vLLM is behind on FP8 support, its not a hardware issue.

andersa 3y ago

Both GPUs support both formats, so comparing an 8 bit quantized model on one to a 16 bit model on another is nonsense. That'll massively affect the result.

brucethemoose2 3y ago

Its more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retrieval. Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.

Tuna-Fish 3y ago

The hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inference library. So for now at least the market disagrees with you on the relative merits of 8 vs 16 bit processing. In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.

dhruvdh 3y ago

It should also be noted that H100 has 80 GB VRAM and MI300X has 192 GB. So, it can happily load the largest 8 bit model H100 can in 16 bit. Also, like others noted, vLLM is behind on FP8 support, its not a hardware issue.