3 ms·
It has fp8 support. Not sure whether fp8 on MI300x is supported by vLLM yet. Also, many of these comparisons use vLLM for both setups, but for Nvidia you can a
by Palmik 2y ago
It has fp8 support. Not sure whether fp8 on MI300x is supported by vLLM yet.
Also, many of these comparisons use vLLM for both setups, but for Nvidia you can and should use TensorRT-LLM which tends to do quite a bit better than vLLM at high loads.
- deleted 2y ago[deleted]
- latchkey 2y agoElio, the person who did the testing confirmed with me that he has fp8 working.