It would be great to have real world inference benchmarks for LLMs. These aren't it. That means e.g. 8xH100 with TensorRT-LLM / vLLM vs 8xMI300X with vLLM runn

It would be great to have real world inference benchmarks for LLMs. These aren't it. That means e.g. 8xH100 with TensorRT-LLM / vLLM vs 8xMI300X with vLLM running many concurrent requests with reasonable # of input and output tokens. Ran both in fp8 and fp16. Most of the benchmarks I've seen had setups that no one would use in production. For example running on a single MI300X or 2xH100 -- this will likely be memory bound, you need to go to higher batch sizes (more VRAM) to be compute bound to properly utilize these. Or benchmarking requests with unrealistically low # of input tokens.