3 ms·
Somebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?
by oezi 2y ago
Somebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?
- Zambyte 2y agoFWIW I'm getting 29 TPS on Ollama on my 7900 XTX with the 27b qat. You can't really compare inference engine to inference engine without keeping the hardware and model fixed. Unfortunately Ollama and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856 https://github.com/vllm-project/vllm/issues/16856