2 ms·
vLLM tested with Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and Qwen3.5-27B. I'm wondering if it needs to be tested with
by Tepix 2mo ago
vLLM tested with Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and Qwen3.5-27B.
I'm wondering if it needs to be tested with every other model or not.
- scrlk 2mo agoIMO, yes. For example, Qwen 3.x is insensitive to weight and KV cache quantisation, whereas Gemma 4 is more sensitive: https://localbench.substack.com/p/kv-cache-quantization-benchmark https://localbench.substack.com/p/kv-cache-quantization-benc...