2 ms·

Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others, and they use more tokens per task, in part thanks to that xhigh default. On

• petu 2mo ago
  > Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others No? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.
  • stymaar 2mo ago
    There's no Qwen3.8-35B-A3B though.
    • hadlock 2mo ago
      I benched Qwen 3.6 35B-A3B against Qwen 3.8 27B with the same parameters, thinking set to low. Despite 35B having 9x fewer active parameters, it benched only 2.34x slower. The 35B got only 50% more agentic tasks done per hour.
  • trouve_search 2mo ago
    What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods). Output TPS in vllm for instance: - Gemma4 26B-A4B: 200-300TPS - Qwen3.6 35B-A3B: 120-180TPS - Gemma4 31B: 80-120TPS - Qwen3.6 27B: 60-80TPS This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
    • petu 2mo ago
      Single 3090 under llama.cpp: | model | size | test | t/s | | ------------------- | ------- | ------ | ---- | | gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 | | gemma4 31B Q4_0 | 16.1 GB | tg512 | 40 | | qwen35 27B Q4_K | 15.9 GB | pp2048 | 1248 | | qwen35 27B Q4_K | 15.9 GB | tg512 | 39 | | gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 | | gemma4 26B.A4B Q4_0 | 13.3 GB | tg512 | 160 | | qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 | | qwen35 35B.A3B Q3_K | 15.7 GB | tg512 | 144 | > with their respective speculative decoding methods You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
      • trouve_search 2mo ago
        I think it's a vllm vs llama_cpp performance thing, will pay more into it. One note I had between the two is that gemma has a much higher prefix cache hit rate in general.
    • xfalcox 2mo ago
      Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?
      • trouve_search 2mo ago
        Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both. I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding). Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
    • mirekrusin 2mo ago
      Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc): ./llama.cpp/llama-server \ -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \ --webui-mcp-proxy \ --no-mmproj \ --parallel 1 \ --kv-unified \ --flash-attn on \ --fit off \ --split-mode tensor \ -ngl 999 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -ub 256 \ --no-context-shift \ --host 0.0.0.0 \ --tools all \ --jinja \ --ctx-size 262144 \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --reasoning on \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --reasoning-preserve \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 Use claude/codex/whatever with /goal to optimize params for you. IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
      • trouve_search 2mo ago
        thanks for posting your setup! I think it's smart to set the reasoning effort default to something saner in the base config. Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out: ``` PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ vllm serve Qwen/Qwen3.6-27B-FP8 \ --dtype auto \ --kv-cache-dtype fp8 \ --enable-chunked-prefill \ --enable-prefix-caching \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \ --default-chat-template-kwargs '{ "enable_thinking": true, "reasoning_effort":"medium" }' \ --tensor-parallel-size 2 \ --max-model-len 250000 \ --gpu-memory-utilization 0.9 \ --max-num-batched 12000 \ --max-num-seqs 24 ``` I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
        • mirekrusin 2mo ago