2 ms·
What sort of pp/tg speed do you get on a Strix Halo?
by puzzlingcaptcha 1mo ago
What sort of pp/tg speed do you get on a Strix Halo?
- cpburns2009 1mo agoThis is the best I got, all with Unsloth's quantizations. Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s Laguna actually performed better than I remembered. I thought it was slower.
- deleted 1mo ago[deleted]
- htrp 1mo agoHave you benchmarked against full precision models for accuracy/ performance?
- cpburns2009 1mo agoNot full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere. Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6. Qwen3.6-27B Q3 Q6 ARC-Challenge 97.0 97.0 BIG-Bench Hard 57.9 59.3 GPQA Diamond 77.8 83.3 GSM8K 92.4 92.6 Hendrycks Math 35.5 38.9 HumanEval 80.5 85.4 HumanEval+ 75.0 79.3 IFEval 87.3 88.0 MBPP 75.2 77.2 MBPP+ 88.4 88.9 MMLU-Pro 83.1 83.5
- ascii0eks84 1mo agoWhat are pp/tg? I get 30t/s on 27B qwen.
- throwawayffffas 1mo agopp is prompt processing how fast it processes the prompt. Tg is token generation how fast, it generates tokens.
- ascii0eks84 1mo agookay, so prefill and decode would be the terms I was already familiar with.