10 ms·
I benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent, 2x3090 (has an nvlink br
by mips_avatar 3mo ago
I benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent,
2x3090 (has an nvlink bridge though it didn't seem to matter hugely for inference)
Qwen 3.6 27b int4:
Concurrency 1: 68 tok/s output
Concurrency 32: 363 tok/s output
Prompt processing speed: 1520 tok/s
Qwen 3.6 35ba3b int4:
Concurrency 1: 150 tok/s output
Concurrency 32: 1083 tok/s output
Prompt processing speed: 4324 tok/s
Macbook Pro m3 36gb RAM:
Qwen 3.6 27b int4:
Concurrency 1: 18 tok/s output didn't measure the other metrics and it was a slightly different benchmark.
- titanomachy 3mo agoYeah that’s a huge difference. I don’t think I want to interactively use any model with 18 tps.