3 ms·
The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second
by titanomachy 4mo ago
The bandwidth argument is compelling, do we have benchmarks for these models? I’m curious what it translates to in tokens per second
- mips_avatar 4mo agoI benchmarked mine for a deep research workload I was running. Concurrency 1 is the speed you'd get if you're chatting with one agent, 2x3090 (has an nvlink bridge though it didn't seem to matter hugely for inference) Qwen 3.6 27b int4: Concurrency 1: 68 tok/s output Concurrency 32: 363 tok/s output Prompt processing speed: 1520 tok/s Qwen 3.6 35ba3b int4: Concurrency 1: 150 tok/s output Concurrency 32: 1083 tok/s output Prompt processing speed: 4324 tok/s Macbook Pro m3 36gb RAM: Qwen 3.6 27b int4: Concurrency 1: 18 tok/s output didn't measure the other metrics and it was a slightly different benchmark.
- titanomachy 4mo agoYeah that’s a huge difference. I don’t think I want to interactively use any model with 18 tps.