3 ms·
I think the Mac Studio is a poor fit for gpt-oss-120b. On my 96 GB DDR5-6000 + RTX 5090 box, I see ~20s prefill latency for a 65k prompt and ~40 tok/s decode,
by EnPissant 1y ago
I think the Mac Studio is a poor fit for gpt-oss-120b.
On my 96 GB DDR5-6000 + RTX 5090 box, I see ~20s prefill latency for a 65k prompt and ~40 tok/s decode, even with most experts on the CPU.
A Mac Studio will decode faster than that, but prefill will be 10s of times slower due to much lower raw compute vs a high-end GPU. For long prompts that can make it effectively unusable. That’s what the parent was getting at. You will hit this long before 65k context.
If you have time, could you share numbers for something like:
llama-bench -m <path-to-gpt-oss-120b.gguf> -ngl 999 -fa 1 --mmap 0 -p 65536 -b 4096 -ub 4096
Edit: The only Mac Studio pp65536 datapoint I’ve found is this Reddit thread:
https://old.reddit.com/r/LocalLLaMA/comments/1jq13ik/mac_stu https://old.reddit.com/r/LocalLLaMA/comments/1jq13ik/mac_stu
...
They report ~43.2 minutes prefill latency for a 65k prompt on a 2-bit DeepSeek quant. Gpt-oss-120b should be faster than that, but still very slow.
- int_19h 1y agoThis is Mac Studio M1 Ultra with 128Gb of RAM. > llama-bench -m ./gpt-oss-120b-MXFP4-00001-of-00002.gguf -ngl 999 -fa 1 --mmap 0 -p 65536 -b 4096 -ub 4096 | model | size | params | backend | threads | n_batch | n_ubatch | fa | mmap | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -: | ---: | --------------: | -------------------: | | gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | Metal,BLAS | 16 | 4096 | 4096 | 1 | 0 | pp65536 | 392.37 ± 43.91 | | gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | Metal,BLAS | 16 | 4096 | 4096 | 1 | 0 | tg128 | 65.47 ± 0.08 | build: a0e13dcb (6470)
- EnPissant 1y agoThanks. That’s better than I expected. It's only 8.3x worse than a 5090 + CPU: 167s latency.