3 ms·

Testing on a MBP m4 pro 24gb ~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context. The issue is I have yet to find a useful agentic local llm th

Testing on a MBP m4 pro 24gb ~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context. The issue is I have yet to find a useful agentic local llm that I can run on this machine. Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.

aetherspawn 8d ago
Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
- huseyinkeles 8d ago
  I just used their `Bonsai-demo` repo like this; `cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh` then used it in a very minimalistic pi with a very small system prompt. Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
  - yearolinuxdsktp 7d ago
    Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
  - aetherspawn 6d ago
    Yes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.
- piyh 8d ago
  30B is useless on 24 gigs of ram as there's ~4 gigs of ram left for everything else even with unsloth quants
- BoredomIsFun 7d ago
  31B
sean_pedersen 8d ago
Try a MoE model like Qwen3.6 35B-A3B for better tok/s
- huseyinkeles 8d ago
  I tried this one, but I found Ornith1.5 to be a better MoE model for me, also very fast. But I still couldn't make it implement a real task on a real repo :( it only worked with an extremely clear directions and very small tasks.

aetherspawn 8d ago

Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.

huseyinkeles 8d ago
I just used their `Bonsai-demo` repo like this; `cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh` then used it in a very minimalistic pi with a very small system prompt. Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
- yearolinuxdsktp 7d ago
  Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
- aetherspawn 6d ago
  Yes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.
piyh 8d ago
30B is useless on 24 gigs of ram as there's ~4 gigs of ram left for everything else even with unsloth quants
BoredomIsFun 7d ago
31B

sean_pedersen 8d ago

Try a MoE model like Qwen3.6 35B-A3B for better tok/s

huseyinkeles 8d ago
I tried this one, but I found Ornith1.5 to be a better MoE model for me, also very fast. But I still couldn't make it implement a real task on a real repo :( it only worked with an extremely clear directions and very small tasks.