3 ms·

Cool! I'm thinking about a local set up. What's your usual tokens/second rate?

victords 1mo ago
Not OP, but I’m running local models on a M1 Max as well with 64GB RAM. It varies by model, but I’m getting 50-60 t/s with Qwen 3.6 35B and Qwen 3 coder 30B. I’ve also used Qwen 3.8 27B but I get 10t/s on it. It’s useable in some use cases, but I rely mostly on my $20 Claude subscription.
- copperx 1mo ago
  That's so cool. I wonder if the regular M5 can run those models too.
  - darthcircuit 1mo ago
    I run qwen 3.8 27b on my m5 mbp, with 48gb of unified ram and I’m getting around 10-15 tok/s. 3.6 35b a3b, I’m getting upwards of 100
- spider-mario 1mo ago
  Try 3.8 27B in MTPLX; I get about 30 tok/s with the same hardware as you. (Although it does use around 90-95W of power, compared to the ~60W that 3.6 35B-A3B uses to generate 55 tok/s. That’s about 3 J/tok instead of 1.)
solarkraft 1mo ago
NGL: I don’t really have a good way to find out right now. It also doesn’t matter that much because the way the models use the tokes varies a lot. Qwen 3.8 is known for overthinking while Muse Glimmer may be a little slower per token, but it uses them very efficiently, caveman style. Generation speed isn’t the bottleneck anyway, at least on pre M4/M5 devices (the newer chips got significant processing acceleration). It’s prompt processing time. OpenCode’s system prompt can take up to 3 minutes to process, which is why good prompt caching is essential. For that I use omlx, which can persist the KV cache to disk, chunked so you can reuse parts. This helps with the usability a lot, when an agentic session is warm it runs pretty smoothly. New requests can take a couple seconds (sometimes many, which must be fixable somehow). So: It’s not fast, but I also don’t find it awfully slow. My use is typically semi-interactive, for fully interactive use you have to wait a bit, but it’s possible. I personally am still regularly amazed that something even close to this is possible on completely local hardware.

Not OP, but I’m running local models on a M1 Max as well with 64GB RAM. It varies by model, but I’m getting 50-60 t/s with Qwen 3.6 35B and Qwen 3 coder 30B. I’ve also used Qwen 3.8 27B but I get 10t/s on it. It’s useable in some use cases, but I rely mostly on my $20 Claude subscription.

copperx 1mo ago
That's so cool. I wonder if the regular M5 can run those models too.
- darthcircuit 1mo ago
  I run qwen 3.8 27b on my m5 mbp, with 48gb of unified ram and I’m getting around 10-15 tok/s. 3.6 35b a3b, I’m getting upwards of 100
spider-mario 1mo ago
Try 3.8 27B in MTPLX; I get about 30 tok/s with the same hardware as you. (Although it does use around 90-95W of power, compared to the ~60W that 3.6 35B-A3B uses to generate 55 tok/s. That’s about 3 J/tok instead of 1.)

solarkraft 1mo ago

NGL: I don’t really have a good way to find out right now. It also doesn’t matter that much because the way the models use the tokes varies a lot. Qwen 3.8 is known for overthinking while Muse Glimmer may be a little slower per token, but it uses them very efficiently, caveman style. Generation speed isn’t the bottleneck anyway, at least on pre M4/M5 devices (the newer chips got significant processing acceleration). It’s prompt processing time. OpenCode’s system prompt can take up to 3 minutes to process, which is why good prompt caching is essential. For that I use omlx, which can persist the KV cache to disk, chunked so you can reuse parts. This helps with the usability a lot, when an agentic session is warm it runs pretty smoothly. New requests can take a couple seconds (sometimes many, which must be fixable somehow). So: It’s not fast, but I also don’t find it awfully slow. My use is typically semi-interactive, for fully interactive use you have to wait a bit, but it’s possible. I personally am still regularly amazed that something even close to this is possible on completely local hardware.