5 ms·

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I

by amanzi 25d ago

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

argee 25d ago
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
- dolebirchwood 24d ago
  > enough for my needs Which are...?
  - argee 24d ago
    Some examples (keep in mind this is all indefinitely free for me, no burning quota away): 1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case. 2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode. 3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text. 4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word). 5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on. Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit. Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
- phazonoverload 24d ago
  My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
  - argee 24d ago
    Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini. I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these: https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10ts_slowlier_than_lm_studio_qwen36_35ba3_on/ https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
- dd8601fn 24d ago
  I have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn. What are you using them with/for?
  - argee 24d ago
    I do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
- pwthornton 23d ago
  What kinds of tasks are you using this for?
pcarolan 25d ago
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
- Gigachad 25d ago
  It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
  - api 25d ago
    There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.
    - mrngld 24d ago
      Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity. That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline. On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden. And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there. Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.
hkchad 25d ago
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
- deleted 24d ago
  [deleted]
- brettdav 24d ago
  Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
  - hkchad 24d ago
    What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
    - brettdav 22d ago
      Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.
ericd 25d ago
I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber. That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.
- gruez 25d ago
  >I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks https://unsloth.ai/docs/models/qwen3.8#benchmarks
  - ericd 25d ago
    Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though. A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
    - bel8 24d ago
      > We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway. I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.
whatsThisBtn4 25d ago
I can't imagine using CPU... Oh I did twice. If you are work from home and do dishes between prompts you can get a gpt3-like result. I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.
- ramgine 25d ago
  With which model. I have a 3060 with a bunch of system ram
  - whatsThisBtn4 24d ago
    Old school Berkeley Sterling or an abliterared model.
- pjmlp 24d ago
  That is a lot of dishes.
cdnsteve 25d ago
These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=coding https://tokenstead.ai/find/results?hardware_id=23&use_case=c...
- amanzi 24d ago
  That's useful thanks. Also, looks painfully slow!
taylorhou 24d ago
i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram. granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
- aa-jv 24d ago
  What sort of business can you run with this setup?
- Normal_gaussian 24d ago
  $0 per-token bill You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity. Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
  - icedchai 24d ago
    Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
- icedchai 24d ago
madduci 24d ago
Are you using the right configuration for your own CPU? On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.
traceroute66 24d ago
> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible With all due respect, I'm not clear why you are so surprised ? By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ? Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
- AdamN 24d ago
  That's their point
- amanzi 24d ago
  Not surprised at all - just making the point that getting a model to run isn't that impressive if it runs at a few tokens per second.
c16 24d ago
Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.
visarga 24d ago
I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.
phazonoverload 24d ago
I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.

by amanzi 25d ago

argee 25d ago

I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.

dolebirchwood 24d ago
> enough for my needs Which are...?
- argee 24d ago
  Some examples (keep in mind this is all indefinitely free for me, no burning quota away): 1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case. 2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode. 3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text. 4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word). 5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on. Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit. Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.
phazonoverload 24d ago
My perf sucks compared to yours. Added it to the post - same model averages 325 tok/s in processing prompts, and 34 tok/s in token generation. What am I doing wrong..?
- argee 24d ago
  Wow, that's just about half the perf. I'm not sure what you're doing differently, though our hardware is a bit different: I am on a Macbook Pro M4 Pro, while you're on a Mac Mini. I would try a different version of the model from HuggingFace while ensuring it's MLX. I'm also using LM Studio, not oMLX, and I've seen some threads like these: https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10ts_slowlier_than_lm_studio_qwen36_35ba3_on/ https://www.reddit.com/r/LocalLLaMA/comments/1spuwir/omlx_10...
dd8601fn 24d ago
I have a (now discontinued) 64gb mini pro and I’ve found the same qwen model to be almost unusable unless I kill Thinking on each turn. What are you using them with/for?
- argee 24d ago
  I do turn thinking off most of the time for both models. I made a separate comment detailing my use cases.
pwthornton 23d ago
What kinds of tasks are you using this for?

pcarolan 25d ago

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

Gigachad 25d ago
It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.
- api 25d ago
  There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.
  - mrngld 24d ago
    Here's the thing that's a little different about data centers; we can tell from Anthropic and OpenAI that they're capacity constrained. Inference demand is there. I notice Cerebras doesn't offer much directly any more, all their capacity is getting completely sucked up by B2B sales. Grok did overbuild, but Anthropic was so desperate for more compute they ate their pride and leased the excess capacity. That means all these data centers are being heavily utilized by actual end user inference demand. Well, some is research on new models, but a lot is actual end user demand. No one has given an explanation of why peoples usage would decline. On top of that, margin on inference appears to be decent. It's model training that's a serious financial burden. And maybe that's where there will be a slowdown, maybe the market doesn't justify spending as much on R&D as it does, but the end demand for inference is there. Does that justify these stock prices? That's a different question. But the housing boom left behind endless rows of empty homes because demand disappeared. The 'dot com' boom left behind thousands of miles of dark fiber that'd been built out well ahead of demand for bandwidth. I can see the stock market having a giant sell off, but I don't see data centers sitting idle in that same fashion.

hkchad 25d ago

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).

deleted 24d ago
[deleted]
brettdav 24d ago
Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
- hkchad 24d ago
  What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.
  - brettdav 22d ago
    Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.

ericd 25d ago

I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber. That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.

gruez 25d ago
>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks https://unsloth.ai/docs/models/qwen3.8#benchmarks
- ericd 25d ago
  Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though. A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.
  - bel8 24d ago
    > We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway. I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.

whatsThisBtn4 25d ago

I can't imagine using CPU... Oh I did twice. If you are work from home and do dishes between prompts you can get a gpt3-like result. I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.

ramgine 25d ago
With which model. I have a 3060 with a bunch of system ram
- whatsThisBtn4 24d ago
  Old school Berkeley Sterling or an abliterared model.
pjmlp 24d ago
That is a lot of dishes.

cdnsteve 25d ago

These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=coding https://tokenstead.ai/find/results?hardware_id=23&use_case=c...

amanzi 24d ago
That's useful thanks. Also, looks painfully slow!

taylorhou 24d ago

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram. granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.

aa-jv 24d ago
What sort of business can you run with this setup?
Normal_gaussian 24d ago
$0 per-token bill You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity. Hosted GLM 5.3 flash is like $0.15/mtok in $0.50/mtok out
- icedchai 24d ago
  Time to completion also must be considered. If I have to wait around for hours for a prompt to complete locally and I’ll need to iterate quickly, I’m better off hosted than local. If it’s “free” and slow it may just not be worth it.
icedchai 24d ago

madduci 24d ago

Are you using the right configuration for your own CPU? On a Laptop with 32 GB RAM and Iris Xe integrated graphic card, I get between 11-18 Tokens/Second with Qwen 3.8 27B and llama.cpp with sysl Intel optimisations. Same results with the vulkan back end, although sometimes it ends in weird segmentation faults due to the memory consumption.

traceroute66 24d ago

> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible With all due respect, I'm not clear why you are so surprised ? By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ? Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.

AdamN 24d ago
That's their point
amanzi 24d ago
Not surprised at all - just making the point that getting a model to run isn't that impressive if it runs at a few tokens per second.

c16 24d ago

Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.

visarga 24d ago

I took this thread and summarized it with Qwen3.6-35B-A3B, it had 1400 tps prefix and 60 tps completion. Very good performance. Using oMLX on MacBook M5 Pro 64GB.

phazonoverload 24d ago

I'm the author - hello! Added to the post! Qwen averages 325 tok/s in processing prompts, and 34 tok/s in token generation. That isn't instant, but it's quick enough that I never really think about it.