4 ms·

Run Qwen3.8 27B locally: real numbers from my Mac Studio

140 points by speckx 2mo ago

Infernal 2mo ago
I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8? Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.
- woadwarrior01 2mo ago
  I suspect this might be due to MTP mispedictions. Either because the 3.8 quantized model weights do not include MTP heads, or as what happened with Ornith-1.5 recently, corrupted MTP heads, or due to a software issue in Ollama.
  - rbanffy 2mo ago
    I remember an article a couple weeks back where someone used an HPE server with two older Xeon E-series CPUs and got reasonable performance by compiling llama with optimal switches for the architecture (memory alignment, page sizes, etc). I was impressed because those Xeons only had AVX2. If you are willing to spend a little more, you can get a slightly newer Xeon with AVX-512 and 4 sockets. I can't find the post though. Edit: a quick search from a server refurbisher nearby gives me a dual Xeon Gold 6330 28-core machine with 512GB and a 16GB V100 GPU. The memory is spread across 32 slots, for about £6,470.
liuliu 2mo ago
The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.
- vunderba 2mo ago
  I’ve never personally used it, but isn’t that what LM Studio is positioned as? From what I can gather, it seems extremely beginner-friendly.
  - liuliu 2mo ago
    What you get with Draw Things: 1. Download the app from Mac AppStore; 2. Download the model; 3. Tap "Try recommended settings", have the guarantee that for whatever model it supports, it is the fastest in Mac ecosystem, no need to fiddle. What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb. To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
    - throw83939303 2mo ago
      Or you can just ask Sol to do all that for you, and do split testing, to find best performance. I would like to see "guarantee" for that app. You get full refound and it is free, right?
    - viccis 2mo ago
      Ollama was/is this but they pulled a heel turn...
    - e28eta 2mo ago
      I’ve casually used Draw Things a couple of times (great tool!), and I did not find it beginner friendly. I remember feeling a lot of doubt, confusion, and frustration. Am I getting these results because I picked the wrong model? Or I need to improve my prompt? Or the tool just can’t do what I’m trying to do? How current are these model recommendations? Have they been superseded by something newer? As a true beginner to AI at the time, even the sizes and bits were meaningless to me. And I don’t remember having any context as to what I should be attempting to run on my mac. So I think you need to include steps 4, 5, and 6 of swapping between different models, quants, and prompts. And step 7 is probably wading through the complicated UI, full of jargon that most people don’t know. Don’t get me wrong, I recognize that it’s a powerful tool, and the steep learning curve exists because it exposes quite a few power-user features. But for someone graduating from commercial AI image generators that take a text prompt and maybe a choice of couple models, it’s not easy.
ActorNightly 2mo ago
> Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on my Mac Studio M3 Ultra >~14 tokens/s For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)
- wolvoleo 2mo ago
  What? Below 100tk/s is worthless? I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace. Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally. And will it remain free? How are they going to make back all those trillions of investment? But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
  - zaj00l 2mo ago
    I wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing. Still, as you said - for day to day, 50-60 tokens / s is a good baseline.
- 0x457 2mo ago
  I get 80 tok/s on the same model, and it's pretty usable. I'm not chatting with it; it's either given a bag of tokens to generate an answer or it's doing some agentic coding. https://openrouter.ai/anthropic/claude-opus-5 https://openrouter.ai/anthropic/claude-opus-5 is it worthless because its 65 tps? re: gemini https://openrouter.ai/google/gemini-3.7-flash https://openrouter.ai/google/gemini-3.7-flash worthless as well? That being said, 14 tok/s is pretty slow.
  - ActorNightly 2mo ago
Atreiden 2mo ago
I'm encountering the same behavior. I've tried 4-8bit quants and get 14-17 tok/s with one run that achieved 19. I'm eagerly awaiting dflash2 support in Unsloth or LM Studio, as allegedly that should increase throughput to around 30tok/s, which is the baseline for what I consider at least somewhat interactive. Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.
- seanmcdirmid 2mo ago
  That isn’t right. Use oMLX or something similar to serve the model with internal MTP enabled. You should get at least 40 tok/sec, but I’m not sure what hardware you are using. Dflash will mess up batching, not really worth it. You need to sift through hugging face for the right model though, and some of the MTP models are meant for rapid MLX and not oMLX.
  - bilbo0s 2mo ago
    Well, yes and no. The author has an M3. Here's reality, MLX on the software layer will not magically place hardware matrix multiplication units in your GPU cores. Newer Macs are always just gonna smoke anything earlier than an M5. Not even sure why this person is trying to get this stuff to run on hardware that wasn't designed for AI? Everyone is pointing him to newer hardware precisely because you need the newer stuff to get models to be performant. You can go with AMD, NVidia or Apple, but you're gonna be using stuff designed well after the M3 if you want to push >100tok/s.
    - refulgentis 2mo ago
      > Not even sure why this person is trying to get this stuff to run on hardware that wasn't designed for AI? Posturing/overclaiming like this shades rather than illuminates, there are no worlds in which the "M3...wasn't designed for AI". My M4 Max 64 GB gets the same speed.
    - pcf 2mo ago
      I would say that an M3 Ultra was designed for the LLMs around that time, plus Apple was pushing hard towards MLX over time.
wolvoleo 2mo ago
These numbers are a lot lower than I expected from such pricey hardware. I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.
- jdright 2mo ago
  any numbers to share? also, what inference engine do you use and with which api?
  - wolvoleo 2mo ago
    I use Llama.cpp and OpenAI-style API of course (who doesn't, unless you use Ollama perhaps?) Ollama sucks, they dropped ROCm support for my card out of the blue, and the Vulkan replacement wasn't ready. The developer just shrugged and didn't care about the people affected so I moved to llama-server. Also, here's some other reasons to avoid ollama: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ And really, ollama are trying to sell their cloud platform and are always running behind llama.cpp in new features like the KV Cache quantisation. So numbers, I don't really measure it from day to day, I'm not really interested in benchmarks at all. But I just did a quick test, llama 3.1 8b q8_0 does about 70tk/s (with Vulkan up from 45 when I was still running ROCm), and qwen 3.5 9b q8_0 does about 45, I think that includes its time spent thinking, not sure. Overall I'm happy with that. Smaller sizes and quants are of course faster than this. It's not terribly much faster than a Mac Studio, but it's one hell of a lot cheaper of course. And for the price of a Mac Studio you can get a lot faster hardware. The only thing where the studio excels is if you need large amounts of vram like 64GB and up and run the models to go with that (I run multiple small ones and spread them over multiple cards)
- fotcorn 2mo ago
  I am also running some old AMD datacenter cards, 2x MI25 in my case. Getting around 30 tokens/second with short context. Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins. ROCm nightlies do actually support these old cards, just not the ROCm stable releases.
kennywinker 2mo ago
This low-effort slop post is misleading. It suggest that 3.8 is half as fast as 3.6, but this is almost definitely because MTP isn't enabled by default. The two models should perform the same - they are for me. When you get AI to think for you, you lose.
mickael-kerjean 2mo ago
Apart from learning, I cant see the point of spending that kind of money and energy to get such awfull token generation speed. Assuming memory bandwith is the bottleneck, is it just a matter of time until we start to see hbm4 based chip able to run qwen3.8 for normal people running at more than 500 token / seconds? Is memory speed the only technological bottlenecks that prevent us from having fast local model?
- wolvoleo 2mo ago
  For me, a HUGE benefit to running local models is that my data stays mine, on my computers only. Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business. With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap. For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams). PS: I wouldn't be surprised if Microsoft runs something similar on their end :) But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.
  - rbanffy 2mo ago
    > my data stays mine, on my computers only. Which is a regulation constraint on many professions, BTW. Some people simply can't give some data to say, ChatGPT, without comprehensive guarantees written in Sam Altman's blood.
rbanffy 2mo ago
Pretty impressive how the Mac ends up less expensive than the Strix Halo boxes, at least here in Ireland. A 128GB Mac Studio with an M5 Max (the Ultra can only have 96 or 256GB) still costs less than the "GMKtec EVO-X2" or the Nvidia DGX Spark with similar performance. Is it the same in the US?
- bronson 2mo ago
  That would be such a sizeable disconnect that you might do some arbitrage. Current prices in USD: 128 GB 1 TB M5 Mac Studio: $5399 128 GB 1 TB GMKtec EVO-X2: $3499 128 GB 1 TB Framework Desktop: $3748
- lowlat 2mo ago
  That's not right. Mac Studio M5 Max 128gb, in Apple Store Ireland, starts from 5,909e (~6,847 USD; tax included). On the other hand, GMKtec Evo-X2 128gb can be currently purchased for 3,299e (~3,822 USD; tax included) from GMKtec online store (free shipping from DE warehouse). It's quite the difference in price.
  - rbanffy 2mo ago
    It’s the price I got from amazon.ie
    - lowlat 2mo ago
      I figured it's dodgy amazon ie listings. DGX Spark/Asus Ascent GX10 prices are inflated there too. They'll also be happy to sell you an RTX 5090 for 10 grand.
kgeist 2mo ago
Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's older Llama build from a "couple weeks ago" failed to run Qwen 3.8 because it did not support Qwen35 architecture, but both 3.6 and 3.8 are based on Qwen35 which was released in February 2026. The post doesn't make any sense.
iagooar 2mo ago
I am also seeing slower speeds, roughly the same ballpark, sometimes even lower - 10-11 tok/s on M5 Max. If there is that ONE version (GGUF or MLX) that runs roughly as fast as 3.6 used to run, please let me know. What would be incredible is the 3.8 35B MoE version too, I can run 3.6 with 60 tok/s which is a really, really nice speed.
oceanplexian 2mo ago
Ollama? Running a q1 quant? I don't think the writer knows what the are doing here to be honest. You will get better information cruising r/localllama for about 10 minutes.
- bilbo0s 2mo ago
  There are many, many people who are betraying a lack of facility with these models. Why is anyone trying to run these on an M3? I thought it was common knowledge that, if you insist on using Apple, only M5 processors and higher have matrix multiplication units in the cores? I've seen the same thing with people buying NVidia cards with 16GB of ram and wondering why they aren't getting 100tok/s? Guys, please, be reasonable. You'll have to get the hardware if you want to run these things fast. If you want to experiment, the slow stuff is fine. But try not to get on HN and ask why your M3 can't get 100tok/s. You're kind of out-ing yourself.
- __mharrison__ 2mo ago
  This is a problem that Hugging Face could easily help solve. Crawling through Reddit or forums to find the right incantation to run a model is frustrating.
sciencesama 2mo ago
AI is a killer adblocker ! imagine connecting it to instagram and curating all the images that you actually care about removing all ads !
- craftkiller 2mo ago
  That seems like an expensive way to block ads considering traditional ad blockers have been doing that with a tiny fraction of the compute for decades.
  - rgovostes 2mo ago
    Have you used a social network in the past year? Facebook, YouTube, Pinterest, et al. have become increasingly hostile to blockers with techniques that break pattern-matching on hostnames and element selectors. It may be inevitable that adblockers move toward visual identification.
    - andrekandre 2mo ago
      > Facebook, YouTube, Pinterest, et al. have become increasingly hostile to blockers maybe at some point we should just stop visiting these sites if they are so hostile... we keep going to them despite all of this and it just serves to delay any kind of alternative getting pickup (not an easy task, i know)
      - simonw 2mo ago
        We go to them because that's where the friends and organizations we care about have chosen to post their content.
refulgentis 2mo ago
Slop; all those LocalLlama threads "you" read had #s too. And Q1 quant? Why? You have the memory...
arjie 2mo ago
The cheapest card that will run this model very well is a ln unlocked CMP 170HX. But you can run it on a 3090. I run it on an old spare A6000 Ampere. I think I wouldn’t use anything lower than 60 tok/s though, which you can get with MTP etc. I just use a full vllm stack but some people see a lot of speed with ninfer (there are non 5090 ports). The large RAM Macs are unusable for inference of dense models as of now. Token generation is too slow.
pwython 2mo ago
Yea Qwen3.8 wasn't fun to use on my 64GB M4 Max either (better than these numbers though), so my new daily driver is Ornith-1.5-35B-A3B-MLX-4bit. I recommend giving that a whirl if you're on similar hardware, it's definitely better than Qwen3.6 35b-a3b which was my go-to before. https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B
- TobTobXX 2mo ago
  Can you tell more about your experience with Ornith? I've come across it, and the benchmarks on its landing page are unbeleavably good. But it isn't featured on more established benchmarks like ArtificialAnalysis and it's not on OpenRouter, so I wrote it off as scam.
- mark_l_watson 2mo ago
  I tried Ornith-1.5-35B-A3B-MLX-4bit on a 32G M2-Pro mac mini with pi and OpenCode. I didn’t get very good results coding in Python, Racket, and TypeScript. I have seen several positive comments like yours so I was probably doing something wrong. I amgetting the new 64G mac mini in 4 weeks, and I made a note to try Ornith-1.5-35B-A3B-MLX-4bit again.
ThrowawayTestr 2mo ago
I'm running the 5 bit quant on a 3060 12G. Get a handful of tokens/s. Prompts take 10-20 minutes to complete but it works.
ig0r0 2mo ago
[flagged]
firemelt 2mo ago
is it really worth to run urself? the watts drsin and that 100% gpu usage
- throwa356262 2mo ago
  Short answer: no Long answer: some dude on youtube calculated that even if electricity was free it would take you 37 years to get back your investment. Slightly longer answer: ... but if you absolutely must have local LLM, a GPU with 24-34 GB VRAM is much cheaper and faster than a Studio
petercooper 2mo ago
FWIW, on my Mac Studio I get ~24-27 tok/s generation between 0-16k context in - that's on the Q6_K GGUF with speculative decoding on. I have spent zero effort optimizing/improving this so far but will be trying the 4 bit MLX next (I've tended to find models drop off somewhat below 6 bit but maybe that isn't the case nowadays).
spwa4 2mo ago
Peaks at over 300 tok/s on my 5090 with Dflash2. At maximum context (252000 or so) still get 60 tok/s. And I don't have n-gram optimization working yet, I only have 64G memory.
Youden 2mo ago
I've been thinking about buying a system to run LLMs locally but the price for one that'll run Qwen3.8-27B well is quite offputting to say the least. What I've been looking at instead is inference providers that use TEE and E2EE to provide cryptographic guarantees that my prompts and responses are only visible to me and the GPU itself. Despite their docs and assurances of what their guarantees mean, I'm having trouble getting to a point where I'm actually comfortable trusting them with secrets though. Phala for example seems to be E2EE only to the gateway and will then forward prompts to (potentially third party) providers. Has anyone been down this path and found a provider they feel safe with?
- lzy 2mo ago
  Well, yes me. I was on the same path. Researching hardware and coming to the same conclusion. I found tinfoil.sh which looks promising? My current solution is a private ChatGPT-like interface using OpenRouter’s API with Zero Data Retention enabled. Not perfect or verifiable but I think it’s acceptable for now. https://news.ycombinator.com/item?id=43996555 https://news.ycombinator.com/item?id=43996555 https://openrouter.ai/docs/guides/features/zdr https://openrouter.ai/docs/guides/features/zdr
  - deleted 2mo ago
    [deleted]
- jan_Sate 2mo ago
  Not suggesting a provider but if you're willing to get a Chinamod GPU, go get RTX 3080 20G or RTX 2080 Ti 22G. Get a couple of them and you can probably run Qwen3.8-27B at a reasonable speed. I've got a RTX 3080 20G for $450 like a year ago. With llama.cpp, bf16 kv cache, kv cache offloaded to RAM, Qwen3.8 27B UD-Q4_K_XL, single RTX 3080 20G, I got 10 tok/s initially and it dropped to 5 tok/s at 50k context. I'm thinking of getting another Chinamod GPU. Building a dual Chinamod GPU machine would only cost like $1000~$1500. It's not too bad compared with the alternatives!
  - cyanydeez 2mo ago
jing09928 2mo ago
[dead]
ycui7 2mo ago
the fact that this author cannot get qwen3.8-27b run at the same speed as qwen3.6-27b, says the article is not worth reading. the author does not know anything about how to run local AI. 3.8 and 3.6 are the same model with different weight. both tg and pp speed are so terrible on author's machine.
lowbloodsugar 2mo ago
>I realized I’d been sitting on the one thing most of those threads were missing: a machine that can actually run it properly, and time to measure it. This was written by AI I take it. The one thing, huh? All those idiots running it on their RTX 6000s at 140tok/s must be feeling pretty stupid for not getting a Mac Studio instead.
crossroadsguy 2mo ago
> Realistic minimum RAM 16GB When someone says stuff like this what does it mean? A Mac/computer with this much RAM capacity/marker (how would this even be useful in this case?) or 16 or ≈16GB free RAM? Because RAM usage at least on my Mac is another beast altogether and seems nothing will bring down the RAM usage other tham <rm -rf>ing the / (that is if I can and something survives to notice the miracle)
dexterlagan 2mo ago
Ornith 1.5 35B MoE is very close in most benchmarks, but runs at 100 tk/s where Qwen3.8 27B runs at 30 tk/s (MBP M5 Max 48GB). It's now my go-to model for quick inquiries. Ornith 1.0 had problems with world knowledge, but 1.5 seems very solid on that front. I'll be running some agentic tests today. Ornith 1.5 35B is so fast that my CPU doesn't even have time to heat up, where Qwen3.8 would turn my laptop into a toaster within a few minutes. I hope agentic tasks will hold up!
soricus 2mo ago
[flagged]
FacelessJim 2mo ago
I am running Ornith 1.5 Q4 on my work issued laptop which surprisingly has 40gb of RAM and 20GB of “unified” VRAM. I am getting 50t/s of prefill and 5-6t/s of inference. You can’t get crazy with it, but it’s pretty nice to just let it work away at simpler prompts overnight or during lunch break or while I am working on other things. Being able to literally a 30gb blob and then being able to talk with it while running on your machine it’s a different kind of magic compared to a far away magic in the cloud.

140 points by speckx 2mo ago

I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8? Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.

woadwarrior01 2mo ago
I suspect this might be due to MTP mispedictions. Either because the 3.8 quantized model weights do not include MTP heads, or as what happened with Ornith-1.5 recently, corrupted MTP heads, or due to a software issue in Ollama.
- rbanffy 2mo ago
  I remember an article a couple weeks back where someone used an HPE server with two older Xeon E-series CPUs and got reasonable performance by compiling llama with optimal switches for the architecture (memory alignment, page sizes, etc). I was impressed because those Xeons only had AVX2. If you are willing to spend a little more, you can get a slightly newer Xeon with AVX-512 and 4 sockets. I can't find the post though. Edit: a quick search from a server refurbisher nearby gives me a dual Xeon Gold 6330 28-core machine with 512GB and a 16GB V100 GPU. The memory is spread across 32 slots, for about £6,470.

liuliu 2mo ago

The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.

vunderba 2mo ago
I’ve never personally used it, but isn’t that what LM Studio is positioned as? From what I can gather, it seems extremely beginner-friendly.
- liuliu 2mo ago
  What you get with Draw Things: 1. Download the app from Mac AppStore; 2. Download the model; 3. Tap "Try recommended settings", have the guarantee that for whatever model it supports, it is the fastest in Mac ecosystem, no need to fiddle. What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb. To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
  - throw83939303 2mo ago
    Or you can just ask Sol to do all that for you, and do split testing, to find best performance. I would like to see "guarantee" for that app. You get full refound and it is free, right?
  - viccis 2mo ago
    Ollama was/is this but they pulled a heel turn...
  - e28eta 2mo ago
    I’ve casually used Draw Things a couple of times (great tool!), and I did not find it beginner friendly. I remember feeling a lot of doubt, confusion, and frustration. Am I getting these results because I picked the wrong model? Or I need to improve my prompt? Or the tool just can’t do what I’m trying to do? How current are these model recommendations? Have they been superseded by something newer? As a true beginner to AI at the time, even the sizes and bits were meaningless to me. And I don’t remember having any context as to what I should be attempting to run on my mac. So I think you need to include steps 4, 5, and 6 of swapping between different models, quants, and prompts. And step 7 is probably wading through the complicated UI, full of jargon that most people don’t know. Don’t get me wrong, I recognize that it’s a powerful tool, and the steep learning curve exists because it exposes quite a few power-user features. But for someone graduating from commercial AI image generators that take a text prompt and maybe a choice of couple models, it’s not easy.

ActorNightly 2mo ago

> Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on my Mac Studio M3 Ultra >~14 tokens/s For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)

wolvoleo 2mo ago
What? Below 100tk/s is worthless? I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace. Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally. And will it remain free? How are they going to make back all those trillions of investment? But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
- zaj00l 2mo ago
  I wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing. Still, as you said - for day to day, 50-60 tokens / s is a good baseline.
0x457 2mo ago
I get 80 tok/s on the same model, and it's pretty usable. I'm not chatting with it; it's either given a bag of tokens to generate an answer or it's doing some agentic coding. https://openrouter.ai/anthropic/claude-opus-5 https://openrouter.ai/anthropic/claude-opus-5 is it worthless because its 65 tps? re: gemini https://openrouter.ai/google/gemini-3.7-flash https://openrouter.ai/google/gemini-3.7-flash worthless as well? That being said, 14 tok/s is pretty slow.
- ActorNightly 2mo ago

Atreiden 2mo ago

I'm encountering the same behavior. I've tried 4-8bit quants and get 14-17 tok/s with one run that achieved 19. I'm eagerly awaiting dflash2 support in Unsloth or LM Studio, as allegedly that should increase throughput to around 30tok/s, which is the baseline for what I consider at least somewhat interactive. Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.

seanmcdirmid 2mo ago
That isn’t right. Use oMLX or something similar to serve the model with internal MTP enabled. You should get at least 40 tok/sec, but I’m not sure what hardware you are using. Dflash will mess up batching, not really worth it. You need to sift through hugging face for the right model though, and some of the MTP models are meant for rapid MLX and not oMLX.
- bilbo0s 2mo ago
  Well, yes and no. The author has an M3. Here's reality, MLX on the software layer will not magically place hardware matrix multiplication units in your GPU cores. Newer Macs are always just gonna smoke anything earlier than an M5. Not even sure why this person is trying to get this stuff to run on hardware that wasn't designed for AI? Everyone is pointing him to newer hardware precisely because you need the newer stuff to get models to be performant. You can go with AMD, NVidia or Apple, but you're gonna be using stuff designed well after the M3 if you want to push >100tok/s.
  - refulgentis 2mo ago
    > Not even sure why this person is trying to get this stuff to run on hardware that wasn't designed for AI? Posturing/overclaiming like this shades rather than illuminates, there are no worlds in which the "M3...wasn't designed for AI". My M4 Max 64 GB gets the same speed.
  - pcf 2mo ago
    I would say that an M3 Ultra was designed for the LLMs around that time, plus Apple was pushing hard towards MLX over time.

wolvoleo 2mo ago

These numbers are a lot lower than I expected from such pricey hardware. I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.

jdright 2mo ago
any numbers to share? also, what inference engine do you use and with which api?
- wolvoleo 2mo ago
  I use Llama.cpp and OpenAI-style API of course (who doesn't, unless you use Ollama perhaps?) Ollama sucks, they dropped ROCm support for my card out of the blue, and the Vulkan replacement wasn't ready. The developer just shrugged and didn't care about the people affected so I moved to llama-server. Also, here's some other reasons to avoid ollama: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/ And really, ollama are trying to sell their cloud platform and are always running behind llama.cpp in new features like the KV Cache quantisation. So numbers, I don't really measure it from day to day, I'm not really interested in benchmarks at all. But I just did a quick test, llama 3.1 8b q8_0 does about 70tk/s (with Vulkan up from 45 when I was still running ROCm), and qwen 3.5 9b q8_0 does about 45, I think that includes its time spent thinking, not sure. Overall I'm happy with that. Smaller sizes and quants are of course faster than this. It's not terribly much faster than a Mac Studio, but it's one hell of a lot cheaper of course. And for the price of a Mac Studio you can get a lot faster hardware. The only thing where the studio excels is if you need large amounts of vram like 64GB and up and run the models to go with that (I run multiple small ones and spread them over multiple cards)
fotcorn 2mo ago
I am also running some old AMD datacenter cards, 2x MI25 in my case. Getting around 30 tokens/second with short context. Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins. ROCm nightlies do actually support these old cards, just not the ROCm stable releases.

kennywinker 2mo ago

This low-effort slop post is misleading. It suggest that 3.8 is half as fast as 3.6, but this is almost definitely because MTP isn't enabled by default. The two models should perform the same - they are for me. When you get AI to think for you, you lose.

mickael-kerjean 2mo ago

Apart from learning, I cant see the point of spending that kind of money and energy to get such awfull token generation speed. Assuming memory bandwith is the bottleneck, is it just a matter of time until we start to see hbm4 based chip able to run qwen3.8 for normal people running at more than 500 token / seconds? Is memory speed the only technological bottlenecks that prevent us from having fast local model?

wolvoleo 2mo ago
For me, a HUGE benefit to running local models is that my data stays mine, on my computers only. Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business. With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap. For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams). PS: I wouldn't be surprised if Microsoft runs something similar on their end :) But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.
- rbanffy 2mo ago
  > my data stays mine, on my computers only. Which is a regulation constraint on many professions, BTW. Some people simply can't give some data to say, ChatGPT, without comprehensive guarantees written in Sam Altman's blood.

rbanffy 2mo ago

Pretty impressive how the Mac ends up less expensive than the Strix Halo boxes, at least here in Ireland. A 128GB Mac Studio with an M5 Max (the Ultra can only have 96 or 256GB) still costs less than the "GMKtec EVO-X2" or the Nvidia DGX Spark with similar performance. Is it the same in the US?

bronson 2mo ago
That would be such a sizeable disconnect that you might do some arbitrage. Current prices in USD: 128 GB 1 TB M5 Mac Studio: $5399 128 GB 1 TB GMKtec EVO-X2: $3499 128 GB 1 TB Framework Desktop: $3748
lowlat 2mo ago
That's not right. Mac Studio M5 Max 128gb, in Apple Store Ireland, starts from 5,909e (~6,847 USD; tax included). On the other hand, GMKtec Evo-X2 128gb can be currently purchased for 3,299e (~3,822 USD; tax included) from GMKtec online store (free shipping from DE warehouse). It's quite the difference in price.
- rbanffy 2mo ago
  It’s the price I got from amazon.ie
  - lowlat 2mo ago
    I figured it's dodgy amazon ie listings. DGX Spark/Asus Ascent GX10 prices are inflated there too. They'll also be happy to sell you an RTX 5090 for 10 grand.

kgeist 2mo ago

Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's older Llama build from a "couple weeks ago" failed to run Qwen 3.8 because it did not support Qwen35 architecture, but both 3.6 and 3.8 are based on Qwen35 which was released in February 2026. The post doesn't make any sense.

iagooar 2mo ago

I am also seeing slower speeds, roughly the same ballpark, sometimes even lower - 10-11 tok/s on M5 Max. If there is that ONE version (GGUF or MLX) that runs roughly as fast as 3.6 used to run, please let me know. What would be incredible is the 3.8 35B MoE version too, I can run 3.6 with 60 tok/s which is a really, really nice speed.

oceanplexian 2mo ago

Ollama? Running a q1 quant? I don't think the writer knows what the are doing here to be honest. You will get better information cruising r/localllama for about 10 minutes.

bilbo0s 2mo ago
There are many, many people who are betraying a lack of facility with these models. Why is anyone trying to run these on an M3? I thought it was common knowledge that, if you insist on using Apple, only M5 processors and higher have matrix multiplication units in the cores? I've seen the same thing with people buying NVidia cards with 16GB of ram and wondering why they aren't getting 100tok/s? Guys, please, be reasonable. You'll have to get the hardware if you want to run these things fast. If you want to experiment, the slow stuff is fine. But try not to get on HN and ask why your M3 can't get 100tok/s. You're kind of out-ing yourself.
__mharrison__ 2mo ago
This is a problem that Hugging Face could easily help solve. Crawling through Reddit or forums to find the right incantation to run a model is frustrating.

sciencesama 2mo ago

AI is a killer adblocker ! imagine connecting it to instagram and curating all the images that you actually care about removing all ads !

craftkiller 2mo ago
That seems like an expensive way to block ads considering traditional ad blockers have been doing that with a tiny fraction of the compute for decades.
- rgovostes 2mo ago
  Have you used a social network in the past year? Facebook, YouTube, Pinterest, et al. have become increasingly hostile to blockers with techniques that break pattern-matching on hostnames and element selectors. It may be inevitable that adblockers move toward visual identification.
  - andrekandre 2mo ago
    > Facebook, YouTube, Pinterest, et al. have become increasingly hostile to blockers maybe at some point we should just stop visiting these sites if they are so hostile... we keep going to them despite all of this and it just serves to delay any kind of alternative getting pickup (not an easy task, i know)
    - simonw 2mo ago
      We go to them because that's where the friends and organizations we care about have chosen to post their content.

refulgentis 2mo ago

Slop; all those LocalLlama threads "you" read had #s too. And Q1 quant? Why? You have the memory...

arjie 2mo ago

The cheapest card that will run this model very well is a ln unlocked CMP 170HX. But you can run it on a 3090. I run it on an old spare A6000 Ampere. I think I wouldn’t use anything lower than 60 tok/s though, which you can get with MTP etc. I just use a full vllm stack but some people see a lot of speed with ninfer (there are non 5090 ports). The large RAM Macs are unusable for inference of dense models as of now. Token generation is too slow.

pwython 2mo ago

Yea Qwen3.8 wasn't fun to use on my 64GB M4 Max either (better than these numbers though), so my new daily driver is Ornith-1.5-35B-A3B-MLX-4bit. I recommend giving that a whirl if you're on similar hardware, it's definitely better than Qwen3.6 35b-a3b which was my go-to before. https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B

TobTobXX 2mo ago
Can you tell more about your experience with Ornith? I've come across it, and the benchmarks on its landing page are unbeleavably good. But it isn't featured on more established benchmarks like ArtificialAnalysis and it's not on OpenRouter, so I wrote it off as scam.
mark_l_watson 2mo ago
I tried Ornith-1.5-35B-A3B-MLX-4bit on a 32G M2-Pro mac mini with pi and OpenCode. I didn’t get very good results coding in Python, Racket, and TypeScript. I have seen several positive comments like yours so I was probably doing something wrong. I amgetting the new 64G mac mini in 4 weeks, and I made a note to try Ornith-1.5-35B-A3B-MLX-4bit again.

ThrowawayTestr 2mo ago

I'm running the 5 bit quant on a 3060 12G. Get a handful of tokens/s. Prompts take 10-20 minutes to complete but it works.

ig0r0 2mo ago

[flagged]

firemelt 2mo ago

is it really worth to run urself? the watts drsin and that 100% gpu usage

throwa356262 2mo ago
Short answer: no Long answer: some dude on youtube calculated that even if electricity was free it would take you 37 years to get back your investment. Slightly longer answer: ... but if you absolutely must have local LLM, a GPU with 24-34 GB VRAM is much cheaper and faster than a Studio

petercooper 2mo ago

FWIW, on my Mac Studio I get ~24-27 tok/s generation between 0-16k context in - that's on the Q6_K GGUF with speculative decoding on. I have spent zero effort optimizing/improving this so far but will be trying the 4 bit MLX next (I've tended to find models drop off somewhat below 6 bit but maybe that isn't the case nowadays).

spwa4 2mo ago

Peaks at over 300 tok/s on my 5090 with Dflash2. At maximum context (252000 or so) still get 60 tok/s. And I don't have n-gram optimization working yet, I only have 64G memory.

Youden 2mo ago

I've been thinking about buying a system to run LLMs locally but the price for one that'll run Qwen3.8-27B well is quite offputting to say the least. What I've been looking at instead is inference providers that use TEE and E2EE to provide cryptographic guarantees that my prompts and responses are only visible to me and the GPU itself. Despite their docs and assurances of what their guarantees mean, I'm having trouble getting to a point where I'm actually comfortable trusting them with secrets though. Phala for example seems to be E2EE only to the gateway and will then forward prompts to (potentially third party) providers. Has anyone been down this path and found a provider they feel safe with?

lzy 2mo ago
Well, yes me. I was on the same path. Researching hardware and coming to the same conclusion. I found tinfoil.sh which looks promising? My current solution is a private ChatGPT-like interface using OpenRouter’s API with Zero Data Retention enabled. Not perfect or verifiable but I think it’s acceptable for now. https://news.ycombinator.com/item?id=43996555 https://news.ycombinator.com/item?id=43996555 https://openrouter.ai/docs/guides/features/zdr https://openrouter.ai/docs/guides/features/zdr
- deleted 2mo ago
  [deleted]
jan_Sate 2mo ago
Not suggesting a provider but if you're willing to get a Chinamod GPU, go get RTX 3080 20G or RTX 2080 Ti 22G. Get a couple of them and you can probably run Qwen3.8-27B at a reasonable speed. I've got a RTX 3080 20G for $450 like a year ago. With llama.cpp, bf16 kv cache, kv cache offloaded to RAM, Qwen3.8 27B UD-Q4_K_XL, single RTX 3080 20G, I got 10 tok/s initially and it dropped to 5 tok/s at 50k context. I'm thinking of getting another Chinamod GPU. Building a dual Chinamod GPU machine would only cost like $1000~$1500. It's not too bad compared with the alternatives!
- cyanydeez 2mo ago

jing09928 2mo ago

[dead]

ycui7 2mo ago

the fact that this author cannot get qwen3.8-27b run at the same speed as qwen3.6-27b, says the article is not worth reading. the author does not know anything about how to run local AI. 3.8 and 3.6 are the same model with different weight. both tg and pp speed are so terrible on author's machine.

lowbloodsugar 2mo ago

>I realized I’d been sitting on the one thing most of those threads were missing: a machine that can actually run it properly, and time to measure it. This was written by AI I take it. The one thing, huh? All those idiots running it on their RTX 6000s at 140tok/s must be feeling pretty stupid for not getting a Mac Studio instead.

crossroadsguy 2mo ago

> Realistic minimum RAM 16GB When someone says stuff like this what does it mean? A Mac/computer with this much RAM capacity/marker (how would this even be useful in this case?) or 16 or ≈16GB free RAM? Because RAM usage at least on my Mac is another beast altogether and seems nothing will bring down the RAM usage other tham <rm -rf>ing the / (that is if I can and something survives to notice the miracle)

dexterlagan 2mo ago

Ornith 1.5 35B MoE is very close in most benchmarks, but runs at 100 tk/s where Qwen3.8 27B runs at 30 tk/s (MBP M5 Max 48GB). It's now my go-to model for quick inquiries. Ornith 1.0 had problems with world knowledge, but 1.5 seems very solid on that front. I'll be running some agentic tests today. Ornith 1.5 35B is so fast that my CPU doesn't even have time to heat up, where Qwen3.8 would turn my laptop into a toaster within a few minutes. I hope agentic tasks will hold up!

soricus 2mo ago

[flagged]

FacelessJim 2mo ago

I am running Ornith 1.5 Q4 on my work issued laptop which surprisingly has 40gb of RAM and 20GB of “unified” VRAM. I am getting 50t/s of prefill and 5-6t/s of inference. You can’t get crazy with it, but it’s pretty nice to just let it work away at simpler prompts overnight or during lunch break or while I am working on other things. Being able to literally a 30gb blob and then being able to talk with it while running on your machine it’s a different kind of magic compared to a far away magic in the cloud.