8 ms·
The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Some people will be happy to pay
by bensyverson 4mo ago
The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0]
Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs.
[0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space-black-standard-display-apple-m5-max-chip-18-core-cpu-40-core-gpu-128gb-memory-2tb-storage https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...
- Insanity 4mo agoBut you have to factor in that this device will last you 5-10 years. That said, I wouldn't spend almost $7k USD on this macbook lol.
- someperson 4mo agoIn 5-10 years, incremental cloud tokens will be far cheaper (likely but not guaranteed).
- petilon 4mo agoMemory requirements of newer models will increase, so while the hardware may last 10 years it won't be able to run the latest models for 10 years.
- Insanity 4mo agoYou raise a fair point, but I'm not convinced it'll offer a meaningful difference in performance as long as we're stuck with the current AI paradigm.
- simonw 4mo agoIt can't run the latest models today - GLM-5.2 class models already need 1TB+ of RAM. ... but, the models that WILL run on 128GB (or 64GB or even 32GB) models today are a huge improvement on the best models that would run in the same amount of memory six months ago.
- godwinsonsucks 4mo ago[dead]
- johndough 4mo ago> GLM-5.2 class models already need 1TB+ of RAM. If you quantize GLM-5.2 to 4 bit, you can do it in less than 500GB: https://huggingface.co/unsloth/GLM-5.2-GGUF https://huggingface.co/unsloth/GLM-5.2-GGUF (table on the right) If you find three finds that also have a 128GB MacBook, you can chain them together (the MacBooks, not your friends) and make it work. You could also run GLM-5.2 on a single MacBook if you stream the active parameters from disk, but even with speculative decoding, you'd probably only get in the order of 1 token per second, so this is not really practical for most applications.
- bluGill 4mo agoWill they? Or will we find ways to optimize models and need less? Only time will tell.
- roadside_picnic 4mo agoMy experience working in the open model space pretty deeply (both LLMs and diffusion models) for years now is that it is not quite as simple as that. In the open model space an insane amount of effort goes into getting more powerful models to run with the same or less RAM. For example in the diffusion world many things that could not be run on easily under 24GB of VRAM actually run much better today with much less VRAM than they did a few years ago. You can do many things today with 8-16GB of VRAM that would not have been possible. At the same time the most advanced open models, like LTX 2.3 for video gen, still seem to respect 24GB of VRAM as the upper bound. Similarly the standard "big" but localish open model for LLMs back in the day was Llama 3 70B, this was both a much worse and much larger model than Qwen 3.6 27B So in two different spaces I've witnessed the "RAM required to run the best" decreasing or at least remaining stable, while the performance being achieved in both areas is astounding (LTX 2.3 is faster, better and more capable than the Wan 2.2 model that held popularity before it). The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models.
- petilon 4mo ago> insane amount of effort goes into getting more powerful models to run with the same or less RAM The same can be said about operating system memory requirements. I am sure Linux and Windows kernel developers can confirm. Yet 30 years ago Solaris used to run comfortably in 16 MB of RAM, today you need 512 times that to run Linux.
- prima-facie 4mo ago> The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models. I'm thinking of getting a SoC machine with 128GB RAM but the bandwidth is limited to 256 GBps. Would you even consider such a machine a decent investment, or should I wait for the newer gen of chips? Thanks!
- roadside_picnic 4mo ago
- cyanydeez 4mo agoI think you have too much faith in context AGI. at 128GB, you can find almost it's entire context for Qwen3.6 35B MoE. Again, I think you have too much faith in extrapolation. It's like you got a baby at 0 months, then measured it at 12 months and expect it to be a giant.
- naikrovek 4mo agoAvailable models aren’t really trending upward in size. Not like I thought they would, anyway. They’re trending to be the right size to be good. Qwen3.6-35B is not as good as Qwen3.6-27B. The larger model is faster, but a lot dumber; it gets caught in loops, makes crazy mistakes, and is just not as good. It’s bigger, but it is nowhere near as good as the 27B variant.
- zargon 4mo agoQwen3.6-35B-A3B is worse than 27B because it's an MoE and 27B is dense. 35B only passes each token through 3B of its total parameters, whereas 27B sends each token through all 27B parameters.
- naikrovek 4mo agoI did not know that, thank you. I should have known that probably, but I'm new to all of this.
- regularfry 4mo agoNah. There are already models at every size on the scale. If you want to run an open 1T model today, you can. What's going to happen is that the capability at any given size point is going to get better over time as new training regimes cram more into the available space. A 27b model released next year will be better than a 27b model this year (else why release it?). Hardware will get more useful, not less.
- jubilanti 4mo ago[flagged]
- h4ny 4mo ago[flagged]
- kllrnohj 4mo ago> maybe tell us how much a non-Apple system that you can run that (probably similarly or faster) would cost? Ryzen AI Max 395+ with 128GB of unified memory can be found around $3-4k. But 27B isn't that large, either, especially if you are ok with the quantized models. So this laptop choice seems to more be a "because they had it" rather than "this is what's necessary for this particular workflow"
- h4ny 4mo agoThat's my point. You can run Qwen3.6 27B with MTP and whatever else you want to bolt onto it at 256k context for much less than even a Ryzen AI Max 395+ with 128GB would cost. Even unquantized you don't need 128 GB so given your comment and the downvotes maybe I didn't word my original comment properly for this?
- dang 4mo agoYikes, you broke the site guidelines badly with this post. Could you please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to them? You're welcome to make your substantive points thoughtfully, just not aggressively.
- Catloafdev 4mo agoThe model they reference can be easily run with 24gb+ of VRAM, and there are other similar models capable of running easily on 16gb of VRAM. It's not like 128gb is a requirement here.
- thewebguyd 4mo agoI'd go for at least 32GB+. It'll fit in 24GB but leaves you little to no room for context, and that's at 4-bit quantization. If you want to run unquantized, you definitely need 128GB.
- Catloafdev 4mo agoNobody runs unquantized, there's literally no reason to. Q8 would be the largest anyone actually runs on consumer hardware for inference.
- deleted 4mo ago[deleted]
- bityard 4mo agoHalving the precision of the weights is not a free lunch...
- Catloafdev 4mo agoQ8 is virtually lossless. The quantization is much more noticeable around Q4 and below. FP16->Q8 on consumer hardware is 2x the speed at ~99.99% the quality.
- rvba 4mo agoAny source that confirms the 99.99% quality?
- Catloafdev 4mo ago
- oldfuture 4mo agoa lot of credits? we can’t predict any price change for them
- dofm 4mo agoThe maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it. I don't know how much serious hands-free agentic coding I will ever do on my MacBook alone, but I do know that I would not have got so far into understanding this without tinkering with local models, llama.cpp, LM Studio, and LM Studio and all that. I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus. Until, that is, I could poke around with setting it up on my own (secondhand) machine, watching the API calls, understanding some of the terminology. I didn't even buy the machine for that; it's just adequate to the task. The Neo is too small to really get much benefit from this opportunity to make it more visceral and knowable.
- rusk 4mo ago> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. I found LM studio to be a nice starting point. Frindlier and more featureful than Ollama and not as intimidating as llama.cpp (though you will want to use that eventually)
- dofm 4mo agoLM Studio is also nice because of the way the interface explains things; parameters have explanations and hints. It has been designed by people who really care about making it understandable. I tried Ollama but I've settled on Unsloth Studio generally; once things really settle down I'll just run the llama-server UI, which is pretty nice. A friend is tinkering with LLMs for amusement on a 16GB Raspberry Pi 5, and when I explained that llama.cpp now had a typical web chat interface he was so happy — it's amazing what the "table stakes" are now.
- ddalex 4mo agoI just got Claude to download and install all the models and servers and agents and prepare all the launch scripts for me... no need to learn, just ask it to do it for you
- nozzlegear 4mo agoJust putting it out there: I run Qwen 3.6 on my M1 Mac Studio with 64gb. It's quantized and all that, but I agree with TFA: it's the sweet spot for local development right now.
- dannyw 4mo agoI’m running the same model on a 48GB MBP with a q4 quant and it’s pretty decent. You definitely don’t 128GB. That’s the scale for 70B models at q8 or something.
- doodlesdev 4mo agoHow much does one of those cost in the US? Here in Brazil, your notebook is worth as much as a used Honda Fit, which seems absolutely insane. For comparison, the ThinkPad I'm currently running cost me 1/20 of how much this MBP costs here, leaving me with over $8.000 to spend with LLM inference (if I actually spent money with that).
- dannyw 4mo agoI purchased mine for approximately $4400 AUD before the price hikes. That unit is now ~$5100 AUD. I use my MBP essentially as my workstation, it's almost always plugged in. I have a MBA (M4, 24GB RAM) that I picked up for ~A$1500 or so, and that's an amazing daily driver. I don't do local LLM inference on that unit, I can just hit my own APIs (via LM Studio) on the MBP over Tailscale.
- dom96 4mo agoI've been running it on my 48GB MBP too and it's not particularly great. Super slow and not near enough to the quality provided by even Claude Sonnet.
- DrammBA 4mo ago> I’m running the same model on a 48GB MBP with a q4 quant and it’s pretty decent. Context size?
- georgeven 4mo agoI have a 1500 dollar machine that can run it at 50 tok/s (3 V100s)
- Dig1t 4mo agoHow did you buy 3 V100's for $1500??
- sixdimensional 4mo agoNot OP and just guessing, but probably SXM2 GPU modules for the V100. Those can be acquired fairly inexpensively, but there is work to do to get them working together and the V100 has some limitations on the types of models you can run.
- georgeven 4mo agoPCIE 16 gig is <300 dollars these days a piece
- georgeven 4mo agoon eBay you can buy 4 of them for 1000. PCIE 16 gigs.
- AnimalMuppet 4mo agoHow many credits would it buy? How long would it take to use them up? What's the payback period? From what I understand, for a developer, $5000/month is maybe the high end, but $5000/year is fairly standard. (Is that accurate?) So if it pays back in 15 months, that's pretty decent. If it pays back in two months, that's spectacular.
- eli 4mo agoAre you comparing the cost of hosted Opus to running Qwen 3.6 locally? That doesn't really seem fair.
- deleted 4mo ago[deleted]
- dminik 4mo agoUsing some rough napkin (well, spreadsheet) math, if you ran Qwen 27B for every minute every day at the current price of $0.195/$1.56 with a 2:1 input to output ratio (eg. agentic coding) at the advertised 22 tps it would take you just about 11 years to get to ~$5000 spent. Disclaimer: There's a 35% sale from Alibaba right now. And I'm not accounting for input tokens going faster than output tokens.
- organsnyder 4mo agoI run Qwen 3.6 on my Framework Desktop 128GB, and it's very performant. I know Framework has had to raise the price since I preordered mine, but they're still well under half the cost of that Macbook.
- andy99 4mo agoI get ~55 Tok/s on my framework desktop with the 35B A3B q8 model, and so far am also very happy with the coding performance.
- cyanydeez 4mo agodid you upgrade to MTP?
- imrehg 4mo agoOn the MoE versions of these models the MTP versions have only marginal benefit. In my trials the speed-up is <20% (not the ~2x that happens with some other setup/models) and usually more like 10%. Ie. something like 13 -> 15 token/s... on my device. I still use the MTP version as it _feels_ slightly better quality, and because the unsloth quantizations I can get have more variety to fit into the various systems at hand... but that's not for the MTP aspect, unfortunately. In the article they did have ~2x performance on the 27B (which might be something to retry, though on my Framework that would bring it from 5 -> 10 token/s so still "excrutiating" speed, probably). YMMV for sure.
- andy99 4mo agoThat was with the MTP version
- bityard 4mo agoThere are several variants of Qwen 3.6, the MoE models are performant on Strix Halo, but the 27B dense model (the one spoken about in TFA, and generally regarded as the best of the group in terms of quality) is not so performant: https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/
- porphyra 4mo agoYou can also run Qwen 3.6 27B dense model on DGX Spark with comparable performance [1][2] for about $4000 (Asus Ascent GX10 is $3999 at various retailers). In theory you can also get 48GB of VRAM with, say, two 3090s, but it will take up a lot of space and generate a lot of heat compared to the Macbook Pro and GB10. [1] https://x.com/MiaAI_lab/status/2070859135399182444 https://x.com/MiaAI_lab/status/2070859135399182444 [2] https://github.com/MiaAI-Lab/Qwen3.6-27B-NVFP4-vLLM https://github.com/MiaAI-Lab/Qwen3.6-27B-NVFP4-vLLM
- esperent 4mo ago> 48GB of VRAM with, say, two 3090s So like... $2000+ just for the used GPUs? Plus I assume it's considerably more effort to get it working.
- fluoridation 4mo ago>Plus I assume it's considerably more effort to get it working. Nah, not really. It is a little annoying in terms of space and power, though. Not every case and motherboard can support cards that big.
- lee_ars 4mo agoThe tweet you link shows "Qwen 3.6 35b NVFP4 - 256k ctx, 110 tok/s", but I'm getting only half that, around 50 tok/sec, on a DGX Spark with Qwen3.6-35B-A3B-NVFP4 (via vLLM) plus speculative decode w/EAGLE3. I'd be ecstatic to see 110 tok/sec and I wish they had some more sourcing for the exact config, because it's double what I'm getting. edit - after actually reading the tweets (had to use xcancel) and visiting the source git repo, switching to MTP for speculative decode makes things a hell of a lot faster, and the abliterated model plus dflash makes it even faster! I'm now seeing 70-90 tok/sec for most stuff. I like!
- porphyra 4mo agoI think Atlas might also be slightly faster than vLLM: https://flowtivity.ai/blog/120-tok-s-1m-context-private-ai-dgx-spark/ https://flowtivity.ai/blog/120-tok-s-1m-context-private-ai-d...
- dvduval 4mo agoAbsolutely for the average developer the token speed is just going to be too slow for it to be workable. I think we’re looking at 2028 when memory becomes cheaper again and they’ll be a lot more people using local models.
- stymaar 4mo ago> The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0] Qwen3.6-27B would be faster on a 3090 that costs around $1000-1200 though so I don't think it's a good counter-argument. Op just happened to have that MacBook, but it doesn't mean it's necessary to run the model.
- boutell 4mo agoThat 3090 is going to burn 750W and it will still cap you at a 4 bit quant and ~48K context. Here's someone who worked through it: https://github.com/noonghunna/qwen36-27b-single-3090 https://github.com/noonghunna/qwen36-27b-single-3090 Flies though (50-70tps is impressive for a model this smart) I went through roughly the same process to get it working on my M2 Macbook Pro... at awful speeds of course, since models like this one are mostly bound by memory bandwidth.
- stymaar 4mo ago> That 3090 is going to burn 750W The 3090's TPD is 350W, but given that LLM's token generation isn't compute bound, people usually undervolt these cards to reduce power consumption. IIRC you can get as low as 200-250W without any degradation. Caveat these figures are without speculative decoding and at batch size =1.
- 4chandaily 4mo agoThis is correct. I have (4) 3090s in my inference server, and they are each capped at 250w. I run Qwen 3.5 122B-A10 at about 45-50tok/s on this and am quite happy with it. At idle it draws around 95-105w for all four, which is a bit high, but tolerable.
- hughw 4mo agoMy eyes glaze over reading all the AI produced verbiage. I did find a few useful parameter settings I've already discovered using my single 3090 and ollama. I'm just remarking that the LLMs overwhelm me with minutiae, especially as I'm working on code design. I frequently ask it to restate concisely, and that helps. [edited to mention ollama as a nice alt]
- cyanydeez 4mo agoAMD started their 128GB Halo Strix at a pretty damn good point at ~2.5k; I got mine after the first memory bump at $3k. I think you might be a little to into the stew here.
- zdragnar 4mo agoI got mine at the same price point, and I've been pretty pleased with it. Tailscale lets me use it from my ultrabook / lightweight laptop, no burning lap or crazy fan noises. Desktops with the amd ai+ 395 are still fairly affordable for what they can do. I haven't tried it with https://lemonade-server.ai/ https://lemonade-server.ai/ yet but I just might give it a shot.
- organsnyder 4mo agoI'm running Lemonade on Nixos on my Framework Desktop. I had been trying other tools out before finding Lemonade, but Lemonade really made it plug-and-play.
- dmayle 4mo agoFor that price you can put together a PC with 128GB of ram ($2000) and an RTX 5090 ($3600) and get 70-100 tokens per second instead of 45
- deleted 4mo ago[deleted]
- throw1234567891 4mo agoBut the tokens or credits are gone. MacBook stays. You can run other models on the same MacBook. What I read people burn every month on saas… for that money you break even on that MacBook in 5 months. Edit: it’s not just “data privacy”, when you are using Claude, you are shipping EVERYTHING to Anthropic. It’s crazy.
- wilsonnb3 4mo agoCompanies are already shipping everything to Microsoft or Google and 17 other companies, just the cost of doing business.
- DANmode 4mo agoThat’s at today-prices. If the cost doubles, or 4x, which is seems to need to for them to go profitable, what then?
- throw1234567891 4mo agoSure, but no one gets everything. Just that one.
- wahnfrieden 4mo agoIt's much slower, and often quantized
- throw1234567891 4mo agoOkay, and?
- colinsane 4mo agoi like that people are taking the privacy argument seriously, after however many decades. i think there are other arguments to be made for running these locally which are less settled, but IMO the Fable debacle drives it home: the surest way to embrace this technology without worry that it will be taken away from you down the road is to physically own the compute.
- r_lee 4mo agoif you need to ensure that, then just back up the model and buy hardware if the need arises
- colinsane 4mo agothat's somewhere between saying "use Android, just switch to Graphene if/when they lock it down", and saying "just switch to postmarketOS/Ubuntu Touch/whatever flavor of Linux takes off". i've watched friends try that route; i've been through this before. taking a downgrade is never fun: if it's a thing you're likely to care about in the future, then sometimes it's better to place yourself in the right ecosystem early.
- r_lee 4mo agoI just don't see how with the whole open weight system this situation would happen or that it'd be likely enough to warrant this in terms of privacy, yes that's a real application, but someone taking it all away? I don't see it happening. it's not an OS or a device, it's just a box/thing that runs a model, it's really commodity stuff we're talking about more realistic concern would be that the open labs wouldn't be able to compete in the future thus development ends, but that means you can't host models that don't come out so... again maybe I misunderstood but I just don't see why this would be worth it just for that one concern
- montebicyclelo 4mo agoIsn't the directionality important. I.e. it is currently possible to run useful / great models locally, but on high end machines; and in a few years we will likely be able to run even better models on standard machines.
- trentor 4mo agoRuns fine on 2x4080s or on two 5060/5070s with 16GBVRAM... and faster than on the mac.
- elorant 4mo agoYou can get an AMD Strix Halo with half that price even after hardware price adjustments. Besides you don't need 128GB of RAM to run a 27B model.
- acchow 4mo agoThat $6700 is a $5000 upgrade over a base model Macbook Pro. $5000 in US Treasuries (currently at 4.89%) yields $244.5/yr. That's more than enough to cover the annual Claude Pro subscription ($200/yr) which includes Claude Code with lots of Sonnet usage (far better than Qwen 3.6)
- neonstatic 4mo agoI think the argument isn't that local is cheaper - it's that local is doable and delivers unparalleled privacy.
- iosjunkie 4mo agoAnd your government can’t take it away on a Friday afternoon.
- redox99 4mo agoI bought 2 used 3090s some years ago for $500 each. They're probably a bit more expensive now, but I guess for something like $2000 you can build a barebones 2x3090 PC which will be way faster than a Macbook. (you're fine with very basic hardware outside the GPUs)
- ricardobayes 4mo agoOh definitely. I've seen GLM 5.2 go for around $4 per million output tokens.
- stared 4mo agoAll experiments with Qwen 3.6 required no more than 48GB Apple Silicon. I believe you can go even further with more aggressive quantizations - one can go down even further. In any cases, from the economic point of view, running models on laptops make little sense. Even at the pure cost of energy consumption, it might be hard to beat pricing at tokens generated at scale. At the same time, it is a breaktrough, that will change the game. Previously such vibe coding on consumer device was not hard or costly - it was impossible.
- jeffybefffy519 4mo agoI still dont trust the Anthopic and OpenAI are not training on my code. I even just thinking keeping track of what code you have received in prompts and to train/not train on it seems like an impossibly difficult task.
- andrekandre 4mo agoam i right in assuming your code is closed-source? i'd expect anything on github for example to be already in their training set or is training on actual usage more useful to them?
- shockembopper 4mo agoI’ve got qwen3.6 27b running on my media server atm. Given that I built on top of what I already had, it didn’t cost me nearly that amount. I’ve been running 2x 5060 ti 16gbs, and when using text only and nvfp4, I can run the model with 200k context length and roughly 50-60 toks. It’s very good, and costed me about $800 after buying the gpus from microcenter.
- ant6n 4mo agoDoesnt it run on the Macbook Neo... just slower?
- pimeys 4mo agoYes. It is very expensive now. I'm still so so happy I decided last summer to bite the bullet and pre-ordered the Framework Desktop 128GB model. I paid 2424 euros in total for this machine. And it can easily run the models discussed in the comments and in the article. It's tiny, and runs CachyOS like a champ. Over 4000 euros less than the price you listed. We can all send a thank you letter for our friendly billionaires such as Sam Altman for the price situation we're in today: https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram-deal https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...
- razster 4mo agoI'm running it on my 4070 12gb with 96gb mem, I'm very happy with the results even if I have to wait a couple minutes for results. To me this is far better than I expected and will continue to use it and improve with skills.md. Pi.dev is amazing by the way.