11 ms·
A guide to local coding models
- nzeid 10mo agoI appreciate the author's modesty but the flip-flopping was a little confusing. If I'm not mistaken, the conclusion is that by "self-hosting" you save money in all cases, but you cripple performance in scenarios where you need to squeeze out the kind of quality that requires hardware that's impractical to cobble together at home or within a laptop. I am still toying with the notion of assembling an LLM tower with a few old GPUs but I don't use LLMs enough at the moment to justify it.
- a_victorp 10mo agoIf you ever do it, please make a guide! I've been toying with the same notion myself
- suprjami 10mo agoIf you want to do it cheap, get a desktop motherboard with two PCIe slots and two GPUs. Cheap tier is dual 3060 12G. Runs 24B Q6 and 32B Q4 at 16 tok/sec. The limitation is VRAM for large context. 1000 lines of code is ~20k tokens. 32k tokens is is ~10G VRAM. Expensive tier is dual 3090 or 4090 or 5090. You'd be able to run 32B Q8 with large context, or a 70B Q6. For software, llama.cpp and llama-swap. GGUF models from HuggingFace. It just works. If you need more than that, you're into enterprise hardware with 4+ PCIe slots which costs as much as a car and the power consumption of a small country. You're better to just pay for Claude Code.
- le-mark 10mo agoI was going to post snark such as “you could use the same hardware to also lose money mining crypto” then realized there are a lot of crypto miners out their that could probably make more money running tokens then they do on crypto. Does such a market place exist?
- satvikpendem 10mo agoJeff Geerling has (not quite but sort of) guides: https://news.ycombinator.com/item?id=46338016 https://news.ycombinator.com/item?id=46338016
- a96 10mo agoAlso worth looking is stuff from Donato Capitella : https://github.com/kyuz0 https://github.com/kyuz0 https://www.youtube.com/@donatocapitella https://www.youtube.com/@donatocapitella https://llm-chronicles.com/ https://llm-chronicles.com/ etc
- whitehexagon 10mo agoSimonW used to have more articles/guides on local LLM setup, at least until he got the big toys to play with, but well worth looking through his site. Although if you are in parts of Europe, the site is blocked at weekends, something to do with the great-firewall of streamed sports. https://simonwillison.net/ https://simonwillison.net/ Indeed, his self hosting inspired me to get Qwen3:32B ollama working locally. Fits nicely on my M1 pro 32GB (running Asahi). Output is a nice read-along speed and I havent felt the need for anything more powerful. I'd be more tempted with a maxed out M2 Ultra as an upgrade, vs tower with dedicated GPU cards. The unified memory just feels right for this task. Although I noticed the 2nd hand value of those machine jumped massively in the last few months. I know that people turn their noses up at local LLM's, but it more than does the job for me. Plus I decided a New Years Resolution of no more subscriptions / Big-AdTech freebies.
- cloudhead 10mo agoIn my experience the latest models (Opus 4.5, GPT 5.2) Are _just_ starting to keep up with the problems I'm throwing at them, and I really wish they did a better job, so I think we're still 1-2 years away from local models not wasting developer time outside of CRUD web apps.
- OptionOfT 10mo agoEh, these things are trained on existing data. The further you are from that the worse the models get. I've noticed that I need to be a lot more specific in those cases, up to the point where being more specific is slowing me down, partially because I don't always know what the right thing is.
- cloudhead 10mo agoFor sure, and I guess that's kind of my point -- if the OP says local coding models are now good enough, then it's probably because he's using things that are towards the middle of the distribution.
- dkdcio 10mo agosimilar for me —- also how do you get the proper double dashes —- anyway, I’d love to be able to run CLI agents fully local, but I don’t see it being good enough (relative to what you can get for pretty cheap from SOTA models) anytime soon
- simonw 10mo ago> I realized I looked at this more from the angle of a hobbiest paying for these coding tools. Someone doing little side projects—not someone in a production setting. I did this because I see a lot of people signing up for $100/mo or $200/mo coding subscriptions for personal projects when they likely don’t need to. Are people really doing that? If that's you, know that you can get a LONG way on the $20/month plans from OpenAI and Anthropic. The OpenAI one in particular is a great deal, because Codex is charged a whole lot lower than Claude. The time to cough up $100 or $200/month is when you've exhausted your $20/month quota and you are frustrated at getting cut off. At that point you should be able to make a responsible decision by yourself.
- hamdingers 10mo agoAnd as a hobbyist the time to sign up for the $20/month plan is after you've spent $20 on tokens at least a couple times. YMMV based on the kinds of side projects you do, but it's definitely been cheaper for me in the long run to pay by token, and the flexibility it offers is great.
- iOSThrowAway 10mo agoI spent $240 in one week through the API and realized the $20/month was a no-brainer.
- __mharrison__ 10mo agoI'm convinced the $20 gpt plus plan is the best plan right now. You can use Codex with gpt5.2. I've been very impressed with this. (I also have the same MBP the author has and have used Aider with Qwen locally.)
- baq 10mo agobit the bullet this week and paid for a month of claude and a month of chatgpt plus. claude seems to have much lower token limits, both aggregate and rate-limited and GPT-5.2 isn't a bad model at all. $20 for claude is not enough even for a hobby project (after one day!), openai looks like it might be.
- simonw 10mo agoThis story talks about MLX and Ollama but doesn't mention LM Studio - https://lmstudio.ai/ https://lmstudio.ai/ LM Studio can run both MLX and GGUF models but does so from an Ollama style (but more full-featured) macOS GUI. They also have a very actively maintained model catalog at https://lmstudio.ai/models https://lmstudio.ai/models
- ZeroCool2u 10mo agoLMStudio is so much better than Ollama it's silly it's not more popular.
- thehamkercat 10mo agoLMStudio is not open source though, ollama is but people should use llama.cpp instead
- behnamoh 10mo ago> LMStudio is not open source though, ollama is and why should that affect usage? it's not like ollama users fork the repo before installing it.
- thehamkercat 10mo agoIt was worth mentioning.
- smcleod 10mo agoI suspect Ollama is at least partly moving away open source as they look to raise capitol, when they released their replacement desktop app they did so as closed source. You're absolutely right that people should be using llama.cpp - not only is it truly open source but it's significantly faster, has better model support, many more features, better maintained and the development community is far more active.
- parthsareen 10mo ago
- maranas 10mo agoCline + RooCode and VSCode already works really well with local models like qwen3-coder or even the latest gpt-oss. It is not as plug-and-play as Claude but it gets you to a point where you only have to do the last 5% of the work
- rynn 10mo agoWhat are you working on that you’ve had such great success with gpt-oss? I didn’t try it long because I got frustrated waiting for it to spit out wrong answers. But I’m open to trying again.
- maranas 10mo agoI use it to build some side-projects, mostly apps for mobile devices. It is really good with Swift for some reason. I also use it to start off MVP projects that involve both frontend and API development but you have to be super verbose, unlike when using Claude. The context window is also small, so you need to know how to break it up in parts that you can put together on your own
- embedding-shape 10mo ago> What are you working on that you’ve had such great success with gpt-oss? I'm doing programming on/off (mostly use Codex with hosted models) with GPT-OSS-120B, and with reasoning_effort set to high, it gets it right maybe 95% of the times, rarely does it get anything wrong.
- NelsonMinar 10mo ago"This particular [80B] model is what I’m using with 128GB of RAM". The author then goes on to breezily suggest you try the 4B model instead of you only have 8GB of RAM. With no discussion of exactly what a hit in quality you'll be taking doing that.
- ethmarks 10mo agoThis is like if an article titled "A guide to growing your own food instead of buying produce" explained that the author was using a four-acre plot of farmland but suggested that that reader could also use a potted plant instead. Absolutely baffling.
- Workaccount2 10mo agoI'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was. Somewhat comically, the author seems to have made it about 2 days. Out of 1,825. I think the real story is the folly of fixating your eyes on shiny new hardware and searching for justifications. I'm too ashamed to admit how many times I've done that dance... Local models are purely for fun, hobby, and extreme privacy paranoia. If you really want privacy beyond a ToS guarantee, just lease a server (I know they can still be spying on that, but it's a threshold.)
- ekjhgkejhgk 10mo agoI agree with everything you said, and yet I cannot help but respect a person who wants to do it himself. It reminds me of the hacker culture of the 80s and 90s.
- slicktux 10mo agoAgreed, Everyone seems to shun the DIY hacker now a days; saying things like “I’ll just pay for it”. It’s not about just NOT paying for it but doing it yourself and learning how to do it so that you can pass the knowledge on and someone else can do it.
- davidw 10mo agoI loathe the idea of being beholden to large corporations for what may be a key part of this job in the future.
- Eupolemos 10mo agoAnd we all know that enshittyfication is coming.
- ekjhgkejhgk 10mo agoExactly. Google doesn't show you what it knows is the most appropriate answer, it shows you a compromise between the most appropriate answer and the one that makes them the most money. Same thing will happen with these tools, just a matter of time.
- holyknight 10mo agoyour premise would've been right, if memory wouldn't skyrocketed like 400% in like 2 weeks.
- freeone3000 10mo agoWhat are you doing with these models that you’re going above free tier on copilot?
- satvikpendem 10mo agoSome just like privacy and working without internet, I for example travel regularly on the train and like to have my laptop when there's not always good WiFi.
- ardme 10mo agoIsnt the math of buying Nvidia stock with what you pay for all the hardware and then just paying $20 a month for codex with the annual returns better?
- phainopepla2 10mo agoIf you can see into the future and know the stock price, then sure.
- Muromec 10mo agoThe line only ever goes up, until we all cry and find a new false messiah. Or die
- andix 10mo agoI wouldn't run local models on the development PC. Instead run them on a box in another room or another location. Less fan noise and it won't influence the performance of the pc you're working on. Latency is not an issue at all for LLMs, even a few hundred ms won't matter. It doesn't make a lot of sense to me, except when working offline while traveling.
- snoman 10mo agoLess of a concern these days with hardware like a Mac Studio or Nvidia dgx which are accessible and aren’t noisy at all.
- andix 10mo agoI'm not fully convinced that those devices don't create noise at full power. But one issue still remains: LLMs eating up compute on the device you're working on. This will always be noticeable.
- raw_anon_1111 10mo agoI don’t think I’ve ever read an article where the reason I knew the author was completely wrong about all of their assumptions was that they admitted it themselves and left the bad assumptions in the article. The above paragraph is meant to be a compliment. But justifying it based on keeping his Mac for five years is crazy. At the rate things are moving, coding models are going to get so much better in a year, the gap is going to widen. Also in the case of his father where he is working for a company that must use a self hosted model or any other company that needed it, would a $10K Mac Studio with 512GB RAM be worth it? What about two Mac Studios connected over Thunderbolt using the newly released support in macOS 26? https://news.ycombinator.com/item?id=46248644 https://news.ycombinator.com/item?id=46248644
- baq 10mo agoYes, it’s worth it, if only because that Mac will be worth $20k in 3 months…
- john_minsk 10mo agoDo you think prices will go up for mac?
- phrotoma 10mo agobaq's comment is a joke about RAM prices.
- kergonath 10mo agoThat comment was a joke, but still. Resale prices for Macs are quite high. I didn’t run the calculation but it is entirely plausible the TCO including resale over a couple of years is much less than $200/month, if that’s the alternative.
- pseudony 10mo agoHe actually addressed your point by pointing out that, in his view, the models this machine can run today are the worst it’ll ever be - he thinks, and my own experience over the last year, is that local models improve and at a pace which is CLOSING the gap to commercial models.
- chrisischris 10mo ago[dead]
- hmokiguess 10mo agoThis seems really interesting. Reminds of IPFS but for AI
- SamDc73 10mo agoIf privacy is your top priority, then sure spend a few grand on hardware and run everything locally. Personally, I run a few local models (around 30B params is the ceiling on my hardware at 8k context), and I still keep a $200 ChatGPT subscription cause I'm not spending $5-6k just to run models like K2 or GLM-4.6 (they’re usable, but clearly behind OpenAI, Claude, or Gemini for my workflow) I was got excited about aescoder-4b (model that specialize in web design only) after its DesignArena benchmarks, but it falls apart on large codebases and is mediocre at Tailwind That said, I think there’s real potential in small, highly specialized models like 4B model trained only for FastAPI, Tailwind or a single framework. Until that actually exists and works well, I’m sticking with remote services.
- eblanshey 10mo agoWhat hardware can you buy for $5k to be able to run K2? That's a huge model.
- SamDc73 10mo agoThis older HN thread shows R1 running on a ~$2k box using ~512 GB of system RAM, no GPU, at ~3.5-4.25 TPS: https://news.ycombinator.com/item?id=42897205 https://news.ycombinator.com/item?id=42897205 If you scale that setup and add a couple of used RTX 3090s with heavy memory offloading, you can technically run something in the K2 class.
- nl 10mo agoIs 4 TPS actually useful for anything? That's around 350,000 tokens in a day. I don't track my Claude/Codex usage, but Kilocode with the free Grok model does and I'm using between 3.3M and 50M tokens in a day (plus additional usage in Claude + Codex + Mistral Vibe + Amp Coder) I'm trying to imagine a use case where I'd want this. Maybe running some small coding task overnight? But it just doesn't seem very useful.
- zarzavat 10mo ago3.5-50M tokens a day? What are you doing with all those tokens? Yesterday I asked Claude to write one function. I didn't ask it to do anything else because it wouldn't have been helpful.
- threethirtytwo 10mo agoI hope hardware becomes so cheap local models become the standard.
- rynn 10mo agoIt will be like the rest of computing, some things will move to the edge and others stay on the cloud. Best choice will depend on use cases.
- Terr_ 10mo agoI think the long term will depends on the legal/rent-seeking side. Imagine having the hardware capacity to run things locally, but not the necessary compliance infrastructure to ensure that you aren't committing a felony under the Copyright Technofeudalism Act of 2030.
- lelanthran 10mo ago> It will be like the rest of computing, some things will move to the edge and others stay on the cloud. It will become like cloud computing - some people will have a cloud bill of $10k/m to host their apps, other people would run their app on a $15/m VPS. Yes, the cost discrepancy will be as big as the current one we see in cloud services.
- layer8 10mo agoI hope that as well, but if cloud AI keeps buying up most of the world’s GPU and RAM production, it might not come to that.
- BoredPositron 10mo agoNot worth it yet. I run a 6000 black for image and video generation, but local coding models just aren't on the same level as the closed ones. I grabbed Gemini for $10/month during Black Friday, GPT for $15, and Claude for $20. Comes out to $45 total, and I never hit the limits since I toggle between the different models. Plus it has the benefit of not dumping too much money into one provider or hyper focusing on one model. That said, as soon as an open weight model gets to the level of the closed ones we have now, I'll switch to local inference in a heartbeat.
- ineedasername 10mo agoI’ve been using Qwen3 Coder 30b quantized down to IQ3_XSS to fit in < 16gb vram. Blazing fast 200+ tokens per second on a 4080. I don’t ask anything complicated, but one off scripts to do something I’d normally have to do manually by hand or take an hour to write the script myself? Absolutely. These are no more than a few dozen lines I can easily eyeball and verify with confidence- that’s done in under 60 seconds and leaves Claude code with plenty of quota for significant tasks.
- Bukhmanizer 10mo agoAre people really so naive to think that the price/quality of proprietary models is going to stay the same forever? I would guess sometime in the next 2-3 years all of the major AI companies are going to increase the price/enshittify their models to the point where running local models is really going to be worth it.
- jollymonATX 10mo agoThis is not really a guide to local coding models which is kinda disappointing. Would have been interested in a review of all the cutting edge open weight models in various applications.
- h0rmelchilly 10mo ago[dead]
- fny 10mo agoMy takeaway is that clock is ticking on Claude, Codex et al's AI monopoly. If a local setup can do 90% of what Claude can do today, what do things look like in 5 years?
- ukuina 10mo agoExactly, imagine what Claude can do in five years!
- rester324 10mo ago10% on top of what we have now and the same things that the local models can do of those times ahead of us?
- maranas 10mo agoI think they have already realized this, which is why they are moving towards tool use instead of text generation. Also explains why there are no more free APIs nowadays (even for search)
- artursapek 10mo agoImagine buying hardware that will be obsolete in 2 years instead of paying Anthropic $200 for $1000+ worth of tokens per month
- selcuka 10mo ago> Imagine buying hardware that will be obsolete in 2 years Unless the PC you buy is more than $4,800 (24 x $200) it is still a good deal. For reference, a MacBook M4 Max with 128GB of unified RAM is $4,699. You need a computer for development anyway, so the extra you pay for inference is more like $2-3K. Besides, it will still run the same model(s) at the same speed after that period, or even maybe faster with future optimisations in inference.
- hu3 10mo agoThe value depreciation of the hardware alone is going to be significant. Probably enough to pay for 3x ~$20 subscriptions to OpenAI, Anthropic and Gemini. Also, if you use the same mac to work, you can't reserve all 128GB for LLMs. Not to mention a mac will never run SOTA models like Opus 4.5 or Gemini 3.0 which subscriptions gives you. So unless you're ready to sacrifice quality and speed for privacy, it looks like a suboptimal arrangement to me.
- dchftcs 10mo agoI suspect depreciation will be a bit slower for a while, because there is a supply crunch.
- artursapek 10mo agoYeah, didn't even mention the fact that you can't Opus on your own hardware. Total waste of cash.
- dackdel 10mo agono one using exo?
- redrove 10mo agohttps://github.com/exo-explore/exo https://github.com/exo-explore/exo I keep hearing about it but unfortunately I myself only have one mac and nvidia GPUs and those can’t cluster together :/
- tempodox 10mo ago> You might need to install Node Package Manager for this. How anyone in this day and age can still recommend this is beyond me.
- jszymborski 10mo agoI just got a RTX 5090, so I thought I'd see what all the fuss was about these AI coding tools. I've previously copy pasted back and forth from Claude but never used the instruct models. So I fired up Cline with gpt-oss-120b, asked it to tell me what a specific function does, and proceeded to watch it run `cat README.md` over and over again. I'm sure it's better with other the Qwen Coder models, but it was a pretty funny first look.
- kelvie 10mo agogpt-oss-120b doesn't fit on a 5090 without offloading or crazy quants -- or did you mean you ran it via openrouter or something?
- kube-system 10mo agoSounds like a crazy quant. IME 2 bit quants are pretty dumb.
- jszymborski 10mo agoI'm running the MXFP4 [0] quants at like 10-13 toks/sec. It is actually really good, I'm starting to think its a problem with Cline since I just tried it with Qwen3 and the same thing happened. Turns out Cline _hates_ empty files in my projects, although they aren't required for this to happen. [0] https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-me https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-...
- Myrmornis 10mo agoCan anyone give any tips for getting something that runs fairly fast under ollama? It doesn't have to be very intelligent. When I tried gpt-oss and qwen using ollama on an M2 Mac the main problem was that they were extremely slow. But I did have a need for a free local model.
- am17an 10mo agoUse llama.cpp? I get 250 toks/sec on gpt-oss using a 4090, not sure about the mac speeds
- parthsareen 10mo agoHow much ram are you running with? Qwen3 and gpt-oss:20b punch a good bit above their weight. Personally use it for small agents.
- brainless 10mo agoI do not spend $100/month. I spend for 1 Claude Pro subscription and then a (much cheaper) z.ai Coding Plan, which is like one fifth the cost. I use Claude for all my planning, create task documents and hand over to GLM 4.6. It has been my workhorse as a bootstrapped founder (building nocodo, think Lovable for AI agents).
- alok-g 10mo agoI have heard about this approach elsewhere too. Could you please provide some more details on the set up steps and usage approach. I would like to replicate. Thanks.
- baconner 10mo agoThere are a couple of decent approaches to having a planning/reviewer model set (eg. claude, codex, gemini) and an execution model (eg. glm 4.6, flash models, etc) workflow that I've tried. All three of these will let you live in a single coding cli but swap in different models for different tasks easily. - claude code router - basically allows you to swap in other models using the real claude code cli and set up some triggers for when to use which one (eg. plan mode use real claude, non plan or with keywords use glm) - opencode - this is what im mostly using now. similar to ccr but i find it a lot more reliable against alt models. thinking tasks go to claude, gemini, codex and lesser execution tasks go to glm 4.6 (on ceberas). - sub-agent mcp - Another cool way is to use an mcp (or a skill or custom /command) that runs another agent cli for certain tasks. The mcp approach is neat because then your thinker agent like claude can decide when to call the execution agents, when to call in another smart model for a review of it's own thinking, etc instead of it being explicit choice from you. So you end up with the mcp + an AGENTS.md that instructs it to aggressively use the sub-agent mcp when it's a basic execution task, review, ... I also find that with this setup just being able to tap in an alt model when one is stuck, or get review from an alt model can help keep things unstuck and moving.
- KronisLV 10mo agoRooCode and KiloCode also have an Orchestrator mode that can create sub-tasks and you can specify which model to use for what - and since they report their results back after finishing a task (implement X, fix Y), the context of the more expensive model doesn’t get as polluted. Probably one of the most user friendly ways to do that. A simpler approach without subtasks would be to just use the smart model for Ask/Plan/whatever mode and the dumb but cheap one for the Code one, so the smart model can review the results as well and suggest improvements or fixes.
- altx 10mo agoIts interesting to notice that here https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... we default to measuring LLM coding performance as how long[~5h] a human task a model can complete with 50% success-rate (with 80% fall back for the second chart [~.5h]), while here it seems that for actual coding we really care about the last 90-100% of the costly model's performance.
- amarant 10mo agoBuying a maxed out MacBook Pro seems like the most expensive way to go about getting the necessary compute. Apple is notorious for overcharging for hardware, especially on ram. I bet you could build a stationary tower for half the price with comparable hardware specs. And unless I'm missing something you should be able to run these things on Linux. Getting a maxed out non-apple laptop will also be cheaper for comparable hardware, if portability is important to you.
- Renaud 10mo agoCan any x86 based system actually comes with that much unified memory? Not an Apple fanboy, but I was under the impression that having access to up to 512GB usable GPU memory was the main feature in favour of the mac. And now with Exo, you can even break the 512GB barrier.
- nl 10mo agoYou want unified RAM. On Linux your options are the NVidia Spark (and other vendor versions) or the AMD Ryzen AI series. These are good options, but there are significant trade-offs. I don't think there are Ryzen AI laptops with 128GB RAM for example, and they are pricey compared to traditional PCs. You also have limited upgradeability anyway - the RAM is soldered.
- kube-system 10mo agoYou need memory hooked up to the GPU. Apple’s unified memory is actually one of the cheaper ways to do this. On a typical x86-64 desktop, this means VRAM… for 100+ GB of VRAM you’re deep into tens of thousand of dollars. Also, if you think Apple’s RAM prices are crazy… you might be surprised at what current DDR5 pricing is today. The $800 that Apple charges to upgrade a MBP from 64-128GB is the current price of 64GB desktop DDR5-6000. Which is actually slower memory than the 8533 MT/s memory you’re getting in the MacBook.
- 2001zhaozhao 10mo agoUnder current prices buying hardware just to run local models is not worth it EVER, unless you already need the hardware for other reasons or you somehow value having no one else be able to possibly see your AI usage. Let's be generous and assume you are able to get a RTX 5090 at MSRP ($2000) and ignore the rest of your hardware, then run a model that is the optimal size for the GPU. A 5090 has one of the best throughputs in AI inference for the price, which benefits the local AI cost-efficiency in our calculations. According to this reddit post it outputs Qwen2.5-Coder 32B at 30.6 tokens/s. https://www.reddit.com/r/LocalLLaMA/comments/1ir3rsl/inference_speed_of_a_5090/ https://www.reddit.com/r/LocalLLaMA/comments/1ir3rsl/inferen... It's probably quantized, but let's again be generous and assume it's not quantized any more than models on OpenRouter. Also we assume you are able to keep this GPU busy with useful work 24/7 and ignore your electricity bill. At 30.6 tokens/s you're able to generate 993M output tokens in a year, which we can conveniently round up to a billion. Currently the cheapest Qwen2.5-Coder 32B provider on OpenRouter that doesn't train on your input runs it at $0.06/M input and $0.15/M output tokens. So it would cost $150 to serve 1B tokens via API. Let's assume input costs are similar since providers have an incentive to price both input and output proportionately to cost, so $300 total to serve the same amount of tokens as a 5090 can produce in 1 year running constantly. Conclusion: even with EVERY assumption in favor of the local GPU user, it still takes almost 7 years for running a local LLM to become worth it. (This doesn't take into account that API prices will most likely decrease over time, but also doesn't take into account that you can sell your GPU after the breakeven period. I think these two effects should mostly cancel out.) In the real world in OP's case, you aren't running your model 24/7 on your MacBook; it's quantized and less accurate than the one on OpenRouter; a MacBook costs more and runs AI models a lot slower than a 5090; and you do need to pay electricity bills. If you only change one assumption and run the model only 1.5 hours a day instead of 24/7, then the breakeven period already goes up to more than 100 years instead of 7 years. Basically, unless you absolutely NEED a laptop this expensive for other reasons, don't ever do this.
- rester324 10mo agoThese are the comments of the people who will cry a f@cking river when all the f@cking bubbles burst. You really think that it's "$300 total to serve the same amount of tokens as a 5090 can produce in 1 year running constantly"??? Maybe you forgot to read the news how much fucking money these companies are burning and losing each year. So these kind of comments as "to run local models is not worth it EVER" make me chuckle. Thanks for that!
- mungoman2 10mo agoThe money argument doesn't make sense here as that Mac depreciates more per month than the subscription they want to avoid. There may be other reasons to go local, but the proposed way is not cost effective.
- mungoman2 10mo agoThe money argument is IMHO not super strong, here as that Mac depreciates more per month than the subscription they want to avoid. There may be other reasons to go local, but I would say that the proposed way is not cost effective. There's also a fairly large risk that this HW may be sufficient now, but will be too small in not too long. So there is a large financial risk built into this approach. The article proposes using smaller/less capable models locally. But this argument also applies to online tools! If we use less capable tools even the $20/mo subscriptions won't hit their limit.
- m3kw9 10mo agoNobody doing serious coding will use local models when frontier models are that much better, and no they are not half a gen behind frontier. More like 2 gen.
- ikidd 10mo agoSo I can't see bothering with this when I pumped 260M tokens through running in Auto mode on a $20/mo Cursor plan. It was my first month of a paid subscription, if that means anything. Maybe someone can explain how this works for them? Frankly, I don't understand it at all, and I'm waiting for the other shoe to drop.
- lelanthran 10mo ago> So I can't see bothering with this when I pumped 260M tokens through running in Auto mode on a $20/mo Cursor plan. It was my first month of a paid subscription, if that means anything. Maybe someone can explain how this works for them? They're running at a loss and covering up the losses using VC? > Frankly, I don't understand it at all, and I'm waiting for the other shoe to drop. I think that the providers are going to wait until there are a significant number of users that simply cannot function in any way without the subscription, and then jack up the prices. After all, I can all but guarantee that even the senior devs at most places now won't be able to function if every single tool or IDE provided by a corporate (like VSCode) was yanked from them. Myself, you can scrub my main dev desktop of every corporate offering, and I might not even notice (emacs or neovim, plugins like Slime, Lsp plugins, etc) is what I am using daily, along with programming languages.
- NumberCruncher 10mo agoI am freelancing on the side and charge 100€ by the hour. Spending roughly 100€ per month on AI subscriptions has a higher ROI for me personally than spending time on reading this article and this thread. Sometimes we forget that time is money...
- dhruv3006 10mo agor/locallama has very good discussion for this!
- jwr 10mo agoI am still hoping, but for the moment… I have been trying every 30-80B model that came out in the last several months, with crush and opencode, and it's just useless. They do produce some output, but it's nowhere near the level that claude code gets me out of the box. It's not even the same league. With LLMs, I feel like price isn't the main factor: my time is valuable, and a tool that doesn't improve the way I work is just a toy. That said, I do have hope, as the small models are getting better.
- larodi 10mo agoClaude Code is a lot about prompting and orchestration of the conversation. The LLM is just a tool in these agentic frameworks. Whats truly ingenious is how context is engineered/managed, how is the code-RAG approached, and them LLM memory that is used. So my guess would be - we need open conversation or something along the line of "useful linguistic-AI approaches for combing and grooming code"
- jwr 10mo agoAgreed. I've been trying to use opencode and crush, and none of them do anything useful for me. In contrast, claude code "just works" and does genuinely useful work. And it's not just because of the specific LLM used, it's the overall engineering of the tool, the prompt behind the scenes, etc. But the bottom line is that I still can't find a way to use either local LLMs and/or opencode and crush for coding.
- larodi 10mo agoWhich is very sad and perhaps she should be aiming to introduce some very smart linguists into the whole ML:LLM thing that can learn and explore how to best to interact with the funny archive that models are.
- sbene970 10mo agoSearch for "Claude Code Router" on GitHub, which you can use to route any models through Claude Code.
- 10mo ago
- flowinghorse 10mo agoLocal models less than 2b are good enough for code auto completion. Even you don't have 128G memory.
- avhception 10mo agoI tried local models for general-purpose LLM tasks on my Radeon 7800 XT (20GB VRAM), and was disappointed. But I keep thinking: It should be possible to run some kind of supercharged tab completion on there, no? I'm spending most of my time writing Ansible or in the shell, and I have a feeling that even a small local model should give me vastly more useful completion options...
- dfischer96 10mo agoNice guide! I want to point out opencode CLI, which is far superior to Qwen CLI in my opinion.
- throw-12-16 10mo agoI never see devs containerize their coding agents. It seems so obvious to me, but I guess people are happy with claude living in their home directory and slurping up secrets.
- onion2k 10mo agoThe devs I work with don't put secrets in their home directories. ;)
- throw-12-16 10mo agomany many tools default to this, claude included
- littlestymaar 10mo agoAnd where are all their software putting their data then? Unless you consider only private keys to be secrets… (In particular the fact that Claude Code has access to your Anthropic API key is ironic given that Dario and Anthropic spend a lot of time fearmongering about how the AI could go rogue and “attempt to escape”).
- rester324 10mo agoHow do you know? Do you snoop on their work machines?
- bjt12345 10mo agoHere's my take on it though... Just as we had the golden era of the internet in the late 90s, when the WWW was an eden of certificate-less homepages with spinning skulls on geocities without ad tracking, we are now in the golden era of agentic coding where massive companies make eye watering losses so we can use models without any concerns. But this won't last and Local Llamas will become a compelling idea to use, particularly when there will be a big second hand market of GPUs from liquidated companies.
- yread 10mo agoYep, when the tide goes away no company will be able to keep swimming naked offering stuff for free
- aleggg 10mo agoYes. This heavily subsidized LLM inference usage will not last forever. We have already seen cost cutting for some models. A model starts strong, but over time the parent company switches to heavily quantized versions to save on compute costs. Companies are bleeding money, and eventually this will need to adjust, even for a behemoth like Google. That is why running local models is important.
- sesm 10mo agoUnfortunately, GPUs die in datacenters very quickly, and GPU manufacturers don't care about hardware longevity.
- stuaxo 10mo agoIs the conclusion the same if you have a computer that is just for the LLM, and a separate one that runs your dev tools ?
- KronisLV 10mo agoMy experience: even for the run of the mill stuff, local models are often insufficient, and where they would be sufficient, there is a lack of viable software. For example, simple tasks CAN be handled by Devstral 24B or Qwen3 30B A3B, but often they fail at tool use (especially quantized versions) and you often find yourself wanting something bigger, where the speed falls a bunch. Even something like zAI GLM 4.6 (through Cerebras, as an example of a bigger cloud model) is not good enough for doing certain kinds of refactoring or writing certain kinds of scripts. So either you use local smaller models that are hit or miss, or you need a LOT of expensive hardware locally, or you just pay for Claude Code, or OpenAI Codex, or Google Gemini, or something like that. Even Cerebras Code that gives me a lot of tokens per day isn't enough for all tasks, so you most likely will need a mix - but running stuff locally can sometimes decrease the costs. For autocomplete, the one thing where local models would be a nearly perfect fit, there just isn't good software: Continue.dev autocomplete sucks and is buggy (Ollama), there don't seem to be good enough VSC plugins to replace Copilot (e.g. with those smart edits, when you change one thing in a file but have similar changes needed like 10, 25 and 50 lines down) and many aren't even trying - KiloCode had some vendor locked garbage with no Ollama support, Cline and RooCode aren't even trying to support autocomplete. And not every model out there (like Qwen3) supports FIM properly, so for a bit I had to use Qwen2.5 Coder, meh. Then when you have some plugins coming out, they're all pretty new and you also don't know what supply chain risks you're dealing with. It's the one use case where they could be good, but... they just aren't. For all of the billions going into AI, someone should have paid a team of devs to create something that is both open (any provider) and doesn't fucking suck. Ollama is cool for the ease of use. Cline/RooCode/KiloCode are cool for chat and agentic development. OpenCode is a bit hit or miss in my experience (copied lines getting pasted individually), but I appreciate the thought. The rest is lacking.
- evanreichard 10mo agoHave you tried llama.vscode [0]? I use the vim equivalent, llama.vim [1] with Qwen3 Coder 30B and personally feel that it's better than Copilot. I have hot keys that allow me to quickly switch between the two and find myself always going back to local. [0] https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode [1] https://github.com/ggml-org/llama.vim https://github.com/ggml-org/llama.vim
- Simplita 10mo agoOne thing that surprised us when testing local models was how much easier debugging became once we treated them as decision helpers instead of execution engines. Keeping the execution path deterministic avoided a lot of silent failures. Curious how others are handling that boundary.
- Roark66 10mo agoI found the winning combination is to use all of them in this way: - first you need a vendor agnostic tool like opencode (I had to add my own vendors as it didn't support it out of the box properly) - second you set up agents with different models. I use: - for architecture and planning - opus, Sonet, gpt 5.2, gemini3 (depending on specifics, for example I found got better in troubleshooting, Sonet better in pure code planning, opus better in DevOps, Gemini the best for single shot stuff) - for execution of said plans (Qwen 2.5 Coder 30B - yes, it's even better in my use cases than Qwen3 despite benchmarks, Sonet - only when absolutely necessary, Qwen3-235B - between Qwen 2.5 and Sonet) - verification (Gemini 3 flash, Qwen3-480B etc) The biggest saving you make is by making the context smaller and where many turns are required going for smaller models. For example a single 30min troubleshooting session with Gemini 3 can cost $15 if you run it "normally" or it can cost $2 if you use the agents, wipe context after most turns (can be done thanks to tracking progress in a plan file)
- d4rkp4ttern 10mo agoI recently found myself wanting to use Claude Code and Codex-CLI with local LLMs on my MacBook Pro M1 Max 64GB. This setup can make sense for cost/privacy reasons and for non-coding tasks like writing, summarization, q/a with your private notes etc. I found the instructions for this scattered all over the place so I put together this guide to using Claude-Code/Codex-CLI with Qwen3-30B-A3B, 80B-A3B, Nemotron-Nano and GPT-OSS spun up with Llama-server: https://github.com/pchalasani/claude-code-tools/blob/main/docs/local-llm-setup.md https://github.com/pchalasani/claude-code-tools/blob/main/do... Llama.cpp recently started supporting Anthropic’s messages API for some models, which makes it really straightforward to use Claude Code with these LLMs, without having to resort to say Claude-Code-Router (an excellent library), by just setting the ANTHROPIC_BASE_URL.
- bubbi 10mo ago[dead]
- bearjaws 10mo agoI am sorry but anyone who actually has tried this knows it is horrifically slow, significantly slower than you just typing for any model worth its weight. That 128gb of RAM is nice but the time to first token is so long on any context over 32k, and the results are not even close to a Codex or Sonnet.
- embedding-shape 10mo ago> because GPT-OSS frequently gave me “I cannot fulfill this request” responses when I asked it to build features. This is something that frequently comes up and whenever I ask people to share the full prompts, I'm never able to reproduce this locally. I'm running GPT-OSS-120B with the "native" weights in MXFP4, and I've only seen "I cannot fulfill this request" when I actually expect it, not even once had that happen for a "normal" request you expect to have a proper response for. Has anyone else come across this when not using the lower quantizations or 20b (So GPT-OSS-120B proper in MXFP4) and could share the exact developer/system/user prompt that they used that triggered this issue? Just like at launch, from my point of view, this seems to be a myth that keeps propagating, and no one can demonstrate a innocent prompt that actually triggers this issue on the weights OpenAI themselves published. But then the author here seems to actually have hit that issue but again, no examples of actual prompts, so still impossible to reproduce this issue.
- bilater 10mo agoIf you are using local models for coding you are midwiting this. Your code should be worth more than a subscription. The only legit use case for local models is privacy. I don't know why anyone would want to code with an intern level model when they can get a senior engineer level model for a couple of bucks more. It DOESN'T MATTER if you're writing a simple hello world function or building out a complex feature. Just use the f*ing best model.
- jgalt212 10mo agoI will use a local coding model for our proprietary / trade secrets internal code when Google uses Claude for its internal code and Microsoft starts using Gemini for internal code. The flip side of this coin is I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets.
- Aurornis 10mo ago> I'd be very excited if Jane Street or DE Shaw were running their trading models through Claude. Then I'd have access to billions of dollars of secrets. Using Claude for inference does not mean the codebase gets pulled into their training set. This is a tired myth that muddies up every conversation about LLMs
- jgalt212 10mo ago> This is a tired myth that muddies up every conversation about LLMs Many copyright holders, and the courts would beg to differ.
- bilater 10mo agolol yeah its weird to me why even ppl on HN can't wrap their heads around stateless calls.
- jgalt212 10mo agounless you control both the client and the server, you cannot prove a call is stateless.
- yoan9224 10mo agoThe cost analysis here is solid, but it misses the latency and context window trade-offs that matter in practice. I've been running Qwen2.5-Coder locally for the past month and the real bottleneck isn't cost - it's the iteration speed. Claude's 200k context window with instant responses lets me paste entire codebases and get architectural advice. Local models with 32k context force me to be more surgical about what I include. That said, the privacy argument is compelling for commercial projects. Running inference locally means no training data concerns, no rate limits during critical debugging sessions, and no dependency on external API uptime. We're building Prysm (analytics SaaS) and considered local models for our AI features, but the accuracy gap on complex multi-step reasoning was too large. We ended up with a hybrid: GPT-4o-mini for simple queries, GPT-4 for analysis, and potentially local models for PII-sensitive data processing. The TCO calculation should also factor in GPU depreciation and electricity costs. A 4090 pulling 450W at $0.15/kWh for 8 hours/day is ~$200/year just in power, plus ~$1600 amortized over 3 years. That's $733/year before you even start inferencing. You need to be spending $61+/month on Claude to break even, and that's assuming local performance is equivalent.
- estimator7292 10mo agoI'd only consider the GPU cost if you intend to chuck it in a dumpster after three years. Why not factor in the cost of your CPU and amortize your RAM and disks? Those aren't useful numbers.
- SpaceManNabs 10mo agoI love that this article added a correction and took ownership in it. This encourages more people to blog stuff and then get more input for parts they missed. The best way to get the correct answer on something is posting the wrong thing. Not sure where I got this from, but I remember it was in the context of stackoverflow questions getting the correct answer in the comments of a reply :) Props to the author for their honesty and having the impetus to blog about this in the first place.
- elestor 10mo agoyeah my 4GB of vram isn't gonna cut it
- Ultimatt 10mo agoFor local MLX inference LM Studio is a much nicer option than Ollama
- lucideng 10mo agoA Mac dev type using a 5-year-old machine, I will believe it when I see it. I know a few older Macs still kicking around, but those people use them for basic stuff, not actual work. Mac people jump to new models faster than Taco Bell leaves my body.
- j45 10mo agoThe work and interest in local coding models reminds me of the early 3D printer community, whatever is possible may take more than average tinkering until someone makes it a lot more possible.
- erusev 10mo agoIf you're on a Mac and want a simple and open-source way to run models locally, check out our app LlamaBarn: https://github.com/ggml-org/LlamaBarn https://github.com/ggml-org/LlamaBarn
- ljosifov 10mo agoNah - given the ergonomics + economics, local coding models are not atm that viable. I like all things local even if just for safety of keeping healthy competitive ecosystem. And I can imagine really specialised uses cases where I run an 8B not-so-smart model to process oodles of data on my local 7900xtx or similar. Got older m2 mbp with 96gb (v)ram and try all things local that fit. Usually LMStudio for the speed add in MLX format models on ASI (as end point; plus chat for vibes test; LMStudio omission from the OP blog post makes me question the post), or llama.cpp for GGUF (llama.cpp is the OG; excellent and universal engine and format; recently got even better). Looking at how agents work - an agent smarts of Claude Code or Codex in using the tools feels like it's half its success (the other half the underlying LLM smarts). From the training on baked in 'Tool Use & Interleaved Thinking' on the right tools in a right way, to the trivial 'DONOTDO bad idea to fill your 100K useful context with random content of multi-MB file as prompt'. The $20/mo plans are insanely competitive. OpenaI is generous with Codex, and in addition to terminal that I mostly use, there is the VSCode addon as well as use in Cline or Roo. Cursor offers in-house model fast and good, insane economy reading large codebases, as well BYOK to latest-greatest LLMs afaik. Claude Code $20/mo is stingy with quotas, but can be supplement with Z.ai standing in - glm-4.7 as of yesterday (saw no difference glm-4.6 v.v. sonnet-4.5 already v.good). It's a 3 lines change to ~/.claude/settings.json to flip Z.ai-Anthropic back and forth at will (e.g. when paused on one to switch to the other). Have not tried the Cerebras high tok/s but wd love to - not waiting makes a ton of difference to productivity.