6 ms·

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines)

• zozbot234 9mo ago
  The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.
  • reilly3000 9mo ago
    Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…
    • teaearlgraycold 9mo ago
      Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.
      • blharr 9mo ago
        What speed are you getting at that level of hardware though?
      • SchemaLoad 9mo ago
        It's still very expensive compared to using the hosted models which are currently massively subsidised. Have to wonder what the fair market price for these hosted models will be after the free money dries up.
        • cactusplant7374 9mo ago
          Inference is profitable. Maybe we hit a limit and we don't need as many expensive training runs in the future.
          • teaearlgraycold 9mo ago
            For sure Claude Code isn’t profitable
            • bdangubic 9mo ago
              Neither was Uber and … and …
              • plagiarist 9mo ago
                Businesses will desire me for my insomnia once Anthropics starts charging congestion pricing.
                • bdangubic 9mo ago
                  that is coming for sure to replace the "500" errors
          • paxys 9mo ago
            Inference APIs are probably profitable, but I doubt the $20-$100 monthly plans are.
            • cactusplant7374 9mo ago
              I wouldn’t be so sure. Most users aren’t going to use up their quota every week.
        • whatsupdog 9mo ago
          I wonder if the "distributed AI computing" touted by some of the new crypto projects [0] works and is relatively cheaper. 0. https://www.daifi.ai/ https://www.daifi.ai/
    • PlatoIsADisease 9mo ago
      [flagged]
      • zozbot234 9mo ago
        70B dense models are way behind SOTA. Even the aforementioned Kimi 2.5 has fewer active parameters than that, and then quantized at int4. We're at a point where some near-frontier models may run out of the box on Mac Mini-grade hardware, with perhaps no real need to even upgrade to the Mac Studio.
        • PlatoIsADisease 9mo ago
          >may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.
          • K0balt 9mo ago
            Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.
            • PlatoIsADisease 9mo ago
              >quants >moderate context windows Really had to modify the problem to make it seem equal? Not that quants are that bad, but the context windows thing is the difference between useful and not useful.
              • K0balt 9mo ago
                Equal to the 2x3090? Yeah it’s about equal in every way, context windows included. As for useful at that scale? I use mine for coding a fair bit, and I don’t find it a detractor overall. It enforces proper API discipline, modularity, and hierarchal abstraction. Perhaps the field of application makes that more important though. (Writing firmware and hardware drivers). It also brings the advantage of focusing exclusively on the problems that are presented in the limited context, and not wandering off on side quests that it makes up. I find it works well up to about 1KLOC at a time. I wouldn’t imply they were equal to commercial models, but I would definitely say that local models are very useful tools. They are also stable, which is not something I can say for SOTA models. You cal learn how to get the best results from a model and the ground doesn’t move underneath you just when you’re on a roll.
    • bigyabai 9mo ago
      "Full quality" being a relative assessment, here. You're still deeply compute constrained, that machine would crawl at longer contexts.
    • 0xbadcafebee 9mo ago
      Most benchmarks show very little improvement of "full quality" over a quantized lower-bit model. You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use.
      • MuffinFlavored 9mo ago
        > You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use. Are there a lot of options how "how far" do you quantize? How much VRAM does it take to get the 92-95% you are speaking of?
        • bigyabai 9mo ago
          > Are there a lot of options how "how far" do you quantize? So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/ https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie... > How much VRAM does it take to get the 92-95% you are speaking of? For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.
          • MuffinFlavored 9mo ago
            Thank you. Could you give a tl;dr on "the full model needs ____ this much VRAM and if you do _____ the most common quantization method it will run in ____ this much VRAM" rough estimate please?
            • omneity 9mo ago
              It’s a trivial calculation to make (+/- 10%). Number of params == “variables” in memory VRAM footprint ~= number of params * size of a param A 4B model at 8 bits will result in 4GB vram give or take, same as params. At 4 bits ~= 2GB and so on. Kimi is about 512GB at 4 bits.
    • deaux 9mo ago
      And that's at unusable speeds - it takes about triple that amount to run it decently fast at int4. Now as the other replies say, you should very likely run a quantized version anyway.
    • deleted 9mo ago
      [deleted]
    • polynomial 9mo ago
      Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment https://buildai.substack.com/i/181542049/the-mac-mini-moment
      • danw1979 9mo ago
        I did not expect this to be a limiting factor in the mac mini RDMA setup ! - > Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics. Thermal throttling of network cables is a new thing to me…
        • cat_plus_plus 9mo ago
          I admire patience of anyone who runs dense models on unified memory. Personally, I would rather feed an entire programming book or code directory to a sparse model and get an answer in 30 seconds and then use cloud in rare cases it's not enough.
        • polynomial 9mo ago
          Luckily we're having a record cold winter and your setup can double as a personal space heater.
  • teaearlgraycold 9mo ago
    Having used K2.5 I’d judge it to be a little better than that. Maybe as good as proprietary models from last June?
  • paxys 9mo ago
    LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.
    • DennisP 9mo ago
      On Macbooks, no. But there are a few lunatics like this guy: https://www.youtube.com/watch?v=bFgTxr5yst0 https://www.youtube.com/watch?v=bFgTxr5yst0
      • HarHarVeryFunny 9mo ago
        Wow! I've never heard of this guy before, but I see he's got 5M YouTube subscribers, which I guess is the clout you need to have Apple loan (I assume) you $50K worth of Mac Studios! I'll be interesting to see how model sizes, capability, and local compute prices evolve. A bit off topic, but I was in best buy the other day and was shocked to see 65" TVs selling for $300 ... I can remember the first large flat screen TVs (plasma?) selling for 100x that ($30K) when they first came out.
      • danw1979 9mo ago
        He must be mad, accepting $50k of free (probably loaned?) hardware from Apple ! Great demo video though. Nice to see some benchmarks of Exo with this cluster across various models.
    • deaux 9mo ago
      Some people spend $50k on a new car, others spend it on running Kimi K2.5 at good speeds locally. No one's running Sonnet/Gemini/GPT-5 locally though.
  • corysama 9mo ago
    The article mentions https://unsloth.ai/docs/basics/claude-codex https://unsloth.ai/docs/basics/claude-codex I'll add on https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next The full model is supposedly comparable to Sonnet 4.5 But, you can run the 4 bit quant on consumer hardware as long as your RAM + VRAM has room to hold 46GB. 8 bit needs 85.
    • deleted 9mo ago
      [deleted]
  • 0xbadcafebee 9mo ago
    Kimi K2.5 is fourth place for intelligence right now. And it's not as good as the top frontier models at coding, but it's better than Claude 4.5 Sonnet. https://artificialanalysis.ai/models https://artificialanalysis.ai/models
  • Aurornis 9mo ago
    > The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago Kimi K2.5 is a trillion parameter model. You can't run it locally on anything other than extremely well equipped hardware. Even heavily quantized you'd still need 512GB of unified memory, and the quantization would impact the performance. Also the proprietary models a year ago were not that good for anything beyond basic tasks.
• nik282000 9mo ago
  > intelligence Whether it's a giant corporate model or something you run locally, there is no intelligence there. It's still just a lying engine. It will tell you the string of tokens most likely to come after your prompt based on training data that was stolen and used against the wishes of its original creators.
• bicx 9mo ago
  Exactly. The comparison benchmark in the local LLM community is often GPT _3.5_, and most home machines can’t achieve that level.
• dheera 9mo ago
  Maybe add to the Claude system prompt that it should work efficiently or else its unfinished work will be handed off to to a stupider junior LLM when its limits run out, and it will be forced to deal with the fallout the next day. That might incentivize it to perform slightly better from the get go.
  • kridsdale3 9mo ago
    "You must always take two steps forward, for when you are off the clock, your adversary will take one step back."
• bityard 9mo ago
  Correct, a rack full of datacenter equipment is not going to compete with anything that fits on your desk or lap. Well spotted. But as a counterpoint: there are whole communities of people in this space who get significant value from models they run locally. I am one of them.
  • deleted 9mo ago
    [deleted]
  • Gravey 9mo ago
    Would you mind sharing your hardware setup and use case(s)?
    • CamperBob2 9mo ago
      Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all. It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its weight class for an 80B model.
      • zozbot234 9mo ago
        IIRC, that new Qwen model has 3B active parameters so it's going to run well enough even on far less than 96GB VRAM. (Though more VRAM may of course help wrt. enabling the full available context length.) Very impressive work from the Qwen folks.
      • redwood_ 9mo ago
        Agree, these new models are a game changer. I switched from Claude to Qwen3-Coder-Next for day-to-day on dev projects and don't see a big difference. Just use Claude when I need comprehensive planning or review. Running Qwen3-Coder-Next-Q8 with 256K context.
      • paxys 9mo ago
• DANmode 9mo ago
  and you really should be measuring based on the worst-case scenario for tools like this.
• richstokes 9mo ago
  This. It's a false economy if you value your time even slightly, pay for the extra tokens and use the premium models.
• 0xbadcafebee 9mo ago
  The best local models are literally right behind Claude/Gemini/Codex. Check the benchmarks. That said, Claude Code is designed to work with Anthropic's models. Agents have a buttload of custom work going on in the background to massage specific models to do things well.
  • girvo 9mo ago
    The benchmarks simply do not match my experience though. I don’t put that much stock in them anymore.
    • Balinares 9mo ago
      I've repeatedly seen Opus 4.5 manufacture malpractice and then disable the checks complaining about it in order to be able to declare the job done, so I would agree with you about benchmarks versus experience.
• andai 9mo ago
  Yeah this is why I ended up getting Claude subscription in the first place. I was using GLM on ZAI coding plan (jerry rigged Claude Code for $3/month), but finding myself asking Sonnet to rewrite 90% of the code GLM was giving me. At some point I was like "what the hell am I doing" and just switched. To clarify, the code I was getting before mostly worked, it was just a lot less pleasant to look at and work with. Might be a matter of taste, but I found it had a big impact on my morale and productivity.
  • MuffinFlavored 9mo ago
    Did you eventually move to a $20/mo Claude plan, $100/mo Claude plan, $200/mo, or API based? if API based, how much are you averaging a month?
    • andai 9mo ago
      The $20 one, but it's hobby use for me, would probably need the $200 one if I was full time. Ran into the 5 hour limit in like 30 minutes the other day. I've also been testing OpenClaw. It burned 8M tokens during my half hour of testing, which would have been like $50 with Opus on the API. (Which is why everyone was using it with the sub, until Anthropic apparently banned that.) I was using GLM on Cerebras instead, so it was only $10 per half hour ;) Tried to get their Coding plan ("unlimited" for $50/mo) but sold out... (My fallback is I got a whole year of GLM from ZAI for $20 for the year, it's just a bit too slow for interactive use.)
      • lostmsu 9mo ago
        Try Codex. It's better (subjectively, but objectively they are in the same ballpark), and its $20 plan is way more generous. I can use gpt-5.2 on high (prefer overall smarter models to -codex coding ones) almost nonstop, sometimes a few in parallel before I hit any limits (if ever).
    • holoduke 9mo ago
      I now have 3 x 100 plans. Only then I an able to full time use it. Otherwise I hit the limits. I am q heavy user. Often work on 5 apps at the same time.
• mycall 9mo ago
  There is tons of improvements in the near future. Even Claude Code developer said he aimed at delivering a product that was built for future models he betted would improve enough to fulfill his assumptions. Parallel vLLM MoE local LLMs on a Strix Halo 128GB has some life in it yet.
• seanmcdirmid 9mo ago
  > (ones you run on beefy 128GB+ RAM machines) PC or Mac? A PC, ya, no way, not without beefy GPUs with lots of VRAM. A mac? Depends on the CPU, an M3 Ultra with 128GB of unified RAM is going to get closer, at least. You can have decent experiences with a Max CPU + 64GB of unified RAM (well, that's my setup at least).
  • QuantumNomad_ 9mo ago
    Which models do you use, and how do you run them?
    • seanmcdirmid 9mo ago
      I have a M3 max 64GB. For VS Code code completion in Continue using a Qwen3-coder 7b model. For CLI work Qwen coder 32b for sidebar. 8 bit quant for both. I need to take a look at Qwen3-coder-next, it is supposed to have made things much faster with a larger model.
• anon373839 9mo ago
  It's true that open models are a half-step behind the frontier, but I can't say that I've seen "sheer intelligence" from the models you mentioned. Just a couple of days ago Gemini 3 Pro was happily writing naive graph traversal code without any cycle detection or safety measures. If nothing else, I would have thought these models could nail basic algorithms by now?
  • cracki 9mo ago
    Did it have reason to assume the graph to be a certain type, such as directed or acyclic?
• mlrtime 9mo ago
  The local ones yeah... I have claude pro $20/mo and sometimes run out. I just set ANTHROPIC_BASE_URL to a localllm API endpoint that connects to a cheaper Openai model. I can continue with smaller tasks with no problem. This has been done for a long time.
• majormajor 9mo ago
  The amount of "prompting" stuff (meta-prompting?) the "thinking" models do behind the scenes even beyond what the harnesses do is massive; you could of course rebuild it locally, but it's gonna make it just that much slower. I expect it'll come along but I'm not gonna spend the $$$$ necessary to try to DIY it just yet.
• deleted 9mo ago
  [deleted]
• acchow 9mo ago
  I agree. You could spin for 100 hours on a sub-par model or get it done in 10 minutes with a frontier model
• EagnaIonat 9mo ago
  The secret is to not run out of quota. Instead have Claude know when to offload work to local models and what model is best suited for the job. It will shape the prompt for the model. Then have Claude review the results. Massive reduction in costs. btw, at least on Macbooks you can run good models with just M1 32GB of memory.
  • BuildTheRobots 9mo ago
    I don't suppose you could point to any resources on where I could get started. I have a M2 with 64gb of unified memory and it'd be nice to make it work rather than burning Github credits.
    • EagnaIonat 9mo ago
      https://ollama.com https://ollama.com Although I'm starting to like LMStudio more, as it has more features that Ollama is missing. https://lmstudio.ai https://lmstudio.ai You can then get Claude to create the MCP server to talk to either. Then a CLAUDE.md that tells it to read the models you have downloaded, determine their use and when to offload. Claude will make all that for you as well.
      • shen 9mo ago
        Which local models are you using for the 32gb MacBooks?
        • EagnaIonat 9mo ago
          Mainly gpt-oss-20b as the thinking mode is really good. I occasionally use granite4 as it is a very fast model. But any 4GB model should easily be used.
    • eek2121 9mo ago
      LM Studio is fantastic for playing with local models.
  • kilroy123 9mo ago
• amelius 9mo ago
  And at best?
• cat_plus_plus 9mo ago
  Depends on whether you want a programmer or a therapist. Given clear description of class structure and key algorithms, Qwen3-Code is way more likely to do exactly what is being asked than any Gemini model. If you want to turn a vague idea into a design, yeah cloud bot is better. Let's not forget that cloud bots have web search, if you hook up a local model to GPT Researcher or Onyx frontend, you will see reasonable performance, although open ended research is where cloud model scale does pay off. Provided it actually bothers to search rather than hallucinating to save backend costs. Also local uncensored model is way better at doing proper security analysis of your app / network.
• altern8 9mo ago
  I was wondering the same thing, e.g. if it takes tens or hundreds of millions of dollars to train and keep a model up-to-date, how can an open source one compete with that?
  • gpm 9mo ago
    Less than a billion of dollars to become the arbiter of truth probably sounds like a great deal to the well off dictatorial powers of the world. So long as models can be trained to have a bias (and it's hard to see that going away) I'd be pretty surprised if they stop being released for free. Which definitely has some questionable implications... but just like with advertising it's not like paying makes the incentives for the people capable of training models to put their thumbs on the scales go away.
• tracker1 9mo ago
  For my relatively limited exposure, I'm not sure if I'd be able to tolerate it. I've found Claude/Opus to e pretty nice to work with... by contrast, I find Github Copilot to be the most annoying thing I've ever tried to work with. Because of how the plugin works in VS code, on my third day of testing with Claude Code, I didn't click the Claude button and was accidentally working with CoPilot for about three hours of torture when I realized I wasn't in Claude Code. Will NEVER make that mistake again... I can only imagine anything I can run at any decent speed lcoally will be closer to the latter. I pretty quickly reach a "I can do this faster/better myself" point... even a few times with Claude/Opus, so my patience isn't always the greatest. That said, I love how easy it is to build up a scaffold of a boilerplate app for the sole reason to test a single library/function in isolation from a larger application. In 5-10 minutes, I've got enough test harness around what I'm trying to work on/solve that it lets me focus on the problem at hand, while not worrying about doing this on the integrated larger project. I've still got some thinking and experimenting to do with improving some of my workflows... but I will say that AI Assist has definitely been a multiplier in terms of my own productivity. At this point, there's literally no excuse not to have actual code running experiments when learning something new, connecting to something you haven't used before... etc. in terms of working on a solution to a problem. Assuming you have at least a rudimentary understanding of what you're actually trying to accomplish in the piece you are working on. I still don't have enough trust to use AI to build a larger system, or for that matter to truly just vibe code anything.