3 ms·
Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then,
by adamtaylor_13 6mo ago
Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you.
Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.
- binyu 6mo agoDeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.
- tuananh 6mo agoFrom what I read, ds v4 is very close with opus 4.6 performance.
- DeathArrow 6mo agoThe full model is, not the quantized versions.
- tuananh 6mo agoyeah that goes without saying. how can openweight, quantized version beat SOTA :)
- array_key_first 6mo agoWell it depends on the task. For agentic coding, more is more, but for tasks that normal consumers use them for there really is a ceiling. OCR, text to speech, that type of thing doesn't really improve when going to a SOTA model, so you'd just be wasting your money. I think local LLMs have more value than software engineers give them credit for.
- tuananh 6mo agototally agree with that. local llm doesn't need to match SOTA performance in order to be useful.
- Aurornis 6mo agoRunning DeepSeek V4 without extreme quantization locally requires a lot of hardware. The IQ2 quants that fit into 128GB machines are very degraded.
- binyu 6mo agoThat is true, it is a 1.6T parameters model so it requires a great deal of memory. I also heard there's a 2bit quantization that works well on Apple metal.
- thefounder 6mo agoNext year there will be Opus 4.5 level available on open source models so theoretically you may be able to run it locally but in reality it will be too expensive (i.e maybe 2 x max Studio 512GB ram each) for “normal” users.
- storus 6mo agoDepending on a task, there are already models matching Opus 4.5. Just not in everything. But you can always swap a local model for a particular task.
- bugglebeetle 6mo agoThe frontier Chinese open source models are already at this level, GLM-5.1 and Kimi K2.6 specifically.
- DeathArrow 6mo agoBut you can't run the locally at full quality. And quantized versions you can run locally are a far cry from Opus 4.6.
- bugglebeetle 6mo agoAnthropic serves quantized versions of their models and you can run q8 locally.
- nicce 6mo agoI don't even use Sonnet anymore. Current feels worse than Claude 3.5 couple years ago. They have quantized that much? Switched to GPT 5.5, let's see how long it will stay good.
- greenavocado 6mo agoThe problem with GPT is it doesn't have the same wholistic worldview of the problem space as Claude
- nicce 6mo agoHmm, what that means or how can you even measure it? At least for all my recent problems, GPT 5.5 has performed better.
- Aurornis 6mo agoI experiment a lot with local models, and I agree. I have a lot of fun with the local models and seeing what they can do. I appreciate the SOTA models even more after my local experiments. The local models are really impressive these days, but the gap to SOTA is huge for complex tasks.
- ThunderSizzle 5mo agoWhat if you split it into less complex tasks? E.g. use the model to help decompose the task into parts, then help it iterate through it. Gives you more control over the outcome and more steering anyway.
- janalsncm 6mo agoReasoning over a large codebase is only one use case for large models. For the use cases in the article (summarizing, classifying, basic text rewrites) most phones can handle them just fine.
- deleted 6mo ago[deleted]
- agnishom 6mo agoThe article is not about those use cases. There are plenty of use cases for which local models are already pretty good
- am17an 6mo agoLocal models embody the hacker spirit, constant Claude glazing is spiritually incompatible with tinkering. Don't upload your spirit to the cloud.
- AlienRobot 6mo agoBut if I run the model locally I have to pay for it, whereas with Claude I can, oh wait I just hit my 5 hour free limit with 2 messages.
- captainbland 6mo agoThey may well do but in practice if you want to embody the hacker spirit, the best thing is to hack rather than trying to get some clearly inadequate local LLM to do it.
- stingraycharles 6mo agoThat’s like saying cloud computing is spiritually incompatible with tinkering.
- xyzal 6mo agoIt is. You can't tinker with someone else's machine.
- seizethecheese 6mo agoYou can’t tinker with someone else’s machine but you can tinker through it.
- thot_experiment 6mo agoYo, MTP for Qwen is sick, thank you! Your work is invaluable.
- moffkalast 6mo agoShould be relatively quickly, 1-2 years for local models to catch up to today's SOTA. Of course then you'll be asking "uhh lemme know when Opus 6.8 level performance is available locally". People are never happy. Gemma 4 and Qwen 3.6 are legit beast models that would steamroll every API offering from 2 years ago.
- stingraycharles 6mo agoOpus is probably somewhere in the 5TB parameter range and needs terabytes of GPU memory. The economics of running SOTA locally just does not make sense, because you’re not using it 24/7 at 80%+ utilization while the cloud based providers can.
- deleted 6mo ago[deleted]