9 ms·
Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
- u1hcw9nx 8mo ago[flagged]
- andsoitis 8mo agoYes.
- ramon156 8mo agoIronically, chinese models so far have been less lobotomized compared to OAI and Anthropic's models
- u1hcw9nx 8mo agoQwen has been broadly aligned to give positive messages about China in English. https://chinamediaproject.org/2026/02/09/tokens-of-ai-bias/ https://chinamediaproject.org/2026/02/09/tokens-of-ai-bias/ An Analysis of Chinese LLM Censorship and Bias with Qwen 2 Instruct https://huggingface.co/blog/leonardlin/chinese-llm-censorship-analysis https://huggingface.co/blog/leonardlin/chinese-llm-censorshi...
- andsoitis 8mo agoDoes it matter for your work?
- joe_mamba 8mo agoImpressive, very nice, now let's see what would be the odds that the US models developed in SV are also highly positive about Californian and Democrats politics.
- u1hcw9nx 8mo agoWhataboutism https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism
- joe_mamba 8mo agoScreaming whataboutism is the only way people know to avoid answering an obvious fact. That LLMs have the biases of the governments where the came from doesn't matter if it's China, US, India, EU, etc
- xenospn 8mo agoAre there any non-Chinese open models that offer comparable performance?
- MarsIronPI 8mo agoI think you could look into Minstral. There's also GPT-OSS but I'm not sure how well it stacks up. What's your problem with Chinese LLMs?
- icase 8mo agoit’s not obvious to you why someone would want to avoid models created by our enemies?
- bigyabai 8mo agoNo, explain it to me. GPT-OSS is one of the most heavily-censored models on the internet, what's the point of buying local if it's crap?
- MarsIronPI 8mo agoNo, it's not. They're just collections of numbers that can be harnessed to produce outputs. I check the outputs and if they're good I use them. If they're not, I ignore them and there's no harm done. Obviously I don't trust them to be accurate sources of information, but I don't trust American corporate LLMs much more.
- shell0x 8mo agoAs an European, I trust China more than America. China doesn't just start bombing other countries and causes regime changes.
- xenospn 8mo agoNothing personally - Our customers send us highly sensitive financial documents to process. Using a foreign model to process their data (or even just for local testing) will most likely result in a u-turn.
- aliljet 8mo agoIs this actually true? I want to see actual evals that match this up with Sonnet 4.5.
- lostmsu 8mo agoNot exactly, but pretty close: https://artificialanalysis.ai/models/capabilities/coding?models=gpt-oss-120b%2Cgpt-5-2%2Cgpt-5-2-codex%2Cgpt-5-3-codex%2Cgemini-3-flash-reasoning%2Cgemini-3-1-pro-preview%2Cclaude-sonnet-4-6-adaptive%2Cclaude-4-5-haiku-reasoning%2Cclaude-opus-4-6-adaptive%2Cdeepseek-v3-2-reasoning%2Cminimax-m2-5%2Ckimi-k2-5%2Cglm-5%2Cqwen3-5-122b-a10b%2Cqwen3-5-397b-a17b%2Cqwen3-5-35b-a3b%2Cclaude-4-5-sonnet-thinking https://artificialanalysis.ai/models/capabilities/coding?mod... Somewhere between Haiku 4.5 and Sonnet 4.5
- CharlesW 8mo ago> Somewhere between Haiku 4.5 and Sonnet 4.5 That's like saying "somewhere between Eliza and Haiku 4.5". Haiku is not even a so-called 'reasoning model'.¹ ¹ To preempt the easily-offended, this is what the latest Opus 4.6 in today's Claude Code update says: "Claude Haiku 4.5 is not a reasoning model — it's optimized for speed and cost efficiency. It's the fastest model in the Claude family, good for quick, straightforward tasks, but it doesn't have extended thinking/reasoning capabilities."
- pityJuke 8mo agoHaiku 4.5 is a reasoning model. [0] [0]: https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c3630... > Claude Haiku 4.5, a new hybrid reasoning large language model from Anthropic in our small, fast model class. > As with each model released by Anthropic beginning with Claude Sonnet 3.7, Claude Haiku 4.5 is a hybrid reasoning model. This means that by default the model will answer a query rapidly, but users have the option to toggle on “extended thinking mode”, where the model will spend more time considering its response before it answers. Note that our previous model in the Haiku small-model class, Claude Haiku 3.5, did not have an extended thinking mode.
- erelong 8mo agoWhat kind of hardware does HN recommend or like to run these models?
- xienze 8mo agoIt's less than you'd think. I'm using the 35B-A3B model on an A5000, which is something like a slightly faster 3080 with 24GB VRAM. I'm able to fit the entire Q4 model in memory with 128K context (and I think I would probably be able to do 256K since I still have like 4GB of VRAM free). The prompt processing is something like 1K tokens/second and generates around 100 tokens/second. Plenty fast for agentic use via Opencode.
- rahimnathwani 8mo agoThere seem to be a lot of different Q4s of this model: https://www.reddit.com/r/LocalLLaMA/s/kHUnFWZXom https://www.reddit.com/r/LocalLLaMA/s/kHUnFWZXom I'm curious which one you're using.
- suprjami 8mo agoUnsloth Dynamic. Don't bother with anything else.
- rahimnathwani 8mo agoUD-Q4_K_XL?
- BoredomIsFun 8mo agoFYI UD quants of 3.5-35BA3B are broken, use bartowski or AesSedai ones.
- regularfry 8mo agoThey've uploaded the fix. If those are still broken something bad has happened.
- mark_l_watson 8mo agoThe new 35b model is great. That said, it has slight incompatibility's with Claude Code. It is very good for tool use.
- johnnyApplePRNG 8mo agoClaude code is designed for anthropic models. Try it with opencode!
- kristianpaul 8mo agoOr Pi
- copperx 8mo agoOr Oh My Pi
- mark_l_watson 8mo agoI will, right now. EDIT: opencode was a bit slow with qwen3.5:35b using Ollama. Faster/nicer to use with Liquid lfm2:latest
- johnnyApplePRNG 8mo agoTry llama.cpp - it usually excels with these MoE models imho.
- stavros 8mo agoHave you tried the 122B one?
- solarkraft 8mo agoWhat are the recommended 4 bit quants for the 35B model? I don’t see official ones: https://huggingface.co/models?other=base_model:quantized:Qwen%2FQwen3.5-35B-A3B&sort=likes https://huggingface.co/models?other=base_model:quantized:Qwe... Edit: The unsloth quants seem to have been fixed, so they are probably the go-to again: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
- solarkraft 8mo agoSmells like hyperbole. A lot of people making such claims don’t seem to have continued real world experience with these models or seem to have very weird standards for what they consider usable. Up until relatively recently, while people had already long been making these claims, it came with the asterisks of „oh, but you can’t practically use more than a few K tokens of context“.
- tempest_ 8mo agoQwen3-Coder-30B-A3B-Instruct is good I think for in line IDE integration or operating on small functions or library code but I dont think you will get too far with one shot feature implementation that people are currently doing with Claude or whatever.
- andy_ppp 8mo agoI have been adding a one shot feature to a codebase with ChatGPT 5.3 Codex in Cursor and it worked out of the box but then I realised everything it had done was super weird and it didn't work under a load of edge cases. I've tried being super clear about how to fix it but the model is lost. This was not a complex feature at all so hopefully I'm employed for a few more years yet.
- rubyn00bie 8mo agoI could be doing something wrong, but I have not had any success with one shot feature implementations for any of the current models. There are always weird quirks, undesired behaviors, bad practices, or just egregiously broken implementations. A week or so ago, I had instructed Claude to do something at compile-time and it instead burned a phenomenal amount of tokens before yeeting the most absurd, and convoluted, runtime implementation—- that didn’t even work. At work I use it (or Codex) for specific tasks, delegating specific steps of the feature implementation. The more I use the cloud based frontier models, the more virtue I find in using local, open source/weights, models because they tend to create much simpler code. They require more direct interaction from me, but the end result tends to be less buggy, easier to refactor/clean up, and more precisely what I wanted. I am personally excited to try this new model out here shortly on my 5090. If read the article correctly, it sounds like even the quantized versions have a “million”[1] token context window. And to note, I’m sure I could use the same interaction loop for Claude or GPT, but the local models are free (minus the power) to run. [1] I’m a dubious it won’t shite itself at even 50% of that. But even 250k would be amazing for a local model when I “only” have 32GB of VRAM.
- kristianpaul 8mo agoThey work great with kagi and pi
- alexpotato 8mo agoI recently wrote a guide on getting: - llama.cpp - OpenCode - Qwen3-Coder-30B-A3B-Instruct in GGUF format (Q4_K_M quantization) working on a M1 MacBook Pro (e.g. using brew). It was bit finicky to get all of the pieces together so hopefully this can be used with these newer models. https://gist.github.com/alexpotato/5b76989c24593962898294038b5b835b https://gist.github.com/alexpotato/5b76989c24593962898294038...
- copperx 8mo agoHow fast does it run on your M1?
- robby_w_g 8mo agoDoes your MBP have 32 GB of ram? I’m waiting on a local model that can run decently on 16 GB
- freeone3000 8mo agoWe can also run LM Studio and get it installed with one search and one click, exposed through an OpenAI-compatible API.
- kpw94 8mo agoOn my 32GB Ryzen desktop (recently upgraded from 16GB before the RAM prices went up another +40%), did the same setup of llama.cpp (with Vulkan extra steps) and also converged on Qwen3-Coder-30B-A3B-Instruct (also Q4_K_M quantization) On the model choice: I've tried latest gemma, ministral, and a bunch of others. But qwen was definitely the most impressive (and much faster inference thanks to MoE architecture), so can't wait to try Qwen3.5-35B-A3B if it fits. I've no clue about which quantization to pick though ... I picked Q4_K_M at random, was your choice of quantization more educated?
- NamlchakKhandro 8mo ago[flagged]
- zargon 8mo agoQuant choice depends on your vram, use case, need for speed, etc. For coding I would not go below Q4_K_M (though for Q4, unsloth XL or ik_llama IQ quants are usually better at the same size). Preferably Q5 or even Q6.
- sunkeeh 8mo agoQwen3.5-122B-A10B BF16 GGUF = 224GB. The "80Gb VRAM" mentioned here will barely fit Q4_K_S (70GB), which will NOT perform as shown on benchmarks. Quite misleading, really.
- CamperBob2 8mo agoThe larger 3.5 quants are actually pretty close to the full-blown 397B model's performance, at least looking at the numbers. Qwen 3.5 seems more tolerant of quantization than most.
- kristianpaul 8mo agohttps://unsloth.ai/docs/models/qwen3.5#qwen3.5-27b https://unsloth.ai/docs/models/qwen3.5#qwen3.5-27b “ Qwen3.5-27B For this guide we will be utilizing Dynamic 4-bit which works great on a 18GB RAM”
- kristianp 8mo ago18GB was an odd 3-channel one-off for the M3 Pros. I guess there's a bunch of them out there, but how slow would 27B be on it, due to not being an MOE model.
- gunalx 8mo agoqwen 3.5 is really decent. oOtside for some weird failures on some scaffolding with seemingly different trained tools. Strong vision and reasoning performance, and the 35-a3b model run s pretty ok on a 16gb GPU with some CPU layers.
- PunchyHamster 8mo agoI asked it to recite potato 100 times coz I wanted to benchmark speed of CPU vs GPU. It's on 150 line of planning. It recited the requested thing 4 times already and started drafting the 5th response. ...yeah I doubt it
- lumirth 8mo agowell hold on now, maybe it’s onto something. do you really know what it means to “recite” “potato” “100” “times”? each of those words could be pulled apart into a dissertation-level thesis and analysis of language, history, and communication. either that, or it has a delusional level of instruction following. doesn’t mean it can’t code like sonnet though
- PunchyHamster 8mo agoIt's still amusing to see those seemingly simple things still put it into loop it is still going > do you really know what it means to “recite” “potato” “100” “times”? asking user question is an option. Sonnet did that a bunch when I was trying to debug some network issue. It also forgot the facts checked for it and told it before...
- lumirth 8mo agoI wonder how much certain models have been trained to avoid asking too many questions. I’ve had coworkers who’ll complete an entire project before asking a single additional question to management, and it has never gone well for them. Unsurprising that the same would be true for the “managing AI” era of programming. The thing I struggle most with, honestly, is when AI (usually GPT5.3-Codex) asks me a question and I genuinely don’t know the answer. I’m just like “well, uh… follow industry best practice, please? unless best practice is dumb, I guess. do a good. please do a good.” And then I get to find out what the answer should’ve been the hard way.
- lachiflippi 8mo agoQwen3.5 pretty much requires a long system prompt, otherwise it goes into a weird planning mode where it reasons for minutes about what to do, and double and triple checks everything it does. Both Gemini's and Claude Opus 4.6's prompts work pretty well, but are so long that whatever you're using to run the model has to support prompt caching. Asking it to "Say the word "potato" 100 times, once per line, numbered.", for example, results in the following reasoning, followed by the word "potato" in 100 numbered lines, using the smallest (and therefore dumbest) quant unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS: "User is asking me to repeat the word "potato" 100 times, numbered. This is a simple request - I can comply with this request. Let me create a response that includes the word "potato" 100 times, numbered from 1 to 100. I'll need to be careful about formatting - the user wants it numbered and once per line. I should use minimal formatting as per my instructions."
- mstaoru 8mo agoI periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes. Today I ran the question against local qwen3.5:35b-a3b - it puffed for 45 (!) minutes, produced a very generic answer with errors, and made my laptop sound like it's going to take off any moment. Wonder what am I doing wrong?.. How am I supposed to use this for any agentic coding on a large enough codebase? It will take days (and a 3M Peltor X5A) to produce anything useful.
- zozbot234 8mo agoRunning local AI models on a laptop is a weird choice. The Mini and especially the Studio form factor will have better cooling, lower prices for comparable specs and a much higher ceiling in performance and memory capacity.
- stavros 8mo agoI can never see the point, though. Performance isn't anywhere near Opus, and even that gets confused following instructions or making tool calls in demanding scenarios. Open weights models are just light years behind. I really, really want open weights models to be great, but I've been disappointed with them. I don't even run them locally, I try them from providers, but they're never as good as even the current Sonnet.
- nu11ptr 8mo agoThinking about getting a new MBP M5 Max 128GB (assuming they are released next week). I know "future proofing" at this stage is near impossible, but for writing Rust code locally (likely using Qwen 3.5 for now on MLX), the AIs have convinced me this is probably my best choice for immediate with some level of longevity, while retaining portability (not strictly needed, but nice to have). Alternatively was considering RTX options or a mac studio, but was leaning towards apple for the unified memory. What does HN think?
- cmenge 8mo agoI've been mulling the same, but decided against (for now) Using Claude Code Max 20 so ROI would be maybe 2+ years. CC gives me unlimited coding in 4-6 windows in parallel. Unsure if any model would beat (or even match) that, both in terms in quality and speed. I wouldn't gamble on that now. With a subscription, I can change any time. With the machine, you risk that this great insane model comes out but you need 138GB and then you'll pay for both.
- nu11ptr 8mo agoWe are on the same wavelength. I'm thinking maybe a pass for now.
- pamcake 8mo ago> What does HN think? Thermals. Your workloads will be throttled hard once it inevitably runs hot. See comments elsewhere in thread about why LLMs on laptops like MBP is underwhelming. The same chips in even a studio form factor would perform much better.
- shell0x 8mo agoI have a Mac Studio with 128GB and a M4 Max and I'd recommend it. The power usage is also pretty good, but you may not care if you live somewhere where energy is cheap.
- nu11ptr 8mo agoHave you used this for Rust coding by chance? I'm curious how it compares to Opus 4.6. I realize it isn't going to think to the same level, but curious how code quality is for a more straight forward task.
- jbellis 8mo agothis is bullshit with a kernel of truth. none of the qwen 3.5 models are anywhere near sonnet 4.5 class, not even the largest 397b. BUT 27b is the smartest local-sized model in the world by a wide wide margin. (35b is shit. fast shit, but shit.) benchmarks are complete, publishing on Monday.
- dimgl 8mo agoYou mean 35B A3B? If this is shit, this is some of the best shit out I've seen yet. Never in a million years did I think I'd have an LLM running locally, actually writing code on my behalf. Accurately too.
- throwdbaaway 8mo agoI would say 27B matches with Sonnet 4.0, while 397B A17B matches with Opus 4.1. They are indeed nowhere near Sonnet 4.5, but getting 262144 context length at good speed with modest hardware is huge for local inference. Will check your updated ranking on Monday.
- oscord 8mo agoSWE chart is missing Claude on front page, interesting way to present your data. Mix and match at will. Grown up people showing public school level sneakiness. That fact alone disqualifies your LL. Business/marketing leaders usually are brighter than average developers... so there.
- car 8mo agoCan it do FizzBuzz in Brainfuck? Thus far all local models have tripped over their feet or looped out.
- CamperBob2 8mo ago122B-A10B-UD-Q4-K-XL generated https://pastebin.com/j3ddfNtS https://pastebin.com/j3ddfNtS -- but I can't get it to do anything in a couple of online interpreters. Guessing it wasn't trained on a lot of Brainfuck code. Edit: it looks like the flagship models work by writing a C or Python program to do the bookkeeping. I don't have Qwen set up to use tools, and even Opus 4.6 shits the bed when told to do it without tools [1], so not too surprising that it didn't work. 1: https://claude.ai/share/1f5289ae-decd-4dfa-98fd-0d34346008c6 https://claude.ai/share/1f5289ae-decd-4dfa-98fd-0d34346008c6 -- I interrupted it and told it not to use a C/Python program or any other tools to generate the Brainfuck code, and it gave me an error message after about 10 minutes that wasn't logged to the chat.
- Aurornis 8mo agoIf you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experience. I have observed that they're configured to be very tenacious. If you can carefully constrain the goal with some tests they need to pass and frame it in a way to keep them on track, they will just keep trying things over and over. They'll "solve" a lot of these problems in the way that a broken clock is right twice a day, but there's a lot of fumbling to get there. That said, they are impressive for open source models. It's amazing what you can do with self-hosted now. Just don't believe the hype that these are Sonnet 4.5 level models because you're going to be very disappointed once you get into anything complex.
- amelius 8mo agoAre you saying that the benchmarks are flawed? And could quantization maybe partially explain the worse than expected results?
- Aurornis 8mo agoThe models outperform on the benchmarks relative to general tasks. The benchmarks are public. They're guaranteed to be in the training sets by now. So the benchmarks are no longer an indicator of general performance because the specific tasks have been seen before. > And could quantization maybe explain the worse than expected results? You can use the models through various providers on OpenRouter cheaply without quantization.
- deleted 8mo ago[deleted]
- girvo 8mo agoFlawed? Possibly, but I think it's more that any kind of benchmark then becomes a target, and is inherently going to be a "lossy" signal as to the models actual ability in practice. Quantisation doesn't help, but even running full fat versions of these models through various cloud providers, they still don't match Sonnet in actual agentic coding uses: at least in my experience.
- xmddmx 8mo agoOllama users: there are notable bugs with ollama and Qwen3.5 so don't let your first impression be the last. Theory is that some of the model parameters aren't set properly and this encourages endless looping behavior when run under ollama: https://github.com/ollama/ollama/issues?q=is%3Aissue%20state%3Aopen%20qwen%203.5 https://github.com/ollama/ollama/issues?q=is%3Aissue%20state... (a bunch of them)
- aplomb1026 8mo ago[flagged]
- jjcm 8mo agoGetting better, but definitely not there yet, nor near Sonnet 4.5 performance. What these open models are great for are for narrow, constrained domains, with good input/output examples. I typically use them for things like prompt expansion, sentiment analysis, reformatting or re-arranging flow of code. What I found they have trouble with is going from ambiguous description -> solved problem. Qwen 3.5 is certainly the best of the OSS models I've found (beating out GPT 120b OSS which was the previous king), and it's just starting to demonstrate true intelligence in unbound situations, but it isn't quite there yet. I have a RTX 6000 pro, so Qwen 3.5 is free for me to run, but I tend to default to Composer 1.5 if I want to be cheap. The trend however is super encouraging. I bought my vid card with the full expectation that we'll have a locally running GPT 5.2 equiv by EoY, and I think we're on track.
- syntaxing 8mo agoA big part that a lot of local users forget is inference is hard. Maybe you have the wrong temperature. Maybe you have the wrong min P. Maybe you have the wrong template. Maybe the implementation in llama cpp has a bug. Maybe Q4 or even Q8 just won’t compare to BF16. Reality is, there’s so many knobs to LLM inferencing and any can make the experience worse. It’s not always the model’s fault.
- hsaliak 8mo agoNo it does not. None of these models have the “depth” that the frontier models have across a variety of conversations, tasks and situations. Working with them is like playing snakes and ladders, you never know when it’s going to do something crazy and set you back.
- karmasimida 8mo agoRaw scale of parameters is POWER, you can't get performance out of a small model from a much larger one.
- jackcosgrove 8mo agoI am a total neophyte when it comes to LLMs, and only recently started poking around into the internals of them. The first thing that struck me was that float32 dimensions seemed very generous. I then discovered what quantization is by reading a blog post about binary quantization. That seemed too good to be true. I asked Claude to design an analysis assessing the fidelity of 1, 2, 4, and 8 bit quantization. Claude did a good job, downloading 10,000 embeddings from a public source and computing a similarity score and correlation coefficient for each level of quantization against the float32 SoT. 1 and 2 bit quantizations were about 90% similar and 8 bit quantization was lossless given the precision Claude used to display the results. 4 bit was interesting as it was 99% similar (almost lossless) yet half the size of 8 bit. It seemed like the sweet spot. This analysis took me all of an hour so I thought, "That's cool but is it real?" It's gratifying to see that 4 bit quantization is actually being used by professionals in this field.
- tymscar 8mo agoThats cool. I do wonder where that extra acuity you get from 1% more shows up in practice. I hate how I have basically no way to intuitively tell that because of how much of a black box the system is
- doctorpangloss 8mo agoWell why would Claude know any of this? Obviously it's the wrong criteria. If you have your own dataset to benchmark, created your own calibration for quantization with it. Scientifically, you wouldn't really believe in the whole process of gradient descent if you didn't think tiny differences in these values matter. So...
- tymscar 8mo agoI think you might be answering to a different person or misunderstanding what I said but you are right that just as I don’t have an intuition for where the acuity shows up in the corpus, I don’t think Claude does either
- 8mo ago
- piyh 8mo agoUnsloth is working magic with the qwen quants
- shell0x 8mo agoCan't wait to try that out locally. Keen to reduce my dependence on American products and services.
- Paddyz 8mo ago[dead]
- mmis1000 8mo agofrom my personal experience, qwen 30b a3b understand command quiet well as long as the input is not big enough that ruin the attention (I feel the boundary is somewhere between 8000 or 12000?). But that isn't really bug of model itself though. A smaller model just have shorter memory, it's simply physical restriction. I made a mixed extraction, cleaning, translation, formatting task on job that have average 6000 token input. And so far, only 30b a3b is smart enough not miss job detail (most of time) I later refactor the task to multi pass using smaller model though. Make job simpler is still a better strategy to get clean output if you can change the pipeline.
- pstuart 8mo agoOne highly annoying facet of the hardware is that AND's support for the NPU under linux is currently non-existent. which abandons 50 of the 126 TOPS stated of AI capability. They seem to think that Windows support is good enough. Grrrrrr.
- renewiltord 8mo agoIn practice I have not seen this. Sonnet is incredible performance. No open model is close. Hosted open models are so much worse that I end up spending more because of inferior intelligence.
- lubitelpospat 8mo agoAll right guys, this is your time - what consumer device do you use for local LLM inference? GPU poor answers only
- carlio 8mo agoAn AMD AI max+ 395 - I use the one from frame.work (https://frame.work/de/en/desktop https://frame.work/de/en/desktop) with 128GB unified RAM and it can run a 120b model (gpt-oss:120b) just fine. See Wendel's review here - https://www.youtube.com/watch?v=L-xgMQ-7lW0 https://www.youtube.com/watch?v=L-xgMQ-7lW0 There are other mini-pc manufacturers, the mainboard is the important part.
- lubitelpospat 8mo agoWow, that's quite beefy.
- OkWing99 8mo agoCan someone who has done this, simplify and say what specs we need on a `local computer` to run and test this, with a reasonable speed? Excluding MBP M5 128GB.
- chvid 8mo agoIt is slow but usable via opencode on a mbp m3 max 48 gb. So I guess hosted is still the better option for most people. The local models are considerably better relative to the hosted ones compared to 6 months ago. Bench maxing or not - stuff is happening in this area for sure.
- regularfry 8mo agoI've got the unsloth q4_K_XL 35b running in llama.cpp on an i9/64G/4090 machine doing double-digit tokens per second with a 90k+ token context window available. The model's completely in VRAM.
- plastic3169 8mo agoAnyone have recommendations on EU services where one could run open models before buying expensive hardware?
- zos_kia 8mo agoKoyeb (recently acquired by Mistral if I'm not mistaken) have GPUs you can rent by the minute and they also have one-click deploy of some open models.
- pram 8mo agoI decided to try Qwen3.5 122B in LM Studio with Opencode and I am impressed. It's not super slow (M4 Max/128GB) and it's pretty close to how Claude Code feels. Getting pretty good code analysis, definitely feels Sonnet-esque. I'm hyped completely local alternatives are getting so good.
- jedisct1 8mo agoQwen3-Coder-Next also remains amazing as a local model. If you want to use small models for coding, I'd highly recommend Swival https://swival.dev https://swival.dev which was explicitly optimized for these.
- jgalt212 8mo agoOur shop cannot use cloud models for sensitive data and code. For shops like ours, we continue to be impressed and appreciative of the progress in open-source / self-hosted models.
- ffsickempire 8mo ago[dead]