12 ms·
Qwen3-Coder-Next
- danielhanchen 8mo agoFor those interested, made some Dynamic Unsloth GGUFs for local deployment at https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF and made a guide on using Claude Code / Codex locally: https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next
- ranger_danger 8mo agoWhat is the difference between the UD and non-UD files?
- danielhanchen 8mo agoUD stands for "Unsloth-Dynamic" which upcasts important layers to higher bits. Non UD is just standard llama.cpp quants. Both still use our calibration dataset.
- CamperBob2 8mo agoPlease consider authoring a single, straightforward introductory-level page somewhere that explains what all the filename components mean, and who should use which variants. The green/yellow/red indicators for different levels of hardware support are really helpful, but far from enough IMO.
- danielhanchen 8mo agoOh good idea! In general UD-Q4_K_XL (Unsloth Dynamic 4bits Extra Large) is what I generally recommend for most hardware - MXFP4_MOE is also ok
- Keats 8mo agoIs there some indication on how the different bit quantization affect performance? IE I have a 5090 + 96GB so I want to get the best possible model but I don't care about getting 2% better perf if I only get 5 tok/s.
- mirekrusin 8mo agoIt takes download time + 1 minute to test speed yourself, you can try different quants, it's hard to write down a table because it depends on your system ie. ram clock etc. if you go out of gpu. I guess it would make sense to have something like max context size/quants that fit fully on common configs with gpus, dual gpus, unified ram on mac etc.
- Keats 8mo agoTesting speed is easy yes, I'm mostly wondering about the quality difference between Q6 vs Q8_K_XL for example.
- danielhanchen 8mo agoI haven't done benchmarking yet (plan to do them), but it should be similar to our post on DeepSeek-V3.1 Dynamic GGUFs: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- segmondy 8mo agoThe green/yellow/red indicators are based on what you set for your hardware on huggingface.
- ranger_danger 8mo agoWhat is your definition of "important" in this context?
- danielhanchen 8mo agoOh we wrote about it here: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- binsquare 8mo agoHow did you do it so fast? Great work as always btw!
- danielhanchen 8mo agoThanks! :) We're early access partners with them!
- genpfault 8mo agoNice! Getting ~39 tok/s @ ~60% GPU util. (~170W out of 303W per nvtop). System info: $ ./llama-server --version ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat version: 7897 (3dd95914d) built with GNU 11.4.0 for Linux x86_64 llama.cpp command-line: $ ./llama-server --host 0.0.0.0 --port 2000 --no-warmup \ -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \ --jinja --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 --fit on \ --ctx-size 32768
- halcyonblue 8mo agoWhat am I missing here? I thought this model needs 46GB of unified memory for 4-bit quant. Radeon RX 7900 XTX has 24GB of memory right? Hoping to get some insight, thanks in advance!
- coder543 8mo agoMoEs can be efficiently split between dense weights (attention/KV/etc) and sparse (MoE) weights. By running the dense weights on the GPU and offloading the sparse weights to slower CPU RAM, you can still get surprisingly decent performance out of a lot of MoEs. Not as good as running the entire thing on the GPU, of course.
- danielhanchen 8mo agoSuper cool! Also with `--fit on` you don't need `--ctx-size 32768` technically anymore - llama-server will auto determine the max context size!
- genpfault 8mo agoNifty, thanks for the heads-up!
- lnenad 8mo agoThanks to you I decided to give it a go as well (didn't think I'd be able to run it on 7900xtx) and I must say it's awesome for a local model. More than capable for more straightforward stuff. It uses full VRAM and about 60GBs of RAM, but runs at about 10tok/s and is *very* usable.
- bityard 8mo agoHi Daniel, I've been using some of your models on my Framework Desktop at home. Thanks for all that you do. Asking from a place of pure ignorance here, because I don't see the answer on HF or in your docs: Why would I (or anyone) want to run this instead of Qwen3's own GGUFs?
- danielhanchen 8mo agoThanks! Oh Qwen3's own GGUFs also works, but ours are dynamically quantized and calibrated with a reasonably large diverse dataset, whilst Qwen's ones are not - see https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- bityard 8mo agoI've read that page before and although it all certainly sounds very impressive, I'm not an AI researcher. What's the actual goal of dynamic quantization? Does it make the model more accurate? Faster? Smaller?
- CamperBob2 8mo agoGood results with your Q8_0 version on 96GB RTX 6000 Blackwell. It one-shotted the Flappy Bird game and also wrote a good Wordle clone in four shots, all at over 60 tps. Thanks! Is your Q8_0 file the same as the one hosted directly on the Qwen GGUF page?
- danielhanchen 8mo agoNice! Yes Q8_0 is similar - the others are different since they use a calibration dataset.
- MrDrMcCoy 8mo agoStill hoping IQuest-Coder gets the same treatment :)
- bytesandbits 8mo agohow are you so fast man
- simonw 8mo agoThis GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/Qwen3-Coder-Next-Q4_K_M https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next
- vessenes 8mo agoI'm thinking the next step would be to include this as a 'junior dev' and let Opus farm simple stuff out to it. It could be local, but also if it's on cerebras, it could be realllly fast.
- ttoinou 8mo agoCerebras already has GLM 4.7 in the code plans
- vessenes 8mo agoYep. But this is like 10x faster; 3B active parameters.
- ttoinou 8mo agoCerebras is already 200-800 tps, do you need even faster ?
- overfeed 8mo agoYes! I don't try to read agent tokens as they are generated, so if code generation decreases from 1 minute to 6 seconds, I'll be delighted. I'll even accept 10s -> 1s speedups. Considering how often I've seen agents spin wheels with different approaches, faster is always better, until models can 1-shot solutions without the repeated "No, wait..." / "Actually..." thinking loops
- vessenes 8mo ago3B active parameters, and slightly worse than GLM 4.7. On benchmarks. That's pretty amazing! With better orchestration tools being deployed, I've been wondering if faster, dumber coding agents paired with wise orchestrators might be overall faster than using the say opus 4.5 on the bottom for coding. At least we might want to deploy to these guys for simple tasks.
- doctorpangloss 8mo agoTime will tell. All this stuff will get more adoption when Anthropic, Google and OpenAI raise prices.
- Alifatisk 8mo agoThey can only raise prices as long as people buy their subscriptions / pay for their api. The Chinese labs are closing in on the SOTA models (I would say they are already there) and offer insane cheap prices for their subscriptions. Vote with your wallet.
- markab21 8mo agoIt's getting a lot easier to do this using sub-agents with tools in Claude. I have a fleet of Mastra agents (TypeScript). I use those agents inside my project as CLI tools to do repetitive tasks that gobble tokens such as scanning code, web search, library search, and even SourceGraph traversal. Overall, it's allowed me to maintain more consistent workflows as I'm less dependent on Opus. Now that Mastra has introduced the concept of Workspaces, which allow for more agentic development, this approach has become even more powerful.
- endymion-light 8mo agoLooks great - i'll try to check it out on my gaming PC. On a misc note: What's being used to create the screen recordings? It looks so smooth!
- kevinsync 8mo agoIt might be Screen Studio [0] -- I was gonna write "99% sure" but now I'm not sure at all!! [0] https://screen.studio https://screen.studio
- zamadatix 8mo agoCan anyone help me understand the "Number of Agent Turns" vs "SWE-Bench Pro (%)" figure? I.e. what does the spread of Qwen3-Coder-Next from ~50 to ~280 agent turns represent for a fixed score of 44.3%: that sometimes it takes that spread of agent turns to achieve said fixed score for the given model?
- edude03 8mo agoEssentially the more turns you have the more the agent is likely to fail since the error compounds per turn. Agentic model are tuned for “long horizon tasks” ie being able to go many many turns on the same problem without failing.
- zamadatix 8mo agoMuch appreciated, but I mean more around "what do the error bars in the figure represent" than what the turn scaling itself is.
- esafak 8mo agoFor the tasks in SWE-Bench Pro they obtained a distribution of agent turns, summarized as the box plot. The box likely describes the inter-quartile range while the whiskers describe the some other range. You'd have to read their report to be sure. https://en.wikipedia.org/wiki/Box_plot https://en.wikipedia.org/wiki/Box_plot
- jsnell 8mo agoThat's a box plot, so those are not error bars but a visualization of the distribution of a metric (min, max, median, 25th percentile, 75th percentile). The benchmark consists of a bunch of tasks. The chart shows the distribution of the number of turns taken over all those tasks.
- yorwba 8mo agoSWE-Bench Pro consists of 1865 tasks. https://arxiv.org/abs/2509.16941 https://arxiv.org/abs/2509.16941 Qwen3-Coder-Next solved 44.3% (826 or 827) of these tasks. To solve a single task, it took between ≈50 and ≈280 agent turns, ≈150 on average. In other words, a single pass through the dataset took ≈280000 agent turns. Kimi-K2.5 solved ≈84 fewer tasks, but also only took about a third as many agent turns.
- throwaw12 8mo agoWe are getting there, as a next step please release something to outperform Opus 4.5 and GPT 5.2 in coding tasks
- gordonhart 8mo agoBy the time that happens, Opus 5 and GPT-5.5 will be out. At that point will a GPT-5.2 tier open-weights model feel "good enough"? Based on my experience with frontier models, once you get a taste of the latest and greatest it's very hard to go back to a less capable model, even if that less capable model would have been SOTA 9 months ago.
- tosh 8mo agoIt feels like the gap between open weight and closed weight models is closing though.
- theshrike79 8mo agoMode like open local models are becoming "good enough". I got stuff done with Sonnet 3.7 just fine, it did need a bunch of babysitting, but still it was a net positive to productivity. Now local models are at that level, closing up on the current SOTA. When "anyone" can run an Opus 4.5 level model at home, we're going to be getting diminishing returns from closed online-only models.
- cyanydeez 8mo agoSee, the market is investing like _that will never happen_.
- theshrike79 8mo agoI'm just riding the VC powered wave of way-too-cheap online AI services and building tools and scaffolding to prepare for the eventual switch to local models =)
- 8mo ago
- skhameneh 8mo agoIt’s hard to elaborate just how wild this model might be if it performs as claimed. The claims are this can perform close to Sonnet 4.5 for assisted coding (SWE bench) while using only 3B active parameters. This is obscenely small for the claimed performance.
- cirrusfan 8mo agoIf it sounds too good to be true…
- theshrike79 8mo agoShould be possible with optimised models, just drop all "generic" stuff and focus on coding performance. There's no reason for a coding model to contain all of ao3 and wikipedia =)
- noveltyaccount 8mo agoI think I like coding models that know a lot about the world. They can disambiguate my requirements and build better products.
- regularfry 8mo agoI generally prefer a coding model that can google for the docs, but separate models for /plan and /build is also a thing.
- noveltyaccount 8mo ago> separate models for /plan and /build I had not considered that, seems like a great solution for local models that may be more resource-constrained.
- regularfry 8mo agoYou can configure aider that way. You get three, in fact: an architect model, a code editor model, and a quick model for things like commit messages. Although I'm not sure if it's got doc searching capabilities.
- alexellisuk 8mo agoIs this going to need 1x or 2x of those RTX PRO 6000s to allow for a decent KV for an active context length of 64-100k? It's one thing running the model without any context, but coding agents build it up close to the max and that slows down generation massively in my experience.
- Soerensen 8mo agoThe agent orchestration point from vessenes is interesting - using faster, smaller models for routine tasks while reserving frontier models for complex reasoning. In practice, I've found the economics work like this: 1. Code generation (boilerplate, tests, migrations) - smaller models are fine, and latency matters more than peak capability 2. Architecture decisions, debugging subtle issues - worth the cost of frontier models 3. Refactoring existing code - the model needs to "understand" before changing, so context and reasoning matter more The 3B active parameters claim is the key unlock here. If this actually runs well on consumer hardware with reasonable context windows, it becomes the obvious choice for category 1 tasks. The question is whether the SWE-Bench numbers hold up for real-world "agent turn" scenarios where you're doing hundreds of small operations.
- cirrusfan 8mo agoI find it really surprising that you’re fine with low end models for coding - I went through a lot of open-weights models, local and "local", and I consistently found the results underwhelming. The glm-4.7 was the smallest model I found to be somewhat reliable, but that’s a sizable 350b and stretches the definition of local-as-in-at-home.
- NitpickLawyer 8mo agoYou're replying to a bot, fyi :)
- CamperBob2 8mo agoIf it weren't for the single em-dash (really an en-dash, used as if it were an em-dash), how am I supposed to know that? And at the end of the day, does it matter?
- axus 8mo agoSome people reply for their own happiness, some reply to communicate with another person. The AI won't remember or care about the reply.
- syntaxing 8mo agoIs Qwen next architecture ironed out in llama cpp?
- raphaelmolly8 8mo ago[dead]
- orliesaurus 8mo agohow can anyone keep up with all these releases... what's next? Sonnet 5?
- Squarex 8mo agoWell there are rumors sonnet 5 is coming today, so...
- gessha 8mo agoTune it out, come back in 6 months, the world is not going to end. In 6 months, you’re going to change your API endpoint and/or your subscription and then spend a day or two adjusting. Off to the races you go.
- bigyabai 8mo agoRelatively, it's not that hard. There's like 4-5 "real" AI labs, who altogether manage to announce maybe 3 products max, per-month. Compared to RISC core designs or IC optimization, the pace of AI innovation is slow and easy to follow.
- Havoc 8mo agoPretty much every lab you can think of has something scheduled for february. Gonna be a wild one
- cmrdporcupine 8mo agoThis is going to be a crazy month because the Chinese labs are all trying to get their releases out prior to their holidays (Lunar New Year / Spring Festival). So we've seen a series of big ones already -- GLM 4.7 Flash, Kimi 2.5, StepFun 3.5, and now this. Still to come is likely a new DeepSeek model, which could be exciting. And then I expect the Big3, OpenAI/Google/Anthropic will try to clog the airspace at the same time, to get in front of the potential competition.
- cedws 8mo agoI kind of lost interest in local models. Then Anthropic started saying I’m not allowed to use my Claude Code subscription with my preferred tools and it reminded me why we need to support open tools and models. I’ve cancelled my CC subscription, I’m not paying to support anticompetitive behaviour.
- _ink_ 8mo agoWhat setup comes close to Claude Code? I am willing to rent cloude GPUs.
- wahnfrieden 8mo agoOpenAI committed to allowing it btw. I don't know why Anthropic gets so much love here
- rustyhancock 8mo agoCause they make the best coding model. It's that simple. Everyone else is trying to compete in other ways and Anthropic are pushing for dominate the market. They'll eventually lose their performance edge and suddenly they will back to being cute and fluffy I've cancelled a clause sub, but still have one.
- bheadmaster 8mo agoAgreed. I've tried all of the models available right now, and Claude Opus is by far the most capable. I had an assertion failure triggered in a fairly complex open-source C library I was using, and Claude Opus not only found the cause, but wrote a self-contained reproduction code I could add to a GitHub issue. And it also added tests for that issue, and fixed the underlying issue. I am sincerely impressed by the capabilities of Claude Opus. Too bad its usage is so expensive.
- jmathai 8mo agoProbably because the alternatives are OpenAI, Google, Meta. Not throwing shade at those companies but it's not hard to win the hearts of developers when that's your competition.
- ossicones 8mo agoWhat browser use agent are they using here?
- novaray 8mo agoYes, the general purpose version is already supported and should have the same identical architecture
- valcron1000 8mo agoStill nothing to compete with GPT-OSS-20B for local image with 16 VRAM.
- storus 8mo agoDoes Qwen3 allow adjusting context during an LLM call or does the housekeeping need to be done before/after each call but not when a single LLM call with multiple tool calls is in progress?
- segmondy 8mo agoNot applicable... the models just process whatever context you provide to them, context management happens outside of the model and depends on your inference tool/coding agent.
- cyanydeez 8mo agoIt's interesting how people can be so into LLMs but dont, at the end of the day, understand they're just passing "well formatted" text to a text processor and everything else is build around encoding/decoding it into familiar or novel interfaces & the rest. The instability of the tooling outside of the LLM is what keeps me from building anything on the cloud, because you're attaching your knowledge and work flow to a tool that can both change dramatically based on context, cache, and model changes and can arbitrarily raise prices as "adaptable whales" push the cost up. Its akin to learning everything about beanie babies in the early 1990's and right when you think you understand the value proposition, suddenly they're all worthless.
- storus 8mo agoThat's why you can use latest open coding models locally that reportedly reached the performance of Sonet-4.5 so almost SOTA. And then you can think of tricks like I mentioned above to directly manipulate GPU RAM for context cleanup when needed which is not possible with cloud models unless their provider enables that.
- Alifatisk 8mo agoAs always, the Qwen team is pushing out fantastic content Hope they update the model page soon https://chat.qwen.ai/settings/model https://chat.qwen.ai/settings/model
- smallerfish 8mo ago> "content" Sorry, but we're talking about models as content now? There's almost always a better word than "content" if you're describing something that's in tech or online.
- Havoc 8mo agoNot everyone on hn is a native english speaker...
- Alifatisk 8mo agoI wasn’t only referring to their new model, I meant their blogpost and the research behind their progress, its always a joyride to read. I didn’t know it was this serious with the vocabulary, I’ll be more cautious in the future.
- getcrunk 8mo agoThat’s a perfectly fine usage of content (primary substance offered by a “website”)
- ionwake 8mo agowill this run on an apple m4 air with 32gb ram? Im currently using qwen 2.5 16b , and it works really well
- Robdel12 8mo agoI really really want local or self hosted models to work. But my experience is they’re not really even close to the closed paid models. Does anyone any experience with these and is this release actually workable in practice?
- littlestymaar 8mo ago> But my experience is they’re not really even close to the closed paid models. They are usually as good as the flagship model for 12-18 months ago. Which may sound like a massive difference, because somehow it is, but it's also fairly reasonable, you don't need to live to the bleeding edge.
- cmrdporcupine 8mo agoAnd it's worth pointing out that Claude Code now dispatches "subagents" from Opus->Sonnet and Opus->Haiku ... all the time, depending on the problem. Running this thing locally on my Spark with 4-bit quant I'm getting 30-35 tokens/sec in opencode but it doesn't feel any "stupider" than Haiku, that's for sure. Haiku can be dumb as a post. This thing is smarter than that. It feels somewhere around Sonnet 4 level, and I am finding it genuinely useful at 4-bit even. Though I have paid subscriptions elsewhere, so I doubt I'll actually use it much. I could see configuration OpenCode somehow to use paid Kimi 2.5 or Gemini for the planning/analysis & compaction, and this for the task execution. It seems entirely competent.
- fudged71 8mo agoI'm thrilled. Picked up a used M4 Pro 64GB this morning. Excited to test this out
- gitpusher 8mo agoPretty cool that they are advertising OpenClaw compatibility. I've tried a few locally-hosted models with OpenClaw and did not get good results – (that tool is a context-monster... the models would get completely overwhelmed them with erroneous / old instructions.) Granted these 80B models are probably optimized for H100/H200 which I do not have. Here's to hoping that OpenClaw compat. survives quantization
- simonw 8mo agoI got this running locally using llama.cpp from Homebrew and the Unsloth quantized model like this: brew upgrade llama.cpp # or brew install if you don't have it yet Then: llama-cli \ -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \ --fit on \ --seed 3407 \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --top-k 40 \ --jinja That opened a CLI interface. For a web UI on port 8080 along with an OpenAI chat completions compatible endpoint do this: llama-server \ -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \ --fit on \ --seed 3407 \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --top-k 40 \ --jinja It's using about 28GB of RAM.
- technotony 8mo agowhat are your impressions?
- simonw 8mo agoI got Codex CLI running against it and was sadly very unimpressed - it got stuck in a loop running "ls" for some reason when I asked it to create a new file.
- danielhanchen 8mo agoYes sadly that sometimes happens - the issue is Codex CLI / Claude Code were designed for GPT / Claude models specifically, so it'll be hard for OSS models directly to utilize the full spec / tools etc, and might get loops sometimes - I would maybe try the MXFP4_MOE quant to see if it helps, and maybe try Qwen CLI (was planning to make a guide for it as well) I guess until we see the day OSS models truly utilize Codex / CC very well, then local models will really take off
- mhitza 8mo agoI would recommend you fiddle with the repeat penalty flags. I use local models often, and almost all I've tried needed that to prevent loops. I'd also recommend dropping temperature down to 0. Any high temperature value feels like instructing the model "copy this homework from me but don't make it obvious".
- StevenNunez 8mo agoGoing to try this over Kimi k2.5 locally. It was nice but just a bit too slow and a resource hog.
- tommyjepsen 8mo agoI got the Qwen3 Coder 30B running locally on mac Mac M4 Max 36GB. It was slow, but it worked and did do some decent stuff: https://www.youtube.com/watch?v=7mAPaRbsjTU https://www.youtube.com/watch?v=7mAPaRbsjTU Video is speed up. I ran it through LM Studio and then OpenCode. Wrote a bit about how I set it all up here: https://www.tommyjepsen.com/blog/run-llm-locally-for-coding https://www.tommyjepsen.com/blog/run-llm-locally-for-coding
- moron4hire 8mo agoMy IT department is convinced these "ChInEsE cCcP mOdElS" are going to exfiltrate our entire corporate network of its essential fluids and vita.. erh, I mean data. I've tried explaining to them that it's physically impossible for model weights to make network requests on their own. Also, what happened to their MitM-style, extremely intrusive network monitoring that they insisted we absolutely needed?
- zokier 8mo agoFor someone who is very out of the loop with these AI models, can someone explain what I can actually run on my 3080ti (12G)? Is this something like that or is this still too big; is there anything remotely useful runnable with my GPU? I have 64G RAM if that helps (?).
- AlbinoDrought 8mo agoThis model does not fit in 12G of VRAM - even the smallest quant is unlikely to fit. However, portions can be offloaded to regular RAM / CPU with a performance hit. I would recommend trying llama.cpp's llama-server with models of increasing size until you hit the best quality / speed tradeoff with your hardware that you're willing to accept. The Unsloth guides are a great place to start: https://unsloth.ai/docs/models/qwen3-coder-next#llama.cpp-tutorial-gguf https://unsloth.ai/docs/models/qwen3-coder-next#llama.cpp-tu...
- zokier 8mo agoThanks for the pointers! one more thing, that guide says: > You can choose UD-Q4_K_XL or other quantized versions. I see eight different 4-bit quants (I assume that is the size I want?).. how to pick which one to use? IQ4_XS Q4_K_S Q4_1 IQ4_NL MXFP4_MOE Q4_0 Q4_K_M Q4_K_XL
- MrDrMcCoy 8mo agoThe I-prefix stands for Imatrix smoothing in the quantization. It trades a little more accuracy for speed than other quant styles. The _0 and _1 quants are older, simpler quants that are very accurate but kinda slow. The K quants, in my limited understanding, primarily quantize at the specified bit depth, but will bump certain important areas higher, and less used parts lower. It generally performs better while providing similar accuracy to the _1 quants. MXFP4 is specific to Nvidia, so I can't use it on my AMD hardware. It's supposed to be very efficient. The UD part includes more of Unsloth's speed optimizations. Also, depending on how much regular system RAM you have, you can offload mixture-of-expert models like this, keeping only the most important layers on your GPU. This may let you use larger, more accurate quants. That is functionality that is supported by llama.cpp and other frameworks and is worth looking into how to do.
- Tepix 8mo agoUsing lmstudio-community/Qwen3-Coder-Next-GGUF:Q8_0 I'm getting up to 32 tokens/s on Strix Halo, with room for 128k of context (out of 256k that the model can manage). From very limited testing, it seems to be slightly worse than MiniMax M2.1 Q6 (a model about twice its size). I'm impressed.
- dimgl 8mo agoHow's the Strix Halo? I'd really like to get a local inference machine so that I don't have to use quantized versions of local models.
- Tepix 8mo agoPrompt preprocessing is slow, the rest is pretty great.
- evilduck 8mo agoWorks great for these type of MOE models. The ability to have large amounts of VRAM let you run different models in parallel easily, or to have actually useful context sizes. Dense models can get sluggish though. AMD's ROCm support has been a little rough for Stable Diffusion stuff (memory issues leading to application stability problems) but it's worked well with LLMs, as does Vulkan. I wish AMD would get around to adding NPU support in Linux for it though, it has more potential that could be unlocked.
- cmrdporcupine 8mo agoI'm getting similar numbers on NVIDIA Spark around 25-30 tokens/sec output, 251 token/sec prompt processing... but I'm running with the Q4_K_XL quant. I'll try the Q8 next, but that would leave less room for context. I tried FP8 in vLLM and it used 110GB and then my machine started to swap when I hit it with a query. Only room for 16k context. I suspect there will be some optimizations over the next few weeks that will pick up the performance on these type of machines. I have it writing some Rust code and it's definitely slower than using a hosted model but it's actually seeming pretty competent. These are the first results I've had on a locally hosted model that I could see myself actually using, though only once the speed picks up a bit. I suspect the API providers will offer this model for nice and cheap, too.
- jtbaker 8mo agoany way to run these via ollama yet?
- dzonga 8mo agothe qwen website doesn't work for me in safari :(. had to read the announcement in chrome
- kylehotchkiss 8mo agoIs there any online resource tracking local model capability on say... a $2000 64gb memory Mac Mini? I'm getting increasingly excited about the local model space because it offers us a future where we can benefit from LLMs without having to listen to tech CEOs saber rattle about removing America of its jobs so they can get the next fundraising round sorted
- cpill 8mo agoI wonder if we could have much smaller models if they train on less languages? i.e. python + yaml + json only or even an single languages with an cluster of models loaded into memory dynamically...?
- 0cf8612b2e1e 8mo agoWhat is the best place to see local model rankings? The benchmarks seem so heavily gamed that I am willing to believe the “objective” rankings are a lie and personal reviews are more meaningful. Are there any clear winners per domain? Code, voice-to-text, text-to-voice, text editing, image generation, text summarization, business-text-generation, music synthesis, whatever.
- blurbleblurble 8mo agoSo dang exciting! There are a bunch of new interesting small models out lately, by the way, this is just one of them...
- macmac_mac 8mo agoI just tried qwen 3 tts and it was mind blowingly good, you can even provide directions for the overall tone etc. Which wasn't the case when I used commercial super expensive products like the (now closed after being bought by meta) play.ht . Does anyone see a reason to still use elevenlabs etc. ?
- mmaunder 8mo agoThese guys are setting up to absolutely own the global south market for AI. Which is in line with the belt and road initiative.
- adefa 8mo agoBenchmarks using DGX Spark on vLLM 0.15.1.dev0+gf17644344 FP8: https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8 Sequential (single request) Prompt Gen Prompt Processing Token Gen Tokens Tokens (tokens/sec) (tokens/sec) ------ ------ ----------------- ----------- 521 49 3,157 44.2 1,033 83 3,917 43.7 2,057 77 3,937 43.6 4,105 77 4,453 43.2 8,201 77 4,710 42.2 Parallel (concurrent requests) pp4096+tg128 (4K context, 128 gen): n t/s -- ---- 1 28.5 2 39.0 4 50.4 8 57.5 16 61.4 32 62.0 pp8192+tg128 (8K context, 128 gen): n t/s -- ---- 1 21.6 2 27.1 4 31.9 8 32.7 16 33.7 32 31.7
- cmrdporcupine 8mo agoI tried the FP8 in vLLM on my Spark and although it fit in memory, I started swapping once I actually tried to run any queries, and, yeah, could not have a context larger than 8k. I figured out later this is because vLLM apparently de-quantizes to BF16 at runtime, so pointless to run the FP8? I get about 30-35 tok/second using llama.cpp and a 4-bit quant. And a 200+k context, using only 50GB of RAM.
- justaboutanyone 8mo agoRunning llama.cpp rather than vLLM, it's happy enough to run the FP8 variant with 200k+ context using about 90GB vram
- cmrdporcupine 8mo agoyeah, what did you get for tok/sec there though? Memory bandwidth is the limitation with these devices. With 4 bit I didn't get over 35-39 tok/sec, and averaged more like 30 when doing actual tool use with opencode. I can't imagine fp8 being faster.
- lysace 8mo agoIs it censored according to the wishes of the CCP?
- mirekrusin 8mo agoWho cares? If you don't like it, you can fine tune.
- deleted 8mo ago[deleted]
- lysace 8mo agoI think a lot of people care. Most decidedely not you.
- mirekrusin 8mo agoI think people care about open weights, so they can use it locally, including fine tuning like unalignment. There are of course people that when you give them something that did cost millions of dollars to build for free will complain and share with the world what exactly they're entitled to.
- asdfss674564 8mo agois the censor after post training or is it applied at the data set
- StevenNunez 8mo agoNot crazy about it. It keeps getting stuck in a loop and filling up the context window (131k, run locally). Kimi's been nice, even if a bit slow.
- lostmsu 8mo agoDid you apply RoPE?
- StevenNunez 8mo agoTil. Trying it now.
- StevenNunez 8mo agoThis made it work a lot better! Thank you.
- featherless 8mo agoI got openclaw to compete Qwen3-Coder-Next vs Minimax M2.1 simultaneously on my Mac Studio 512GB: https://clutch-assistant.github.io/model-comparison-report/ https://clutch-assistant.github.io/model-comparison-report/
- predkambrij 8mo ago17t/s on a laptop with 6GB VRAM and DDR5 system memory. Maximum of 100k context window (then it saturates VRAM). Quite amazing, but tbh I'll still use inference providers, because it's too slow and it's my only machine with "good" specs :) cat docker-compose.yml services: llamacpp: volumes: - llamacpp:/root container_name: llamacpp restart: unless-stopped image: ghcr.io/ggml-org/llama.cpp:server-cuda network_mode: host command: | -hf unsloth/Qwen3-Coder-Next-GGUF:Q4_K_XL --jinja --cpu-moe --n-gpu-layers 999 --ctx-size 102400 --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 --fit on # unsloth/gpt-oss-120b-GGUF:Q2_K deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] volumes: llamacpp:
- ltbarcly3 8mo agoHere's a tip: Never name anything new, next, neo, etc. You will have a problem when you try to name the thing after that!
- dk8996 8mo agoIs there a good way to enable this model within VSCode, looking for something like Copilot?
- SamDc73 8mo agoThis is model 12188, which claims to rival SOTA models while not even being in the same league. In terms of intelligence per compute, it’s probably the best model I can realistically run locally on my laptop for coding. It’s solid for scripting and small projects. I tried it on a mid-size codebase (~50k LOC), and the context window filled up almost immediately, making it basically unusable unless you’re extremely explicit about which files to touch. I tested it with a 8k context window but will try again with 32k and see if it becomes more practical. I think the main blocker for using local coding models more is the context window. A lot of work is going into making small models “smarter,” but for agentic coding that only gets you so far. No matter how smart the model is, an agent will blow through the context as soon as it reads a handful of files.
- halJordan 8mo agoThe small context window has been a recognized problem for a while now. Really only Google has the ability to use a good long context window
- anhner 8mo agoyou should look into using subagents, which each have their own context window and don't pollute the main one
- mirekrusin 8mo agoWhat are you talking about? Qwen3-Coder-Next supports 256k context. Did you wanted to say that you don't have enough memory to run it locally yourself?
- SamDc73 8mo agoYes! I tried to go as far as 32k on the context window but beyond that it won't be usable on my laptop (Ryzen AI 365, 32gb RAM and 6gb of VRAM)
- mirekrusin 8mo agoYou need minimum ie. 2x 24G GPUs for this model (you need 46GB minimum).
- yowang 8mo ago[dead]
- codedokode 8mo agoIt's sad they only have 80B version, given current RAM prices.
- aktenlage 8mo agoI tried it to review some C++ code. It actually found minor bugs, but the signal to noise ratio is too high (maybe 10% of the found issues were real issues)
- mark_l_watson 8mo agoA 3B resident parameter MOE allows absolutely huge savings on inference costs. I use a cloud provider for models to large to run locally, can’t wit for them to support qwen3-coder-next hopefully in a few days. So much expensive inference is provided free or at large discounts - that craziness should end.
- petya415 8mo agoqwen3-coder-next 80B (128k ctx) local benchmark on RTX 5090 + Ryzen 9 9950x3d ran 10 real coding workloads via ollama, kv cache q8. below is claude-generated report from raw numbers: Hardware: Corsair Vengeance a7500 AIR -- RTX 5090 32GB, Ryzen 9 9950X3D, 192GB DDR5, Fedora 43 (kernel 6.17.6) Workload TTFT Time Prompt Gen P tok/s G tok/s ------------------------------------------------------------------------------------ Short Code Gen (Dijkstra) 5.9s 68s 74 754 93.8 12.2 Code Comprehension (100L hash) 3.7s 271s 1,131 3,202 319.3 12.0 Bug Fixing (connection pool) 2.9s 478s 876 5,629 308.7 11.9 Security Review (Flask app) 2.7s 329s 758 3,905 285.6 12.0 System Design (persistent BST) 1.0s 407s 141 4,857 156.1 12.0 Code Refactoring 2.8s 242s 823 2,862 309.1 12.0 Tool Calling (single turn) 3.6s 9s 1,093 66 309.7 12.2 Tool Calling (multi-turn, 3T) 11.2s 43s 2,746 326 407.8 11.8 Needle in Haystack (~15k ctx) 29.4s 210s 10,576 1,962 361.2 10.9 Long Generation (mini-Celery) 1.1s 1,001s 238 11,342 242.3 11.4 ------------------------------------------------------------------------------------ TOTAL ~51min 18,456 34,905 11.4 avg Key observations: - Model is 51GB, runs 53% CPU / 47% GPU split since it exceeds the 5090's 32GB VRAM. - Generation throughput is a dead-flat 12 tok/s regardless of task complexity. The CPU is the bottleneck (53% of layers offloaded to RAM), not the GPU -- GPU utilization sits at 3-4%. - Prompt processing is fast: 280-410 tok/s. The 5090 helps here. - TTFT scales linearly with context: 1-6s for small prompts, 29s for the 10k+ token needle-in-haystack test. - Sustained generation holds steady -- the 16-minute long generation test (11,342 tokens, full distributed task queue implementation) showed no throughput degradation. - Quality was solid: correctly identified a buried bug report comment in 3000 lines of code (JIRA ticket number, proposed fix), found the N+1 query in the multi-turn agent test, caught all major OWASP vulns in the security review, generated correct tool calls with proper parameters. - Thinking mode was not enabled (/nothink). Would be interesting to compare with /think. The 12 tok/s ceiling is purely the CPU/RAM bandwidth wall from the offload split. If you have 48GB+ VRAM (dual GPU or a future card) this model would fly. On this setup it's usable but you feel the wait on anything over ~1000 tokens.