7 ms·

Qwen3-Next

569 points by tosh 1y ago

Jgoauh 1y ago
Seems impressive, i believe better architectures are really the path forward, i don't think you need more than 100B params taking this model and what GPT OSS 120B can acchieve
- NitpickLawyer 1y ago
  New arch seems cool, and it's amazing that we have these published in the open. That being said, qwen models are extremely overfit. They can do some things well, but they are very limited in generalisation, compared to closed models. I don't know if it's simply scale, or training recipes, or regimes. But if you test it ood the models utterly fail to deliver, where the closed models still provide value.
  - vintermann 1y ago
    Could you give some practical examples? I don't know what Qwen's 36T-token training set is like, so I don't know what it's overfitting to...
    - NitpickLawyer 1y ago
      Take math and coding for example: - in math, if they can solve a problem, or a class of problems, they'll solve it. If you use a "thinking" model + maj@x, you'll get strong results. But if you try for example to have the model consider a particular way or method of exploring a problem, it'll default to "solving" mode. It's near impossible to have it do something else with a math problem, other than solving it. Say "explore this part, in this way, using this method". Can't do it. It'll maybe play a bit, but then enter "solving" mdoe and continue to solve it as it was trained. In practice, this means that "massive parallel" test time compute becomes harder to do with these models, because you can't "guide" them towards certain aspects of a problem. They are extremely "stubborn". - in coding it's even more obvious. Ask them to produce any 0shot often tested and often shown things (spa, game, visualisation, etc) - and they do it. Convincingly. But ask them to look at a piece of code and extract meaning, and they fail. Or ask them to reverse an implementation. Figure out what a function does and reverse its use, or make it do something else, and they fail.
      - elbear 1y ago
        It sounds like some people.
      - vintermann 1y ago
        Oof, that sounds frustrating. Yeah, I can relate to this failure mode, it's basically "did you mean (more likely query)" up to 11. It does sound like an artifact of the dialog/thinking tuning though.
      - CuriouslyC 1y ago
        That's the thing people miss that's so good about GPT5. It's incredibly steerable in a way a lot of models aren't.
- CuriouslyC 1y ago
  We definitely need more parameters, low param models are hallucination machines, though low actives is probably fine assuming the routing is good.
croemer 1y ago
ERR_NAME_NOT_RESOLVED
croemer 1y ago
https://archive.is/JH9XL https://archive.is/JH9XL
jychang 1y ago
Coolest part of Qwen3-Next, in my opinion, (after the linear attention parts) is that they do MTP without adding another un-embedding matrix. Deepseek R1 also has a MTP layer (layer 61) https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/model-00163-of-000163.safetensors https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/mod... But Deepseek R1 adds embed_tokens and shared_head.head tensors, which are [129280, 7168] or about 2GB in size at FP8. Qwen3-Next doesn't have that: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob/main/model-00041-of-00041.safetensors https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob... So it saves a few GB in active parameters for MTP, which is a Big Deal. This is one of the changes that helps significantly speeds up inference.
- puilp0502 1y ago
  What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?
  - rfoo 1y ago
    It could be a better draft model than separately trained EAGLE etc for speculative decoding.
  - jychang 1y ago
    Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower than just checking it). Usually, it's correct, so inference is a lot faster.
    - moffkalast 1y ago
      Hmm but isn't the checking only required because the draft model is not the same model and can only speculate what the main one is thinking, hence the name? If the main model generates two tokens itself, then how can it be wrong about its own predictions?
slimebot80 1y ago
Complete newbie here - some questions, if I may! This stuff can run on a local machine without internet access, correct? And it can pretty much match Nano Banana? https://github.com/PicoTrex/Awesome-Nano-Banana-images/blob/main/README_en.md https://github.com/PicoTrex/Awesome-Nano-Banana-images/blob/... Also -- what are the specs for a machine to run it (even if slowly!)
- prawel 1y ago
  what you mean is Qwen Image and Qwen Image Edit, you can run it on local machine, using Draw Things application for example. the model discussed here is text model, so similar to ChatGPT. You can also run it on your local machine, but not yet, as apps need to be updated with Qwen 3 Next support (llama.cpp, Ollama, etc)
- Davidzheng 1y ago
  Isn't this one a text model
  - slimebot80 1y ago
    Ah, maybe! I am lost reading this page with all the terminology
    - arcanemachiner 1y ago
      You'll get used to it. Make sure to lurk on r/LocalLlama.
      - diggan 1y ago
        > Make sure to lurk on r/LocalLlama. Please do take everything you read there with a bit of salt though, as the "hive-mind" effect is huge there, even when compared to other subreddits. I'm guessing the huge influx of money + reputations on the line + a high traffic community is ripe for both hive-minding + influence campaigns.
- NitpickLawyer 1y ago
mynti 1y ago
For anyone curious about what the Gated Delta Network is: https://arxiv.org/pdf/2412.06464 https://arxiv.org/pdf/2412.06464
- yorwba 1y ago
  Also, Gated Attention: https://arxiv.org/abs/2505.06708 https://arxiv.org/abs/2505.06708
yekanchi 1y ago
how much vram it requires?
- DiabloD3 1y ago
  Thats not a meaningful question. Models can be quantized to fit into much smaller memory requirements, and not all MoE layers (in MoE models) have to be offloaded to VRAM to maintain performance.
  - yekanchi 1y ago
    i mean 4bit quantized. i can roughly calculate vram for dense models by model size. but i don't know how to do it for MOE models?
    - EnPissant 1y ago
      MoE models need just as much VRAM as dense models because every token may use a different set of experts. They just run faster.
      - regularfry 1y ago
        This isn't quite right: it'll run with the full model loaded to RAM, swapping in the experts as it needs. It has turned out in the past that experts can be stable across more than one token so you're not swapping as much as you'd think. I don't know if that's been confirmed to still be true on recent MoEs, but I wouldn't be surprised.
        EnPissant 1y ago
        What you are describing would be uselessly slow and nobody does that.
        DiabloD3 1y ago
        I don't load all the MoE layers onto my GPU, and I have only about a 15% reduction in token generation speed while maintaining a model 2-3 times larger than VRAM alone.
irthomasthomas 1y ago
llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _ | .--.____.'._.-. | \ (_) | o -` .-` | | \ |`-._ _ _ _ _\ / \ | | `. |_||_| | | o | \_ \ | -. .-. |.-. \ `--..-' O | `.`-' .' _.' .' | `-.-' /-.__ ' .-' .' `-.` '.|='=.='=.='=.='=|._/_ `-'.' `-._ `. |________/\_____| `-.' .' ).| '=' '='\/ '=' | `._.` '---------------' //___\ //___\ || || ||_.-. ||_.-. (_.--__) (_.--__) Meta: I generated a few dozen spongebobs last night on the same model and NONE where as good as this. Most started well but collapsed into decoherence at the end - missing the legs off. Then this morning the very same prompt to the same model API produced a perfect bob on the first attempt. Can utilization affect response quality, if all else remains constant? Or was it just random luck? Edit: Ok, the very next attempt, a few minutes later, failed, so I guess it is just random, and you have about a 1 in 10 chance of getting a perfect spongebob from qwen3-coder, and ~0 chance with qwen3-next.
- dev_hugepages 1y ago
  memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants https://www.asciiart.eu/cartoons/spongebob-squarepants
  - ginko 1y ago
    Conveniently removed the artist's signature though.
    - eurekin 1y ago
      Certainly not defending LLMs here, don't mistake with that. Humans do it too. I have given up on my country's non-local information sources, because I could recognize original sources that are being deliberately omitted. There's a satiric webpage that is basically a reddit scrape. Most of users don't notice and those who do, don't seem to care.
keyle 1y ago
For a model that can run offline, they've nailed how the website can too. And it appears like it's thinking about it! /s
syntaxing 1y ago
The craziest part is how far MoE has come thanks to Qwen. This beats all those 72B dense models we’ve had before and runs faster than 14B model depending on how you off load your VRAM and CPU. That’s insane.
- moffkalast 1y ago
  In retrospect it's actually funny that last year Meta spent so many resources training a dense 405B model that both underperforms compared to models a tenth its size and is impossible to run at a reasonable speed on any hardware in existence.
  - NitpickLawyer 1y ago
    It's not that clear. Yes, it underperforms in recent benchmarks and usecases (i.e. agentic stuff), but it is still one of the strongest open models in terms of "knowledge". Dense does have that advantage of MoE, even if it's extremely expensive to run inference on. Check out this great exercise - https://open.substack.com/pub/outsidetext/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a https://open.substack.com/pub/outsidetext/p/how-does-a-blind...
    - moffkalast 1y ago
      Ok wow that is incredibly interesting, what a test. I would've honestly expected just random noise (like if you gave this same task a human, lol) but you can even see related models draw similar results. Maybe it is an indicator of overall knowledge, or how consistent the world model is. It also could not correlate at all with non-geographical knowledge.
  - jychang 1y ago
    Strong disagree. Llama 4's release in 2025 is (deservedly) panned, but Llama 3.1 405b does not deserve that slander. https://artificialanalysis.ai/#frontier-language-model-intelligence-over-time https://artificialanalysis.ai/#frontier-language-model-intel... Do not compare 2024 models to the current cutting edge. At the time, Llama 3.1 405b was the very first open source (open weights) model to come close to the closed source cutting edge. It was very very close in performance to GPT-4o and Claude 3.5 Sonnet. In essence, it was Deepseek R1 before Deepseek R1.
pveierland 1y ago
> "The content loading failed." It's amazing how far and how short we've come with software architectures.
techsystems 1y ago
How does the context length scaling at 256K tokens compare to Llama's 1M in terms of performance? How are the contexts treated differently?
jwr 1y ago
Hmm. 80B. These days I am on the lookout for new models in the 32B range, since that is what fits and runs comfortably on my MacBook Pro (M4, 64GB). I use ollama every day for spam filtering: gemma3:27b works great, but I use gpt-oss:20b on a daily basis because it's so much faster and comparable in performance.
- electroglyph 1y ago
  it'll run great, it's an moe.
  - deleted 1y ago
    [deleted]
- bigyabai 1y ago
  The model is 80b parameters, but only 3b are activated during inference. I'm running the old 2507 Qwen3 30B model on my 8gb Nvidia card and get very usable performance.
  - coolspot 1y ago
    Yes, but you don’t know which 3B parameters you will need, so you have to keep all 80B in your VRAM, or wait until correct 3B are loaded from NVMe->RAM->VRAM. And of course it could be different 3B for each next token.
    - drozycki 1y ago
      The latest SSDs benchmark at 3GB/s and up. The marginal latency would be trivial compared to the inference time.
  - jwr 1y ago
    I understand that, but whether it's usable depends on whether ollama can load parts of it into memory on my Mac, and how quickly.
    - 1y ago
boxboxbox4 1y ago
[dead]
Alifatisk 1y ago
Alibaba keeps releasing gold content I just tried Qwen3-Next-80B-A3B on Qwen chat, and it's fast! The quality seem to match Qwen3-235B-A22B. Quite impressive how they achieved this. Can't wait for the benchmarks at Artificial analysis According to Qwen Chat, Qwen3-Next has the following limits: Maximum context length: 262,144 tokens Max summary generation length: 32,768 tokens This is 2x higher on context length and 4x higher on summary generation compared to Qwen3-235B-A22B, damn > Qwen3-Next [...] excels in ultra-long-context understanding and complex tasks Even though their new hybrid architecture is fascinating, I think I'll continue to stick with Qwen2.5-Turbo because it's one of the few models that supports 1M tokens in context length. My use case is uploading large pdfs and ask questions across chapters
- gizmodo59 1y ago
  My take on long context for many frontier models is not about support but the accuracy drops drastically as you increase the context. Even if a model claims to support 10M context, reality is it doesn’t perform well when you saturate. Curious to hear others perspective on this
  - kridsdale3 1y ago
    This is my experience with Gemini. Yes, I really can put an entire codebase and all the docs and pre-dev discussions and all the inter-engineer chat logs in there. I still see the model becoming more intoxicated as turn count gets high.
    - patates 1y ago
      I use repomix to pack a full repository as an xml file and it works wonders. System prompt is very simple: please don't add any comments in the code unless explicitly asked to, including the ones that state what you changed. do not modify/remove any existing comments as long as they are valid. also output the full files that are changed (not the untouched ones), and no placeholders like "no change here" etc. do not output the xml parts in the output.xml file. focus on the individual files. before and after outputting code, write which file it would be and the path (not as a comment in the code but instead, before and after outputting code). Attached is a 400k token xml file, being the output of: https://pastebin.com/raw/SH6JHteg https://pastebin.com/raw/SH6JHteg Main prompt is a general description of the feature needed and PDF exports from figma. All done for free in aistudio and I consistently get better results than the people using claude code.
mmmllm 1y ago
The same week Oracle is forecasting huge data center demand and the stock is rallying. If these 10x gains in efficiency hold true then this could lead to a lot less demand for Nvidia, Oracle, Coreweave etc
- ls65536 1y ago
  I'm not going to speculate about what might be ahead in regards to Oracle's forecasting of data center demand, but regarding the idea of efficiency gains leading to lower demand, don't you think something like Jevons paradox might apply here?
- amelius 1y ago
  https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
  - mmmllm 1y ago
    Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users. Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other views.
    - sauwan 1y ago
      Long running agents?
    - amelius 1y ago
      If you can make an LLM solve a problem but from 100 different angles at the same time, that's worth something.
      - mmmllm 1y ago
        Isn't that essentially how the MoE models already work? Besides, if that were infinitely scalable, wouldn't we have a subset of super-smart models already at very high cost? Besides, this would only apply for very few use cases. For a lot of basic customer care work, programming, quick research, I would say LLMs are already quite good without running it 100X.
kristopolous 1y ago
I was getting a bunch of strange hallucinations and weird dialog. It sounds like some exasperated person on the verge of a mental breakdown
Western0 1y ago
where is gguf?
- daemonologist 1y ago
  Patience, it just came out yesterday and has some architectural changes.
- gre 1y ago
  https://github.com/ggml-org/llama.cpp/issues/15940 https://github.com/ggml-org/llama.cpp/issues/15940
pollre 1y ago
[dead]
binary132 1y ago
I’ve been using gpt-oss-120B with CPU MoE offloading on a 24GB GPU and it’s very usable. Excited to see if I can get good results on this now!
deleted 1y ago
[deleted]
esafak 1y ago
ICYMI qwen3-max was released last week.
- cubefox 1y ago
  That's only Qwen3-Max-Preview. In chat.qwen.ai it doesn't have a reasoning mode.
- Alifatisk 1y ago
  Was Qwen3-max better than Qwen3-235B-A22B-2507 at anything? Except higher token limit?
  - esafak 1y ago
    According to benchmarks. Scroll to the bottom of https://lmarena.ai/leaderboard/ https://lmarena.ai/leaderboard/
KronisLV 1y ago
> The Qwen3-Next-80B-A3B-Instruct performs comparably to our flagship model Qwen3-235B-A22B-Instruct-2507, and shows clear advantages in tasks requiring ultra-long context (up to 256K tokens). This is pretty impressive and a bit like how the GPT-OSS-120B came out and scored pretty well on the benchmarks despite its somewhat limited size. That said, using LLMs for software dev use cases, I wouldn't call 256K tokens "ultra-long" context, I regularly go over 100K when working on tasks with bigger scope, e.g.: Look at the existing code related to this functionality and the existing design patterns in the code as well as the guidelines. Then plan out the implementation in detail and ask me a few questions along the way to figure the details out better. Finally, based on everything so far, do the actual implementation. Then look it over and tell me if anything has been missed from the plan, then refactor the code in any number of ways. It could be split up into multiple separate tasks, but I find that the context being more complete (unless the model starts looping garbage, which poisons the context) leads to better results. My current setup of running Qwen3 Coder 480B on Cerebras bumps into the 131K token limit. If not for the inference speed there (seriously great) and good enough model quality, I'd probably look more in the direction of Gemini or Claude again.
pzo 1y ago
would be interesting how they compare to gpt-oss-120b. The latter one runs also very fast and pricing is currently much better than qwen3-next on many providers. Would expect that if this model is such fast pricing should be similar or even lower.
jug 1y ago
> The Qwen3-Next-80B-A3B-Instruct performs comparably to our flagship model Qwen3-235B-A22B-Instruct-2507 I'm skeptical about these claims. How can this be? Wouldn't there be massive loss of world knowledge? I'm particularly skeptical because a recent trend in Q2 2025 has been benchmaxxing.
- dragonwriter 1y ago
  > I'm skeptical about these claims. How can this be? More efficient architecture. > Wouldn't there be massive loss of world knowledge? If you assume equally efficient architecture and no other salient differences, yes, that’s what you’d expect from a smaller model.
  - jug 1y ago
    Hmm. Let's just say if this is true, that this is actually better with such a much lower total parameter count, it's the greatest accomplishment in over a year of LLM development. With the backdrop of bechmaxxing in 2025, I'll believe in this when I see the results on closed benchmarks and SimpleBench. My concern is this might be a hallucination machine.
    - bigyabai 1y ago
      Might be. FWIW, my experience with the Qwen3 30b model basically took ChatGPT out of rotation for me. It's not hard for me to imagine an 80b model pushing that further, especially with thinking enabled. I recommend playing with the free hosted models to draw your own conclusions: https://chat.qwen.ai/ https://chat.qwen.ai/
    - KaoruAoiShiho 1y ago
      In my testing this model is quite bad and far behind 235b a22b. https://fiction.live/stories/Fiction-liveBench-Sept-12-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Sept-12-2025/...
siliconc0w 1y ago
All these new datacenters are going to be a huge sunk cost. Why would you pay OpenAI when you can host your own hyper efficient Chinese model for like 90% less cost at 90% of the performance. At that is compared to today's subsidized pricing, which they can't keep up forever.
- hadlock 1y ago
  Eventually Nvidia or a shrewd competitor will release 64/128gb consumer cards; locally hosted GPT 3.5+ is right around the corner, we're just waiting for consumer hardware to catch up at this point.
  - mft_ 1y ago
    I think we're still at least an order of magnitude away (in terms of affordable local inference, or model improvements to squeeze more from less, or a combination of the two) from local solutions being seriously competitive for general purpose tasks, sadly. I recently bought a second-hand 64GB Mac to experiment with. Even with the biggest recent local model it can run (llama3.3:70b just about runs acceptably; I've also tried an array of Qwen3 30b variants) the quality is lacking for coding support. They can sometimes write and iterate on a simple Python script, but sometimes fail, and for general-purpose models, often fail to answer questions accurately (not unsurprisingly, considering the model is a compression of knowledge, and these are comparatively small models). They are far, far away from the quality and ability of currently available Claude/Gemini/ChatGPT models. And even with a good eBay deal, the Mac cost the current equivalent of ~6 years of a monthly subscription to one of these. Based on the current state of play, once we can access relatively affordable systems with 512-1024GB fast (v)ram and sufficient FLOPs to match, we might have a meaningfully powerful local solution. Until then, I fear local only is for enthusiasts/hobbyists and niche non-general tasks.
    - hadlock 1y ago
      It would not surprise me at all to see 512, 768, 1024 gb models targeted at commercial or home users in the next 5 years. I can imagine a lot of companies, regulated ones in particular like finance, defense, medical, wanting to run the models in house, inside their own datacenter. A single card or pair of cards would probably be more than adequate for a thousand or more users, or half a dozen developers. If you already have a $25,000 database server, $12,000 for an "ai server" isn't a wild ask.
cchance 1y ago
Those rope tests are impressive AF
davidpfarrell 1y ago
Hyped for the release, but bummed they fell for the ‘next’ naming convention. What will the actual next advanced release be called: * next-next * next (2) * actual-next-final
lend000 1y ago
Prediction: AI will become commoditized ~15 IQ points higher than the state of the art models today, and with larger context, within 4 years as the incremental improvements in training from synthetic data plateaus (we've already used all the "real" data out there) and open source models are cheaply trained on the outputs of the big money models. Then AI development stagnates until someone invents an effective way to use competitive reinforcement learning to train generalized intelligence (similar to how AlphaGo was trained), removing the need for vast quantities of training data. Then, we get real AGI.
- rmuratov 1y ago
  How did we use "all the data"? New knowledge appears on the internet every day, new scientific articles and videos are published.
  - lend000 1y ago
    At the speeds AI is moving, we've effectively used it all; the high quality data you need to make smarter models is coming in at a trickle. We're not getting 10^5 Principia Mathematicas published every day. Maybe I just don't have the vision to understand it, but it seems like AI-generated synthetic data for training shouldn't be able to make a smarter model than whatever produced that data. I can imagine synthetic data would be useful for making models more efficient (that's what quantized models are, after all), but not pushing the frontier.
- HardCodedBias 1y ago
  If that's true and if today's frontier models are around 120 IQ (who knows if that is true, but let's run with it, source: https://www.trackingai.org/home https://www.trackingai.org/home) then we'll have an enormous number of ~135 IQ bots with nearly unlimited conscientiousness. I can't even begin to understand what that would mean.
  - lend000 1y ago
    Very interesting time to be alive.
jbellis 1y ago
Added Qwen3 Next to the Brokk Power Ranking Open Round (coding benchmark). It's roughly GPT-OSS-20b strength. Full set of open weight model results: https://brokk.ai/power-ranking?version=openround&models=ds-r1.1%2Cds-v3.1%2Cglm4.5%2Cglm4.5-air%2Cgpt-oss-120b%2Cgpt-oss-20b%2Ck2%2Cq3c%2Cq3c-fp8%2Cq3next%2Cr1%2Cv3 https://brokk.ai/power-ranking?version=openround&models=ds-r...
- noahbp 1y ago
  Is that the updated Kimi K2, or the old Kimi k2?
  - jbellis 1y ago
    It's the original. I'll update the label to clarify.
- SparkyMcUnicorn 1y ago
  This would be a valuable benchmark if it included languages other than Java, and let me see which models are best at the languages I work with. My real-world usage does not line up with these results, but I'm not working with Java.

569 points by tosh 1y ago

Seems impressive, i believe better architectures are really the path forward, i don't think you need more than 100B params taking this model and what GPT OSS 120B can acchieve

NitpickLawyer 1y ago
New arch seems cool, and it's amazing that we have these published in the open. That being said, qwen models are extremely overfit. They can do some things well, but they are very limited in generalisation, compared to closed models. I don't know if it's simply scale, or training recipes, or regimes. But if you test it ood the models utterly fail to deliver, where the closed models still provide value.
- vintermann 1y ago
  Could you give some practical examples? I don't know what Qwen's 36T-token training set is like, so I don't know what it's overfitting to...
  - NitpickLawyer 1y ago
    Take math and coding for example: - in math, if they can solve a problem, or a class of problems, they'll solve it. If you use a "thinking" model + maj@x, you'll get strong results. But if you try for example to have the model consider a particular way or method of exploring a problem, it'll default to "solving" mode. It's near impossible to have it do something else with a math problem, other than solving it. Say "explore this part, in this way, using this method". Can't do it. It'll maybe play a bit, but then enter "solving" mdoe and continue to solve it as it was trained. In practice, this means that "massive parallel" test time compute becomes harder to do with these models, because you can't "guide" them towards certain aspects of a problem. They are extremely "stubborn". - in coding it's even more obvious. Ask them to produce any 0shot often tested and often shown things (spa, game, visualisation, etc) - and they do it. Convincingly. But ask them to look at a piece of code and extract meaning, and they fail. Or ask them to reverse an implementation. Figure out what a function does and reverse its use, or make it do something else, and they fail.
    - elbear 1y ago
      It sounds like some people.
    - vintermann 1y ago
      Oof, that sounds frustrating. Yeah, I can relate to this failure mode, it's basically "did you mean (more likely query)" up to 11. It does sound like an artifact of the dialog/thinking tuning though.
    - CuriouslyC 1y ago
      That's the thing people miss that's so good about GPT5. It's incredibly steerable in a way a lot of models aren't.
CuriouslyC 1y ago
We definitely need more parameters, low param models are hallucination machines, though low actives is probably fine assuming the routing is good.

croemer 1y ago

ERR_NAME_NOT_RESOLVED

croemer 1y ago

https://archive.is/JH9XL https://archive.is/JH9XL

jychang 1y ago

Coolest part of Qwen3-Next, in my opinion, (after the linear attention parts) is that they do MTP without adding another un-embedding matrix. Deepseek R1 also has a MTP layer (layer 61) https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/model-00163-of-000163.safetensors https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/mod... But Deepseek R1 adds embed_tokens and shared_head.head tensors, which are [129280, 7168] or about 2GB in size at FP8. Qwen3-Next doesn't have that: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob/main/model-00041-of-00041.safetensors https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob... So it saves a few GB in active parameters for MTP, which is a Big Deal. This is one of the changes that helps significantly speeds up inference.

puilp0502 1y ago
What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?
- rfoo 1y ago
  It could be a better draft model than separately trained EAGLE etc for speculative decoding.
- jychang 1y ago
  Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower than just checking it). Usually, it's correct, so inference is a lot faster.
  - moffkalast 1y ago
    Hmm but isn't the checking only required because the draft model is not the same model and can only speculate what the main one is thinking, hence the name? If the main model generates two tokens itself, then how can it be wrong about its own predictions?

slimebot80 1y ago

Complete newbie here - some questions, if I may! This stuff can run on a local machine without internet access, correct? And it can pretty much match Nano Banana? https://github.com/PicoTrex/Awesome-Nano-Banana-images/blob/main/README_en.md https://github.com/PicoTrex/Awesome-Nano-Banana-images/blob/... Also -- what are the specs for a machine to run it (even if slowly!)

prawel 1y ago
what you mean is Qwen Image and Qwen Image Edit, you can run it on local machine, using Draw Things application for example. the model discussed here is text model, so similar to ChatGPT. You can also run it on your local machine, but not yet, as apps need to be updated with Qwen 3 Next support (llama.cpp, Ollama, etc)
Davidzheng 1y ago
Isn't this one a text model
- slimebot80 1y ago
  Ah, maybe! I am lost reading this page with all the terminology
  - arcanemachiner 1y ago
    You'll get used to it. Make sure to lurk on r/LocalLlama.
    - diggan 1y ago
      > Make sure to lurk on r/LocalLlama. Please do take everything you read there with a bit of salt though, as the "hive-mind" effect is huge there, even when compared to other subreddits. I'm guessing the huge influx of money + reputations on the line + a high traffic community is ripe for both hive-minding + influence campaigns.
NitpickLawyer 1y ago

mynti 1y ago

For anyone curious about what the Gated Delta Network is: https://arxiv.org/pdf/2412.06464 https://arxiv.org/pdf/2412.06464

yorwba 1y ago
Also, Gated Attention: https://arxiv.org/abs/2505.06708 https://arxiv.org/abs/2505.06708

yekanchi 1y ago

how much vram it requires?

DiabloD3 1y ago
Thats not a meaningful question. Models can be quantized to fit into much smaller memory requirements, and not all MoE layers (in MoE models) have to be offloaded to VRAM to maintain performance.
- yekanchi 1y ago
  i mean 4bit quantized. i can roughly calculate vram for dense models by model size. but i don't know how to do it for MOE models?
  - EnPissant 1y ago
    MoE models need just as much VRAM as dense models because every token may use a different set of experts. They just run faster.
    - regularfry 1y ago
      This isn't quite right: it'll run with the full model loaded to RAM, swapping in the experts as it needs. It has turned out in the past that experts can be stable across more than one token so you're not swapping as much as you'd think. I don't know if that's been confirmed to still be true on recent MoEs, but I wouldn't be surprised.
      - EnPissant 1y ago
        What you are describing would be uselessly slow and nobody does that.
        DiabloD3 1y ago
        I don't load all the MoE layers onto my GPU, and I have only about a 15% reduction in token generation speed while maintaining a model 2-3 times larger than VRAM alone.

irthomasthomas 1y ago

llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _ | .--.____.'._.-. | \ (_) | o -` .-` | | \ |`-._ _ _ _ _\ / \ | | `. |_||_| | | o | \_ \ | -. .-. |.-. \ `--..-' O | `.`-' .' _.' .' | `-.-' /-.__ ' .-' .' `-.` '.|='=.='=.='=.='=|._/_ `-'.' `-._ `. |________/\_____| `-.' .' ).| '=' '='\/ '=' | `._.` '---------------' //___\ //___\ || || ||_.-. ||_.-. (_.--__) (_.--__) Meta: I generated a few dozen spongebobs last night on the same model and NONE where as good as this. Most started well but collapsed into decoherence at the end - missing the legs off. Then this morning the very same prompt to the same model API produced a perfect bob on the first attempt. Can utilization affect response quality, if all else remains constant? Or was it just random luck? Edit: Ok, the very next attempt, a few minutes later, failed, so I guess it is just random, and you have about a 1 in 10 chance of getting a perfect spongebob from qwen3-coder, and ~0 chance with qwen3-next.

dev_hugepages 1y ago
memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants https://www.asciiart.eu/cartoons/spongebob-squarepants
- ginko 1y ago
  Conveniently removed the artist's signature though.
  - eurekin 1y ago
    Certainly not defending LLMs here, don't mistake with that. Humans do it too. I have given up on my country's non-local information sources, because I could recognize original sources that are being deliberately omitted. There's a satiric webpage that is basically a reddit scrape. Most of users don't notice and those who do, don't seem to care.

keyle 1y ago

For a model that can run offline, they've nailed how the website can too. And it appears like it's thinking about it! /s

syntaxing 1y ago

The craziest part is how far MoE has come thanks to Qwen. This beats all those 72B dense models we’ve had before and runs faster than 14B model depending on how you off load your VRAM and CPU. That’s insane.

moffkalast 1y ago
In retrospect it's actually funny that last year Meta spent so many resources training a dense 405B model that both underperforms compared to models a tenth its size and is impossible to run at a reasonable speed on any hardware in existence.
- NitpickLawyer 1y ago
  It's not that clear. Yes, it underperforms in recent benchmarks and usecases (i.e. agentic stuff), but it is still one of the strongest open models in terms of "knowledge". Dense does have that advantage of MoE, even if it's extremely expensive to run inference on. Check out this great exercise - https://open.substack.com/pub/outsidetext/p/how-does-a-blind-model-see-the-earth?selection=5413dcae-b9f4-4adb-8826-d48e3908de2a https://open.substack.com/pub/outsidetext/p/how-does-a-blind...
  - moffkalast 1y ago
    Ok wow that is incredibly interesting, what a test. I would've honestly expected just random noise (like if you gave this same task a human, lol) but you can even see related models draw similar results. Maybe it is an indicator of overall knowledge, or how consistent the world model is. It also could not correlate at all with non-geographical knowledge.
- jychang 1y ago
  Strong disagree. Llama 4's release in 2025 is (deservedly) panned, but Llama 3.1 405b does not deserve that slander. https://artificialanalysis.ai/#frontier-language-model-intelligence-over-time https://artificialanalysis.ai/#frontier-language-model-intel... Do not compare 2024 models to the current cutting edge. At the time, Llama 3.1 405b was the very first open source (open weights) model to come close to the closed source cutting edge. It was very very close in performance to GPT-4o and Claude 3.5 Sonnet. In essence, it was Deepseek R1 before Deepseek R1.

pveierland 1y ago

> "The content loading failed." It's amazing how far and how short we've come with software architectures.

techsystems 1y ago

How does the context length scaling at 256K tokens compare to Llama's 1M in terms of performance? How are the contexts treated differently?

jwr 1y ago

Hmm. 80B. These days I am on the lookout for new models in the 32B range, since that is what fits and runs comfortably on my MacBook Pro (M4, 64GB). I use ollama every day for spam filtering: gemma3:27b works great, but I use gpt-oss:20b on a daily basis because it's so much faster and comparable in performance.

electroglyph 1y ago
it'll run great, it's an moe.
- deleted 1y ago
  [deleted]
bigyabai 1y ago
The model is 80b parameters, but only 3b are activated during inference. I'm running the old 2507 Qwen3 30B model on my 8gb Nvidia card and get very usable performance.
- coolspot 1y ago
  Yes, but you don’t know which 3B parameters you will need, so you have to keep all 80B in your VRAM, or wait until correct 3B are loaded from NVMe->RAM->VRAM. And of course it could be different 3B for each next token.
  - drozycki 1y ago
    The latest SSDs benchmark at 3GB/s and up. The marginal latency would be trivial compared to the inference time.
- jwr 1y ago
  I understand that, but whether it's usable depends on whether ollama can load parts of it into memory on my Mac, and how quickly.
  - 1y ago

boxboxbox4 1y ago

[dead]

Alifatisk 1y ago

Alibaba keeps releasing gold content I just tried Qwen3-Next-80B-A3B on Qwen chat, and it's fast! The quality seem to match Qwen3-235B-A22B. Quite impressive how they achieved this. Can't wait for the benchmarks at Artificial analysis According to Qwen Chat, Qwen3-Next has the following limits: Maximum context length: 262,144 tokens Max summary generation length: 32,768 tokens This is 2x higher on context length and 4x higher on summary generation compared to Qwen3-235B-A22B, damn > Qwen3-Next [...] excels in ultra-long-context understanding and complex tasks Even though their new hybrid architecture is fascinating, I think I'll continue to stick with Qwen2.5-Turbo because it's one of the few models that supports 1M tokens in context length. My use case is uploading large pdfs and ask questions across chapters

gizmodo59 1y ago
My take on long context for many frontier models is not about support but the accuracy drops drastically as you increase the context. Even if a model claims to support 10M context, reality is it doesn’t perform well when you saturate. Curious to hear others perspective on this
- kridsdale3 1y ago
  This is my experience with Gemini. Yes, I really can put an entire codebase and all the docs and pre-dev discussions and all the inter-engineer chat logs in there. I still see the model becoming more intoxicated as turn count gets high.
  - patates 1y ago
    I use repomix to pack a full repository as an xml file and it works wonders. System prompt is very simple: please don't add any comments in the code unless explicitly asked to, including the ones that state what you changed. do not modify/remove any existing comments as long as they are valid. also output the full files that are changed (not the untouched ones), and no placeholders like "no change here" etc. do not output the xml parts in the output.xml file. focus on the individual files. before and after outputting code, write which file it would be and the path (not as a comment in the code but instead, before and after outputting code). Attached is a 400k token xml file, being the output of: https://pastebin.com/raw/SH6JHteg https://pastebin.com/raw/SH6JHteg Main prompt is a general description of the feature needed and PDF exports from figma. All done for free in aistudio and I consistently get better results than the people using claude code.

mmmllm 1y ago

The same week Oracle is forecasting huge data center demand and the stock is rallying. If these 10x gains in efficiency hold true then this could lead to a lot less demand for Nvidia, Oracle, Coreweave etc

ls65536 1y ago
I'm not going to speculate about what might be ahead in regards to Oracle's forecasting of data center demand, but regarding the idea of efficiency gains leading to lower demand, don't you think something like Jevons paradox might apply here?
amelius 1y ago
https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- mmmllm 1y ago
  Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users. Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other views.
  - sauwan 1y ago
    Long running agents?
  - amelius 1y ago
    If you can make an LLM solve a problem but from 100 different angles at the same time, that's worth something.
    - mmmllm 1y ago
      Isn't that essentially how the MoE models already work? Besides, if that were infinitely scalable, wouldn't we have a subset of super-smart models already at very high cost? Besides, this would only apply for very few use cases. For a lot of basic customer care work, programming, quick research, I would say LLMs are already quite good without running it 100X.

kristopolous 1y ago

I was getting a bunch of strange hallucinations and weird dialog. It sounds like some exasperated person on the verge of a mental breakdown

Western0 1y ago

where is gguf?

daemonologist 1y ago
Patience, it just came out yesterday and has some architectural changes.
gre 1y ago
https://github.com/ggml-org/llama.cpp/issues/15940 https://github.com/ggml-org/llama.cpp/issues/15940

pollre 1y ago

[dead]

binary132 1y ago

I’ve been using gpt-oss-120B with CPU MoE offloading on a 24GB GPU and it’s very usable. Excited to see if I can get good results on this now!

deleted 1y ago

[deleted]

esafak 1y ago

ICYMI qwen3-max was released last week.

cubefox 1y ago
That's only Qwen3-Max-Preview. In chat.qwen.ai it doesn't have a reasoning mode.
Alifatisk 1y ago
Was Qwen3-max better than Qwen3-235B-A22B-2507 at anything? Except higher token limit?
- esafak 1y ago
  According to benchmarks. Scroll to the bottom of https://lmarena.ai/leaderboard/ https://lmarena.ai/leaderboard/

KronisLV 1y ago

> The Qwen3-Next-80B-A3B-Instruct performs comparably to our flagship model Qwen3-235B-A22B-Instruct-2507, and shows clear advantages in tasks requiring ultra-long context (up to 256K tokens). This is pretty impressive and a bit like how the GPT-OSS-120B came out and scored pretty well on the benchmarks despite its somewhat limited size. That said, using LLMs for software dev use cases, I wouldn't call 256K tokens "ultra-long" context, I regularly go over 100K when working on tasks with bigger scope, e.g.: Look at the existing code related to this functionality and the existing design patterns in the code as well as the guidelines. Then plan out the implementation in detail and ask me a few questions along the way to figure the details out better. Finally, based on everything so far, do the actual implementation. Then look it over and tell me if anything has been missed from the plan, then refactor the code in any number of ways. It could be split up into multiple separate tasks, but I find that the context being more complete (unless the model starts looping garbage, which poisons the context) leads to better results. My current setup of running Qwen3 Coder 480B on Cerebras bumps into the 131K token limit. If not for the inference speed there (seriously great) and good enough model quality, I'd probably look more in the direction of Gemini or Claude again.

pzo 1y ago

would be interesting how they compare to gpt-oss-120b. The latter one runs also very fast and pricing is currently much better than qwen3-next on many providers. Would expect that if this model is such fast pricing should be similar or even lower.

jug 1y ago

> The Qwen3-Next-80B-A3B-Instruct performs comparably to our flagship model Qwen3-235B-A22B-Instruct-2507 I'm skeptical about these claims. How can this be? Wouldn't there be massive loss of world knowledge? I'm particularly skeptical because a recent trend in Q2 2025 has been benchmaxxing.

dragonwriter 1y ago
> I'm skeptical about these claims. How can this be? More efficient architecture. > Wouldn't there be massive loss of world knowledge? If you assume equally efficient architecture and no other salient differences, yes, that’s what you’d expect from a smaller model.
- jug 1y ago
  Hmm. Let's just say if this is true, that this is actually better with such a much lower total parameter count, it's the greatest accomplishment in over a year of LLM development. With the backdrop of bechmaxxing in 2025, I'll believe in this when I see the results on closed benchmarks and SimpleBench. My concern is this might be a hallucination machine.
  - bigyabai 1y ago
    Might be. FWIW, my experience with the Qwen3 30b model basically took ChatGPT out of rotation for me. It's not hard for me to imagine an 80b model pushing that further, especially with thinking enabled. I recommend playing with the free hosted models to draw your own conclusions: https://chat.qwen.ai/ https://chat.qwen.ai/
  - KaoruAoiShiho 1y ago
    In my testing this model is quite bad and far behind 235b a22b. https://fiction.live/stories/Fiction-liveBench-Sept-12-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Sept-12-2025/...

siliconc0w 1y ago

All these new datacenters are going to be a huge sunk cost. Why would you pay OpenAI when you can host your own hyper efficient Chinese model for like 90% less cost at 90% of the performance. At that is compared to today's subsidized pricing, which they can't keep up forever.

hadlock 1y ago
Eventually Nvidia or a shrewd competitor will release 64/128gb consumer cards; locally hosted GPT 3.5+ is right around the corner, we're just waiting for consumer hardware to catch up at this point.
- mft_ 1y ago
  I think we're still at least an order of magnitude away (in terms of affordable local inference, or model improvements to squeeze more from less, or a combination of the two) from local solutions being seriously competitive for general purpose tasks, sadly. I recently bought a second-hand 64GB Mac to experiment with. Even with the biggest recent local model it can run (llama3.3:70b just about runs acceptably; I've also tried an array of Qwen3 30b variants) the quality is lacking for coding support. They can sometimes write and iterate on a simple Python script, but sometimes fail, and for general-purpose models, often fail to answer questions accurately (not unsurprisingly, considering the model is a compression of knowledge, and these are comparatively small models). They are far, far away from the quality and ability of currently available Claude/Gemini/ChatGPT models. And even with a good eBay deal, the Mac cost the current equivalent of ~6 years of a monthly subscription to one of these. Based on the current state of play, once we can access relatively affordable systems with 512-1024GB fast (v)ram and sufficient FLOPs to match, we might have a meaningfully powerful local solution. Until then, I fear local only is for enthusiasts/hobbyists and niche non-general tasks.
  - hadlock 1y ago
    It would not surprise me at all to see 512, 768, 1024 gb models targeted at commercial or home users in the next 5 years. I can imagine a lot of companies, regulated ones in particular like finance, defense, medical, wanting to run the models in house, inside their own datacenter. A single card or pair of cards would probably be more than adequate for a thousand or more users, or half a dozen developers. If you already have a $25,000 database server, $12,000 for an "ai server" isn't a wild ask.

cchance 1y ago

Those rope tests are impressive AF

davidpfarrell 1y ago

Hyped for the release, but bummed they fell for the ‘next’ naming convention. What will the actual next advanced release be called: * next-next * next (2) * actual-next-final

lend000 1y ago

Prediction: AI will become commoditized ~15 IQ points higher than the state of the art models today, and with larger context, within 4 years as the incremental improvements in training from synthetic data plateaus (we've already used all the "real" data out there) and open source models are cheaply trained on the outputs of the big money models. Then AI development stagnates until someone invents an effective way to use competitive reinforcement learning to train generalized intelligence (similar to how AlphaGo was trained), removing the need for vast quantities of training data. Then, we get real AGI.

rmuratov 1y ago
How did we use "all the data"? New knowledge appears on the internet every day, new scientific articles and videos are published.
- lend000 1y ago
  At the speeds AI is moving, we've effectively used it all; the high quality data you need to make smarter models is coming in at a trickle. We're not getting 10^5 Principia Mathematicas published every day. Maybe I just don't have the vision to understand it, but it seems like AI-generated synthetic data for training shouldn't be able to make a smarter model than whatever produced that data. I can imagine synthetic data would be useful for making models more efficient (that's what quantized models are, after all), but not pushing the frontier.
HardCodedBias 1y ago
If that's true and if today's frontier models are around 120 IQ (who knows if that is true, but let's run with it, source: https://www.trackingai.org/home https://www.trackingai.org/home) then we'll have an enormous number of ~135 IQ bots with nearly unlimited conscientiousness. I can't even begin to understand what that would mean.
- lend000 1y ago
  Very interesting time to be alive.

jbellis 1y ago

Added Qwen3 Next to the Brokk Power Ranking Open Round (coding benchmark). It's roughly GPT-OSS-20b strength. Full set of open weight model results: https://brokk.ai/power-ranking?version=openround&models=ds-r1.1%2Cds-v3.1%2Cglm4.5%2Cglm4.5-air%2Cgpt-oss-120b%2Cgpt-oss-20b%2Ck2%2Cq3c%2Cq3c-fp8%2Cq3next%2Cr1%2Cv3 https://brokk.ai/power-ranking?version=openround&models=ds-r...

noahbp 1y ago
Is that the updated Kimi K2, or the old Kimi k2?
- jbellis 1y ago
  It's the original. I'll update the label to clarify.
SparkyMcUnicorn 1y ago
This would be a valuable benchmark if it included languages other than Java, and let me see which models are best at the languages I work with. My real-world usage does not line up with these results, but I'm not working with Java.