7 ms·
Consistency diffusion language models: Up to 14x faster, no quality loss
- refulgentis 8mo agoIf this means there’s a 2x-7x speed up available to a scaled diffusion model like Inception Mercury, that’ll be a game changer. It feels 10x faster already…
- blurbleblurble 8mo agoDiffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.
- meatmanek 8mo agoFeels like the sodium ion battery vs lithium ion battery thing, where there are theoretical benefits of one but the other has such a head start on commercialization that it'll take a long time to catch up.
- sroussey 8mo agoSame with digital vs analog
- esafak 8mo agoDigital came later but beat analog at almost everything?
- LarsDu88 8mo agoNot really. Unlike with physical goods like batteries, the hardware for training a diffusion vs an autoregressive language model is more or less exactly the same. Although the lab that did this research (Chris Re and Tri Dao are involved) is run by the world's experts in squeezing CUDA and Nvidia hardware for every last drop of performance. At the API level, the primary differences will be the addition of text infill capabilities for language generation. I also somewhat expect certain types of generation to be more cohesive (e.g. comedy or stories where you need to think of the punchline or ending first!)
- impossiblefork 8mo agoOne appeal of it is for RL. If it ends up being a lot faster for generation, you'll be able to do a lot more RL. If people can make RL scalable-- make it so that RL isn't just a final phase, but something which is as big as the supervised stuff, then diffusion models are going to have an advantage. If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actually refine their outputs, so we're not talking about some kind of refinement along the lines of: initial idea -> better idea -> something actually sound.
- storus 8mo agoRL suffers both from sparsity in a huge amount of dimensions as well as convergence brittleness when it's extremely difficult to get it to converge.
- refulgentis 8mo ago> If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actually refine their outputs, so we're not talking about some kind of refinement along the lines of: initial idea -> better idea -> something actually sound. I'm really curious about this, I'm but a simple client developer, so I don't actually grok some of the differences. For lack of a better word, there's a "normie" position that "omg diffusion means it can edit!!111! big unlock!" -- I think that's cute but I also don't see it as intuitively correct. And I guess I don't even know why I don't see it that way. But regardless, it sounds like I'm correct there. > If not, I think autoregressive models will still be preferred. But here I get lost, at least so far, diffusion models seem strictly significantly faster, and on par with models with the same parameter count. If that is the case, why would autoregressive models still be preferred? Asking this also makes me realize I am treating "diffusion models are better" as a premise, if I'm asserting they're always faster and ~same quality...
- storus 8mo agoDidn't thinking tokens resolve the most problematic part of autoregressive models (the first few tokens set the constraints the model can't overcome later) and give it a massive advantage compared to diffusion models by showing the thinking trace? I can see diffusion models being used as a draft model to quickly predict a bunch of tokens and let the autoregressive model decide to use them or throw them away quickly, speeding it up considerably while keeping thinking traces available.
- blurbleblurble 8mo agoThe reason I mentioned "purely autoregressive" is that realistically I expect hybrid diffusion + autoregressive models to be the first popular diffusion models. I could be wrong though. And diffusion models have other tricks like really easy integration with simple classifiers. Check out this paper where they use diffusion during inference on the autoencoded prediction of an autoregressive model: https://openreview.net/forum?id=c05qIG1Z2B https://openreview.net/forum?id=c05qIG1Z2B
- LarsDu88 8mo agoGoogle is working on a similar line of research. Wonder why they haven't rolled out a GPT40 scaled version of this yet
- vintermann 8mo agoProbably because it's expensive. But I wish there were more "let's scale this thing to the skies" experiments from those who actually can afford to scale things to the skies.
- yorwba 8mo agoScaling laws mean that there's not much need to actually scale things to the skies. Instead, you can run a bunch of experiments at small scale, fit the scaling law parameters, then extrapolate. If the predicted outcome is disappointing (e.g. it's unlikely to beat the previous scaled-to-the-sky model), you can save the really expensive experiment for a more promising approach. It would certainly be nice though if this kind of negative result was published more often instead of leaving people to guess why a seemingly useful innovation wasn't adopted in the end.
- yjftsjthsd-h 8mo agoIs anyone doing any form of diffusion language models that are actually practical to run today on the actual machine under my desk? There's loads of more "traditional" .gguf options (well, quants) that are practical even on shockingly weak hardware, and I've been seeing things that give me hope that diffusion is the next step forward, but so far it's all been early research prototypes.
- Bolwin 8mo agoBased on my experience running diffusion image models I really hope this isn't going to take over anytime soon. Parallel decoding may be great if you have a nice parallel gpu or npu but is dog slow for cpus
- janalsncm 8mo agoI worked on it for a more specialized task (query rewriting). It’s blazing fast. A lot of inference code is set up for autoregressive decoding now. Diffusion is less mature. Not sure if Ollama or llama cpp support it.
- stavros 8mo agoHow was the quality?
- philipportner 8mo agoDid you publish anything you could link wrt. query rewriting?
- MASNeo 8mo agoI wish there would be more of this research to speed things up rather than building ever larger models
- nl 8mo agoWhy not both? Scaling laws are real! But they don't preclude faster processing.
- wongarsu 8mo agoNotice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and Opus 3. OpenAI's o1 was $60/MTok, o1 pro $600/MTok, gpt-5.2 is $14/MTok and 5.2-pro is $168/MTok. Also note how GPT-4 was rumored to be in the 1.8T realm, and now Chinese models in the 1T realm can match or surpass it. And I doubt the Chinese have a monopoly on those efficiency improvements I doubt frontier models have actually substantially grown in size in the last 1.5 years, and potentially have a lot fewer parameters than the frontier models of old
- 5o1ecist 8mo ago[flagged]
- stavros 8mo agoWhy don't we need them? If I need to run a hundred small models to get a given level of quality, what's the difference to me between that and running one large model?
- spockz 8mo agoYou can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.
- nl 8mo agoReleasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/ https://taalas.com/products/
- aurareturn 8mo agoNothing to do with each other. This is a general optimization. Taalas' is an ASIC that runs a tiny 8B model on SRAM. But I wonder how Taalas' product can scale. Making a custom chip for one single tiny model is different than running any model trillions in size for a billion users. Roughly, 53B transistors for every 8B params. For a 2T param model, you'd need 13 trillion transistor assuming scale is linear. One chip uses 2.5 kW of power? That's 4x H100 GPUs. How does it draw so much power? If you assume that the frontier model is 1.5 trillion models, you'd need an entire N5 wafer chip to run it. And then if you need to change something in the model, you can't since it's physically printed on the chip. So this is something you do if you know you're going to use this exact model without changing anything for years. Very interesting tech for edge inference though. Robots and self driving can make use of these in the distant future if power draw comes down drastically. 2.4kW chip running inside a robot is not realistic. Maybe a 150w chip.
- spuz 8mo agoThe 2.5kW figure is for a server running 10 HC1 chips: > The first generation HC1 chip is implemented in the 6 nanometer N6 process from TSMC. ... Each HC1 chip has 53 billion transistors on the package, most of it very likely for ROM and SRAM memory. The HC1 card burns about 200 watts, says Bajic, and a two-socket X86 server with ten HC1 cards in it runs 2,500 watts. https://www.nextplatform.com/2026/02/19/taalas-etches-ai-models-onto-transistors-to-rocket-boost-inference/ https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...
- aurareturn 8mo agoI’m confused then. They need 10 of these to run an 8B model?
- bjt12345 8mo agoI do wonder why diffusion models aren't used alongside constraint decoding for programming - surely it makes better sense then using an auto-regressive model.
- bob1029 8mo agoDiffusion models need to infer the causality of language from within a symmetric architecture (information can flow forward or backward). AR forces information to flow in a single direction and is substantially easier to control as a result. The 2nd sentence in a paragraph of English text often cannot come before the first or the statement wouldn't make sense. Sometimes this is not an issue (and I think these are cases where parallel generation makes sense), but the edge cases are where all the money lives.
- bjt12345 8mo agoBut I do wonder if diffusion models will be used in more complex Software Architecture for their long-term coherence, no exposure bias, and their symmetric architecture could work well with interaction nets.
- hanifbbz 8mo agoIs this available as open source anywhere to try?
- LoganDark 8mo agoCan't wait for the day I can actually try a diffusion model on my own machine (128GB M4 Max) rather than as a hosted service. So far I haven't seen a single piece of software that supports it.
- janalsncm 8mo agoYou can try it today. You can get them from huggingface. Here is an example: https://huggingface.co/tencent/WeDLM-8B-Instruct https://huggingface.co/tencent/WeDLM-8B-Instruct Diffusion isn’t natively supported in the transformers library yet so you have to use their custom inference code.
- LoganDark 8mo agoI don't subscribe to the Python craze, but this could be interesting. Thanks!
- simonw 8mo agoI'd love to know what's going on with the Gemini Diffusion model - they had a preview last May and it was crazy fast but I've not heard anything since then.
- LarsDu88 8mo agoA lot of this post-training recipe feels reminiscent of DINO training (teacher/student, use of stop gradients). I wonder if the more recent leJEPA SigREG regularization research might be relevant here for simpler post-training.
- WiSaGaN 8mo agoI think diffusion makes much more sense than auto-regressive (AR) specifically in code generation comparing to chatbot.
- cubefox 8mo agoThis doesn't mention the drawback of diffusion language models, the main reason why nobody is using them: they have significantly lower performance on benchmarks than autoregressive models at similar size.
- deleted 8mo ago[deleted]
- fumeux_fume 8mo agoSeeing half of an AR LLM's output tokens go to generating a predefined json schema bothers me so much. I would love to have an option to use diffusion for infilling.
- jmalicki 8mo agoOne trick I learned for this was to use csv for LLM I/I and translate json <-> csv at the boundary layer
- fumeux_fume 8mo agoOh neat. So have the llm output csv instead of JSON and then convert it? How would handle nested structures?
- jmalicki 8mo agoDepending on how it's nested, you could denormalize, think of how you could denormalize a one-to-many SQL relationship So if you have a user that has many automobiles, maybe instead of Autos: [...] you could do Auto1Make Auto2Make etc.
- abeppu 8mo agoDiffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples a paren or a comma or something which is entirely plausible at that position, it can still close off an expansion which would be syntactically correct.
- naasking 8mo agoIIRC, some researchers are working on mixed AR+diffusion models for this sort of thing.
- abeppu 8mo agoI think the gap is, if they're building hybrids with _forward_ AR and diffusion, they risk giving up the cool part of diffusion which is reasoning back. I may be imposing unreasonable human biases on to this, but I really think it would be interesting to have the model engage with the structure of the text, rather than just being either a sequence or an array of tokens. E.g. "I'm going to _ tomorrow." If the _ is not just a token but an expansion in context, which might be a noun phrase, a verb phrase etc, it could be filled in with "the mall", "practice guitar". In code "if (_1) { return _2; }", _1 could be an expression whose type is bool, and which makes sense as a check to confirm that some process is finished. I don't care specifically how many tokens either of those is, but I do care that it makes sense in context.
- naasking 8mo agoI was thining of something like LLaDa that uses a Transformer to predict forward masked tokens: https://arxiv.org/abs/2502.09992 https://arxiv.org/abs/2502.09992
- moralestapia 8mo agoI think that having an early draft of the output is part of the appeal of this type of models.