3 ms·
Strengths and limitations of diffusion language models
- cubefox 1y agoThat's a nice explanation. I wonder whether autoregressive and diffusion language models could be combined such that the model only denoises the (most recent) end of a sequence of text, like a paragraph, while the rest is unchangeable and allows for key-value caching.
- gfysfm 1y agoHi, I wrote the post. Thank you! That’s how it does work, but unfortunately denoising the last paragraph requires computing attention scores for every token in that paragraph, which requires checking those tokens against every token in the sequence. So it’s still much less cacheable than the equivalent autoregressive model.
- billconan 1y agoI'm curious, in image generation, flow matching is said to be better than diffusion, then why do these language models still start from diffusion, instead of jumping to flow matching directly?
- gessha 1y agoThis is just a guess but I think it’s due to diffusion training being more popular so we’ve figured more of the kinks with those models. Flow matching models might follow after you figure out some of their hyperparameters.
- mountainriver 1y agoA big discussion on this happened here as well https://news.ycombinator.com/item?id=44057820 https://news.ycombinator.com/item?id=44057820 There is quite a bit of evidence diffusion models work better at reasoning because they don't suffer from early token bias. https://github.com/HKUNLP/diffusion-vs-ar https://github.com/HKUNLP/diffusion-vs-ar https://arxiv.org/html/2410.14157v3 https://arxiv.org/html/2410.14157v3
- accrual 1y agoGreat overview. I wonder if we'll start to see more text diffusion models from other players, or maybe even a mixture of diffusion and transformer models alternating roles behind a single UI, depending on the context and request.
- shrubhub 1y agoThe diffusion models are (or can be) transformer models! They're just not autoregressive.