4 ms·
Continuous Diffusion Language Models (CDLM's)
- amelius 2mo ago"Attention is all you need" should be renamed into "Attention is sufficient but not necessary".
- ViscountPenguin 2mo agoA quick look at the continuous diffusion models linked in the post shows lots of transformer models still
- p1esk 2mo agoIt’s the opposite: attention is necessary but not sufficient.
- slashdave 2mo agoJust about anyone building a large diffusion model is relying on transformers
- NickNaraghi 2mo agoI wonder if we’ll get something like CDLMs for automated harness engineering, sort of piloting the LLM underneath.
- ramon156 2mo agohow do tools like hermes do this? does it just review sessions and rewrite markdown files? also haven't read too deep into the deepseek agent harness but the math in there was really cool. it sounded promising, at least.
- pests 2mo ago> does it just review sessions and rewrite markdown files Yes, same with openclaw etc. Some might have plugins to integrate with graph or vector databases besides only markdown files.
- latentsea 2mo agoThere's a cool research project https://github.com/exoharness/exo https://github.com/exoharness/exo that is designed specifically as a self-modifying harness, so architecturally it separates things in a way that makes it a lot harder for the agent to break itself when modifying itself. It looks pretty neat. Saw the author do an interview on a podcast explaining it.
- 6stringmerc 2mo ago[dead]
- janalsncm 2mo ago> [in 2020/2021] the dominance of autoregression was not as well-established as it is today: GPT-3 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022 I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked. (The only exception I will make is encoder-decoder models which now are often done by decoder-only.) But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass. Before ChatGPT there really wasn’t much of a concept of pre-training and post-training. It was all pre-training. Post training was what made the bots conversational and not just “continuing the thing you wrote to them”. So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.
- echelon 2mo ago> GPT2 was considered too dangerous to release This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat. Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies. They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.
- janalsncm 2mo agoMy impression at the time was they were perhaps overly cautious but this was a bunch of researchers who wanted to self-regulate. Anthropic didn’t exist, deepmind was also much less product-focused and relatively cautious. Chinese models weren’t really a factor either. Government regulation was really not in the picture either in 2020 or 2021 tbh. The government was still trying to beat a pandemic. Some of the Biden admin eventually wanted to but it wasn’t very serious.
- alightsoul 2mo agowhy are chinese models a consideration at all?
- PrinceAliHad2P 2mo ago[dead]
- p1esk 2mo agoIt’s refreshing to read something not AI generated.
- benanne 2mo agoGlad to hear it! My stubbornness about this makes me feel like a luddite sometimes, but the additional effort required is probably still worth it, for the time being.
- ovin_dal 2mo agoDiffusion models for language felt inevitable. Imagine the creative potential once these mature beyond current limits.
- vatsachak 2mo agoI feel like there is still low hanging fruit on the auto regressive LLMs; the encoder
- Marchant_hq 2mo agoCDLMs sound promising for smoother, more coherent text generation. Excited to see how they tackle the token-level discontinuities.
- deleted 2mo ago[deleted]
- ainch 2mo agoA great read - as with all of Sander's diffusion posts.
- 2001zhaozhao 2mo agoI would love to see models that can think at different rates and also output a thinking scratchpad alongside output text instead of before all output. Right now models need to rely on less legible compressed CoT to get high intelligence per token/step, but with diffusion they would just need to output more tokens per step instead.
- deleted 2mo ago[deleted]
- nullbio 2mo agoConspiracy time: The continuous diffusion model has fruit. Frontier labs were trying to bury it with discrete diffusion distraction, because it is what they're using internally. Momentum wins, in the end.
- ACCount37 2mo agoNo conspiracy. Diffusion is just a more finicky, more expensive way of generating the same tokens as autoregressive decoding.
- LoganDark 2mo agoI love the idea of diffusion language models. I think they are potentially superior to autoregressive models, in the way that practically, considering whole systems tends to beat optimizing any one individual detail. To me, the way autoregressive models are sampled feels very fundamentally limited, and diffusion feels much more coherent by comparison.