3 ms·

It is kind of wild that most coding tasks are editing tasks, and we humans care a lot about code editing tools, but automated tools use code generation for edit

It is kind of wild that most coding tasks are editing tasks, and we humans care a lot about code editing tools, but automated tools use code generation for editing where a valid block must be generated top-to-bottom in one go. Fixing a mistake requires re-generating the file or block of code. Or, if something generated later has implications for earlier code--a new import or function parameter's required, something like that--the only option is to go back and re-generate a big chunk. That'd be inefficient for humans, not implausible it's wrong for other code generators too. I don't know if diffusion specifically will be the approach. (Maybe there's something to generating edit sequences?) This post's note that diffusion kills KV caching is something I hadn't even considered. It does seem right to experiment with things other than strict start-to-end generation.

imtringued 1y ago
A massive problem with current generation llms is that they have a single globally ordered context and that the model is only allowed to append to the context. This is like having a single tape Turing machine. They can simulate a multi tape machine, but at O(n^2) complexity. The computation budget of an LLM is finite, so this has a massive practical impact.
- cubefox 1y ago
  The article explains that this is not a problem but an advantage.
  - Sabinus 1y ago
    Apart from execution speedup due to caching, how does the article explain this is an advantage?
  - mattnewton 1y ago
    I wouldn't agree it's an advantage per se, just that the caching workaround works better than you would expect! It's still quite slow to generate each token at a time versus diffusing batches, it's just hard to cache all the work you do in each encoder step.
namibj 1y ago
You can still cache prompts; this just affects the cache for during generation produced tokens. And that's fairly harmless relatively speaking.
- yorwba 1y ago
  If you completely do away with autoregression, prompt tokens can pay attention to generated tokens, so even the prompt tokens' KV vectors change at every step and you cannot cache anything. For this reason, models that generate text using diffusion typically generate blocks of tokens at a time, where tokens within a block freely attend to each other, but across blocks there's causal masking so that each block only depends on the preceding ones and we're back to autoregression again. That makes caching possible, but also means you still can't have diffusion change the beginning of a long text to match the end.
  - namibj 1y ago
    I specifically mean prompts here, and I don't mean they'd have casual attention. Just run an encoder to get your KV cache pre-filling of the prompt, then do non-causal diffusion generation of the response referencing the cached prompt without re-encoding the prompt. You don't need to revert to chunks to get to enjoy prompt caching, especially if you use it in a RAG type way with minor provisions to allow KV caching the RAG fragments (a bunch of work has been done on that, iirc even DeepSeekV3 would allow that).

imtringued 1y ago

A massive problem with current generation llms is that they have a single globally ordered context and that the model is only allowed to append to the context. This is like having a single tape Turing machine. They can simulate a multi tape machine, but at O(n^2) complexity. The computation budget of an LLM is finite, so this has a massive practical impact.

cubefox 1y ago
The article explains that this is not a problem but an advantage.
- Sabinus 1y ago
  Apart from execution speedup due to caching, how does the article explain this is an advantage?
- mattnewton 1y ago
  I wouldn't agree it's an advantage per se, just that the caching workaround works better than you would expect! It's still quite slow to generate each token at a time versus diffusing batches, it's just hard to cache all the work you do in each encoder step.

namibj 1y ago

You can still cache prompts; this just affects the cache for during generation produced tokens. And that's fairly harmless relatively speaking.

yorwba 1y ago
If you completely do away with autoregression, prompt tokens can pay attention to generated tokens, so even the prompt tokens' KV vectors change at every step and you cannot cache anything. For this reason, models that generate text using diffusion typically generate blocks of tokens at a time, where tokens within a block freely attend to each other, but across blocks there's causal masking so that each block only depends on the preceding ones and we're back to autoregression again. That makes caching possible, but also means you still can't have diffusion change the beginning of a long text to match the end.
- namibj 1y ago
  I specifically mean prompts here, and I don't mean they'd have casual attention. Just run an encoder to get your KV cache pre-filling of the prompt, then do non-causal diffusion generation of the response referencing the cached prompt without re-encoding the prompt. You don't need to revert to chunks to get to enjoy prompt caching, especially if you use it in a RAG type way with minor provisions to allow KV caching the RAG fragments (a bunch of work has been done on that, iirc even DeepSeekV3 would allow that).