4 ms·
Diffusion on images is easy to understand for me: you start with noise, the model denoises by shifting the pixels towards their final value. What is the equival
by bondarchuk 2y ago
Diffusion on images is easy to understand for me: you start with noise, the model denoises by shifting the pixels towards their final value. What is the equivalent operation for increasing or reducing noise in language here? Is the "noisy" sentence half-way through training or inference sort-of-correct but not really, and at 90% almost-correct but with slightly wrong words (semantically)? Is the noise somehow semantic at all or is it something else?
- littlestymaar 2y agoNot familiar at all with diffusion LLM but I'd guess you'd have noisy logits.
- simne 2y agoFor LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
- itchyjunk 2y agoHow is that diffusion? What noise is it denoising there?
- simne 2y agoExample, as I understand: (I'm not sure how should look prompt, my guess): Prompt: answer, what word is missing in text query. Query: What is it denoising there?
- PartiallyTyped 2y agoDon't think about defusion as denoising, but rather as learning a delta operator, or rather, the inverse of one. I don't know how diffusion language models work precisely, but if I were to haphazard a guess, I would say you may think of a sentence as a matrix of values, the operator as simply filling a line with zeros, and the inverse — what the model learns — as adding it back. This is equivalent to cutting an image in blocks, and learning how to generate incrementally images by inpainting missing blocks. This in-painting mind you can be generated in multiple steps, where you incrementally add more into the block.
- deleted 2y ago[deleted]