4 ms·
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
by simne 2y ago
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
- itchyjunk 2y agoHow is that diffusion? What noise is it denoising there?
- simne 2y agoExample, as I understand: (I'm not sure how should look prompt, my guess): Prompt: answer, what word is missing in text query. Query: What is it denoising there?
- PartiallyTyped 2y agoDon't think about defusion as denoising, but rather as learning a delta operator, or rather, the inverse of one. I don't know how diffusion language models work precisely, but if I were to haphazard a guess, I would say you may think of a sentence as a matrix of values, the operator as simply filling a line with zeros, and the inverse — what the model learns — as adding it back. This is equivalent to cutting an image in blocks, and learning how to generate incrementally images by inpainting missing blocks. This in-painting mind you can be generated in multiple steps, where you incrementally add more into the block.
- deleted 2y ago[deleted]