4 ms·
Wow, really cool concept! I wonder if this starts to become similar dynamics to what we see in image generation models, where structure/detail emerges in one re
by szvsw 2y ago
Wow, really cool concept! I wonder if this starts to become similar dynamics to what we see in image generation models, where structure/detail emerges in one region of the image and then the surrounding areas start to resolve themselves into place. That kind of behavior seems particularly useful for longer reasoning/logic/planning, where the big ideas might become apparent first, and then the interstitial details and text just naturally fill in…
- byteknight 2y agoThe process you describe is referred to as diffusion
- szvsw 2y agoYep yep I know, but I was trying to suggest something diffusion-like occurring with a language model through a totally separate mechanism that does not rely on the denoising process (at least not literally).
- byteknight 2y agoIt kinda does that but one token at a time.
- immibis 2y agoI'm fairly certain diffusion refers to the overall architecture, not the emergent self-organization process.
- byteknight 2y agoIt refers to the process of taking noise and "diffusing" it until visually appealing