3 ms·
Could you expound on this? From what I'm reading, this sounds like an issue with diffusion models that their block diffusion model is purposefully designed to m
by prophesi 2y ago
Could you expound on this? From what I'm reading, this sounds like an issue with diffusion models that their block diffusion model is purposefully designed to mitigate, by conditioning on previous blocks and allowing for larger blocks if that conditioning still doesn't help maintain coherence.
- 85392_school 2y agoIt's an issue that you run into as long as you're forced to start with a yes/no answer. It's a problem forward-only LLMs have and diffusion models don't, and normal block diffusion is closer to forward LLMs than diffusion models. You could increase the block size to act more like a full diffusion model, but you would lose some of the benefits of block diffusion.
- throwaway314155 2y agoInteresting. Makes me want to play around with an open diffusion LM. Do you have any recommendations?