Diffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilizatio

Diffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilization, when I'm running a diffusion model I'm getting 100%. This means that the only saving you're getting in speed for a diffusion model is being able to do more effective flops since the floats are smaller, e.g. instead of doing one 32bit multiplication, you're doing 8 4bit ones. By comparison for transformers you not only gain the flop increase, but also the improvement in memory shuffling that they do, e.g. it also takes you 8 times less time to load the memory into working memory from vram. The above is a vast over simplification and in practice will have more asterisks than you can shake a stick at.