3 ms·
Core takeaways: - major improvement is from detailed image captioning - they trained an image captioning model to produce short and detailed captions - they
by alievk 3y ago
Core takeaways:
- major improvement is from detailed image captioning
- they trained an image captioning model to produce short and detailed captions
- they use T5 text encoder
- they use GPT-4 to "upsample" short user prompts
- they train a U-net decoder and distill it to 2 denoising steps
- text rendering is still unreliable, they believe the model has hard time mapping word tokens into letters in image
- GaggiX 3y agoThey have only distilled the diffusion model used as a replacement for the VAE decoder, not the main diffusion model.
- alievk 3y agoSo the latent to pixel space conversion is made with another diffusion model? This is kind of refiner used in SDXL?
- GaggiX 3y ago>So the latent to pixel space conversion is made with another diffusion model? Yup >This is kind of refiner used in SDXL? The refiner only works on the latent.