3 ms·

What is a "text encoder" and why does a text-2-image net need it?

• Sharlin 3y ago
  A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip
• brucethemoose2 3y ago
  Its what "interprets" the prompt to give to the unet (which does the actual diffusion). SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
• famouswaffles 3y ago
  They've explained it but i'll point out that Dalle-3 doesn't use GPT-4 for the text encoder but the T5-XXL. The huge leap mostly comes from text descriptions that aren't trash.