A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent

A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip