46 ms·

Bing's DALL-E 3 is bleeding edge and a huge leap forward in prompt comprehension. Explained by the fact that it uses GPT-4 as the text encoder, so… similar perf

Bing's DALL-E 3 is bleeding edge and a huge leap forward in prompt comprehension. Explained by the fact that it uses GPT-4 as the text encoder, so… similar performance probably not available on your local SD instance in the near future.

TekMol 3y ago
What is a "text encoder" and why does a text-2-image net need it?
- Sharlin 3y ago
  A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip
- brucethemoose2 3y ago
  Its what "interprets" the prompt to give to the unet (which does the actual diffusion). SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
- famouswaffles 3y ago
  They've explained it but i'll point out that Dalle-3 doesn't use GPT-4 for the text encoder but the T5-XXL. The huge leap mostly comes from text descriptions that aren't trash.
famouswaffles 3y ago
DALL-E 3 uses T5-XXL for the text encoder not GPT-4. The huge leap comes mostly from using text descriptions that aren't trash. https://cdn.openai.com/papers/dall-e-3.pdf https://cdn.openai.com/papers/dall-e-3.pdf
- Sharlin 3y ago
  Hmm, that paper is interesting, and definitely descriptions are a big bottleneck in training. But they only say they used T5-XXL in the experiments related to the paper. Who knows what Bing is actually running?

TekMol 3y ago

What is a "text encoder" and why does a text-2-image net need it?

Sharlin 3y ago
A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip
brucethemoose2 3y ago
Its what "interprets" the prompt to give to the unet (which does the actual diffusion). SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
famouswaffles 3y ago
They've explained it but i'll point out that Dalle-3 doesn't use GPT-4 for the text encoder but the T5-XXL. The huge leap mostly comes from text descriptions that aren't trash.

famouswaffles 3y ago

DALL-E 3 uses T5-XXL for the text encoder not GPT-4. The huge leap comes mostly from using text descriptions that aren't trash. https://cdn.openai.com/papers/dall-e-3.pdf https://cdn.openai.com/papers/dall-e-3.pdf

Sharlin 3y ago
Hmm, that paper is interesting, and definitely descriptions are a big bottleneck in training. But they only say they used T5-XXL in the experiments related to the paper. Who knows what Bing is actually running?