4 ms·
> The AI does NOT build or use reference boards for specific prompts. The only reference it has is the prompt itself, which gets distilled down into a list of 5
by minimaxir 4y ago
> The AI does NOT build or use reference boards for specific prompts. The only reference it has is the prompt itself, which gets distilled down into a list of 512 numbers, each one of which the AI associates with a particular image feature (or set of features).
Minor technical clarification:
For SD 1.X, CLIPText encodes a prompt and passes a (77, 768) [edit: up to 77] matrix to the core UNet. For SD 2.X, OpenCLIP passes a (77, 1024) matrix.
The black-box interaction between the text-encoded matrices and the UNet is how Stable Diffusion "learns" text associations.
- kmeisthax 4y agoHuh. I thought those got projected into the (512) shared CLIP space before getting passed to the conditional blocks.
- minimaxir 4y agoOnly the text portion is included, otherwise that would be big. Also, I am slightly wrong in that the first dimension will many not always be 77, since apparently there is no padding in the tokenizer. Test notebook here: https://colab.research.google.com/drive/192PDIbc2XiI1HgJQSdNpbOlbZh49xygU?usp=sharing https://colab.research.google.com/drive/192PDIbc2XiI1HgJQSdN...
- HammadB 4y agoNope, this is why you cannot use images as prompts without some workarounds! SD doesn’t use the shared CLIP space but the text encoded before projection