Very good points, but I would start with the obvious one - the contrastive objective of CLIP forces the prompt representation to a single embedding vector. So i

Very good points, but I would start with the obvious one - the contrastive objective of CLIP forces the prompt representation to a single embedding vector. So it acts as a bottleneck, limiting the number of object-attribute pairs it can simultaneously encode. And of course the solution is to use an autoregressive setup like Dalle-1, but then they can't have the same diffusion model decoder for image generation. I bet they consciously prioritised the artistic applications over exact semantics. The CLIP embedding space has nice properties, it's tempting to use it. From my experience semantic similarity based on embedding dot products is much easier to do than exact semantic matching. Three is similar to four and red similar to blue in embedding space.