3 ms·
It would be an interesting experiment to see where along the path some of these details are lost. For example, in the experiment where they ask for a pear cut
by Imnimo 4y ago
It would be an interesting experiment to see where along the path some of these details are lost. For example, in the experiment where they ask for a pear cut into 7 slices, and Dall-E often gives the wrong number. First check if you can train a classifier on CLIP embeddings of the caption to recover the specified number. If so, check if you can do the same for the image embedding the prior produces. Then check if you can do it for unCLIP's internal feature maps, and so on. Similar experiments could be conducted for other lost attributes like object relationships (on top, behind, etc.) or specific identities (Lincoln, Washington, etc.) .
- visarga 4y agoVery good points, but I would start with the obvious one - the contrastive objective of CLIP forces the prompt representation to a single embedding vector. So it acts as a bottleneck, limiting the number of object-attribute pairs it can simultaneously encode. And of course the solution is to use an autoregressive setup like Dalle-1, but then they can't have the same diffusion model decoder for image generation. I bet they consciously prioritised the artistic applications over exact semantics. The CLIP embedding space has nice properties, it's tempting to use it. From my experience semantic similarity based on embedding dot products is much easier to do than exact semantic matching. Three is similar to four and red similar to blue in embedding space.