3 ms·
> What it needs is an equivalent map for the concepts behind the signposts, It's the CLIP text and image encoders of DALL-E 2. > and also a mapping between co
by visarga 4y ago
> What it needs is an equivalent map for the concepts behind the signposts,
It's the CLIP text and image encoders of DALL-E 2.
> and also a mapping between concepts and images.
It's the "prior" module.
You guessed right but DALL-E 2 already has these functionalities in its design.
https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04/DALL-E-2-architecture.jpg https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2022/04...
- astrange 4y agoIt seems like it'd be easier to use if you could see the intermediate steps, i.e. what the CLIP encoder "thought" the text prompt meant, and how it "thought" to turn that into an image prior. That could be done by decoding both into text again. Also, if you could feed it some kind of "knowledge graph" of words and images rather than a sentence, you wouldn't be misunderstood the same way, but you'd presumably run into the limit of what its embeddings can contain.