29 ms·
Exactly this. At a high level, DALL-E is mapping text to a (continuous) matrix and then mapping that matrix to an image (another a matrix). All text inputs will
by axg11 4y ago
Exactly this. At a high level, DALL-E is mapping text to a (continuous) matrix and then mapping that matrix to an image (another a matrix). All text inputs will map to _something_. DALL-E doesn't care if that mapping makes sense, it has been trained to produce high-quality outputs, not to ensure the validity of mappings.
None of this makes DALL-E any less impressive to me. High quality image generation is a truly amazing result. Results from foundational models (GPT-3, PaLM, DALL-E, etc) are so impressive that they're forcing us to reconsider the nature of intelligence and raise the bar. That's a sign of a job well done to me.
- LoveMortuus 4y agoBut if it's just mapping text to image then it would be fair to assume that using the same text would result in the same image. But does that actually happen?
- Jweb_Guru 4y agoNo, it does not. It also doesn't always generate the same category of image. See https://twitter.com/realmeatyhuman/status/1531738616803864577 https://twitter.com/realmeatyhuman/status/153173861680386457.... As much as people would like there to be, there really does not seem to be anything here. The original author doesn't think so, either (would need to refind the tweet).
- Nition 4y agoI took the previous user's question as asking more like: Say there's an image in the training set with the description "Salvador Dali sitting at desk with pen and paper, 1957", and you put in the prompt "Salvador Dali sitting at desk with pen and paper, 1957", how close would the result be to that original training image?
- mirker 4y agoIt doesn’t have to be the same to achieve low loss. The date is not relevant, for example, for the image content. It can memorize a few images, but you can’t “compress” all of the internet’s images into text (if you can, make a startup with the most efficient compression algorithm ever made).
- tomerv 4y ago> it has been trained to produce high-quality outputs, not to ensure the validity of mappings. Just nitpicking here a bit. It has been trained to ensure the validity of mappings, but only for mappings of valid prompts, where "valid" is vaguely "things that appear in the training set". On the other hand, it wasn't trained to ensure the validity of mappings of invalid prompts to images. It's an open question what that would even mean - someone in another thread here suggested it should output "not a valid prompt" in this case.