2 ms·
Its training data might not have negations in it - I'm not sure anyone has tried to explain what CLIP understands of text. Besides that, we know it doesn't bec
by astrange 5y ago
Its training data might not have negations in it - I'm not sure anyone has tried to explain what CLIP understands of text.
Besides that, we know it doesn't because models don't recurse because no matter how big they are, they still have constant execution times. And we gave up on systematically parsing text in favor of just vibing anyway (technical term).
That's just as well since most parsers didn't really work in practice; I think I vaguely remember one as kind of working (https://www.link.cs.cmu.edu/link/ https://www.link.cs.cmu.edu/link/) but it can't handle bad grammar or misspellings like ML can.
- visarga 5y agoThat's not true, the autoregressive LMs (GPTs) can generate variable length outputs allowing for step by step computation. Recent papers get them to generate the full chain of thought reasoning process to the solution (also called rationales in some papers), improving accuracy. But this version of Dall-E is not autoregressive, they've gone for diffusion models in the generator and for a single vector embedding (no internal spatial grid structure) for the encoder. That explains why it's bad at negations, counting and stacking coloured blocks. I believe these skills were not the top priority of OpenAI this time, they understandably went for pretty images.
- astrange 5y agoI agree GPT as a whole generates variable length outputs, but the model isn't generating them in a single pass, is it? I thought there was some sampling process implemented in traditional code outside the model that made it do that, and it wasn't learned. Have forgotten the details. If Dall-E took variable length sequences as inputs, its execution time would still be O(N) on the length of the sequence and it would still have a fixed amount of memory to work with. Whereas a computer program of length N can do pretty much anything, and a human given a description can go off and think about it.