3 ms·
That's not true, the autoregressive LMs (GPTs) can generate variable length outputs allowing for step by step computation. Recent papers get them to generate th
by visarga 5y ago
That's not true, the autoregressive LMs (GPTs) can generate variable length outputs allowing for step by step computation. Recent papers get them to generate the full chain of thought reasoning process to the solution (also called rationales in some papers), improving accuracy.
But this version of Dall-E is not autoregressive, they've gone for diffusion models in the generator and for a single vector embedding (no internal spatial grid structure) for the encoder. That explains why it's bad at negations, counting and stacking coloured blocks. I believe these skills were not the top priority of OpenAI this time, they understandably went for pretty images.
- astrange 5y agoI agree GPT as a whole generates variable length outputs, but the model isn't generating them in a single pass, is it? I thought there was some sampling process implemented in traditional code outside the model that made it do that, and it wasn't learned. Have forgotten the details. If Dall-E took variable length sequences as inputs, its execution time would still be O(N) on the length of the sequence and it would still have a fixed amount of memory to work with. Whereas a computer program of length N can do pretty much anything, and a human given a description can go off and think about it.