4 ms·

I think it's a mistake to judge the performance of GPT-3 like this because it's not comparing apples to apples. What GPT-3 is doing is equivalent to me giving y

I think it's a mistake to judge the performance of GPT-3 like this because it's not comparing apples to apples. What GPT-3 is doing is equivalent to me giving you a book in Chinese (without any further context, translation or grounding in the physical world) and then expecting you to produce articles in Chinese. This is plainly impossible for any human and it's amazing that it can do this at all. A human generates language while having knowledge of entire other modalities that GPT-3 lacks - vision, audio, proprioception and more. No, of course GPT-3 doesn't know what it's talking about, it's only ever seen tokens! What you should be asking is, what will happen when these modalities are incorporated, when it does know what it's talking about. Personally I suspect a massive number of white collar workers that mostly work with text are about to become obsolete.

brabel 5y ago
> vision, audio, proprioception and more... what will happen when these modalities are incorporated First of all, there's no evidence that a token-based model, as you yourself call it, can be made to work on visual or sound data where you have more than one dimension to the data. Second, even if you manage to make it work, it needs to be able to relate the different senses somehow, and no training data is going to be able to give it information about how things relate, it will have to use heuristics to determine that and I can only see that failing spetacularly because much of the relations between what someone sees and how someone acts can take very large amounts of time or never materialize at all (e.g. studying maths VS actually using it to solve a real-world problem).
- Jack000 5y ago
  hm.. have you seen DALL-E/CLIP? It's literally a language model applied to images and text. It's also the dumbest possible way to generate images (generating tokens one at a time, down and to the right) yet it works.
- jeeceebees 5y ago
  There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet. If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities? It's definitely a challenge to get enough high-quality data to feed a 100B parameter version of such a mutli-modal model, but there don't seem to be any theoretically insurmountable issues towards this "dumb" way of giving the models more intuition. [1] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text, https://arxiv.org/abs/2104.11178 https://arxiv.org/abs/2104.11178 [2] Perceiver IO: A General Architecture for Structured Inputs & Outputs, https://arxiv.org/abs/2107.14795 https://arxiv.org/abs/2107.14795
- dunefox 5y ago
  The 'general perceiver' is such a model that is somewhat distanced from the modality of its input: https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206
denton-scratch 5y ago
> not comparing apples to apples. Comparing apples to iPods? (sorry)
TomSwirly 5y ago
> What GPT-3 is doing is equivalent to me giving you a book in Chinese (without any further context, translation or grounding in the physical world) and then expecting you to produce articles in Chinese. This is plainly impossible for any human and it's amazing that it can do this at all. It is generating readable texts, but not _meaningful_ texts in Chinese. It's a fancy madlib from the existing corpus.

brabel 5y ago

> vision, audio, proprioception and more... what will happen when these modalities are incorporated First of all, there's no evidence that a token-based model, as you yourself call it, can be made to work on visual or sound data where you have more than one dimension to the data. Second, even if you manage to make it work, it needs to be able to relate the different senses somehow, and no training data is going to be able to give it information about how things relate, it will have to use heuristics to determine that and I can only see that failing spetacularly because much of the relations between what someone sees and how someone acts can take very large amounts of time or never materialize at all (e.g. studying maths VS actually using it to solve a real-world problem).

Jack000 5y ago
hm.. have you seen DALL-E/CLIP? It's literally a language model applied to images and text. It's also the dumbest possible way to generate images (generating tokens one at a time, down and to the right) yet it works.
jeeceebees 5y ago
There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet. If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities? It's definitely a challenge to get enough high-quality data to feed a 100B parameter version of such a mutli-modal model, but there don't seem to be any theoretically insurmountable issues towards this "dumb" way of giving the models more intuition. [1] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text, https://arxiv.org/abs/2104.11178 https://arxiv.org/abs/2104.11178 [2] Perceiver IO: A General Architecture for Structured Inputs & Outputs, https://arxiv.org/abs/2107.14795 https://arxiv.org/abs/2107.14795
dunefox 5y ago
The 'general perceiver' is such a model that is somewhat distanced from the modality of its input: https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206

denton-scratch 5y ago

> not comparing apples to apples. Comparing apples to iPods? (sorry)

TomSwirly 5y ago

> What GPT-3 is doing is equivalent to me giving you a book in Chinese (without any further context, translation or grounding in the physical world) and then expecting you to produce articles in Chinese. This is plainly impossible for any human and it's amazing that it can do this at all. It is generating readable texts, but not _meaningful_ texts in Chinese. It's a fancy madlib from the existing corpus.