4 ms·

> vision, audio, proprioception and more... what will happen when these modalities are incorporated First of all, there's no evidence that a token-based model,

by brabel 5y ago

> vision, audio, proprioception and more... what will happen when these modalities are incorporated First of all, there's no evidence that a token-based model, as you yourself call it, can be made to work on visual or sound data where you have more than one dimension to the data. Second, even if you manage to make it work, it needs to be able to relate the different senses somehow, and no training data is going to be able to give it information about how things relate, it will have to use heuristics to determine that and I can only see that failing spetacularly because much of the relations between what someone sees and how someone acts can take very large amounts of time or never materialize at all (e.g. studying maths VS actually using it to solve a real-world problem).

Jack000 5y ago
hm.. have you seen DALL-E/CLIP? It's literally a language model applied to images and text. It's also the dumbest possible way to generate images (generating tokens one at a time, down and to the right) yet it works.
jeeceebees 5y ago
There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet. If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities? It's definitely a challenge to get enough high-quality data to feed a 100B parameter version of such a mutli-modal model, but there don't seem to be any theoretically insurmountable issues towards this "dumb" way of giving the models more intuition. [1] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text, https://arxiv.org/abs/2104.11178 https://arxiv.org/abs/2104.11178 [2] Perceiver IO: A General Architecture for Structured Inputs & Outputs, https://arxiv.org/abs/2107.14795 https://arxiv.org/abs/2107.14795
dunefox 5y ago
The 'general perceiver' is such a model that is somewhat distanced from the modality of its input: https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206

by brabel 5y ago

Jack000 5y ago

hm.. have you seen DALL-E/CLIP? It's literally a language model applied to images and text. It's also the dumbest possible way to generate images (generating tokens one at a time, down and to the right) yet it works.

jeeceebees 5y ago

There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet. If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities? It's definitely a challenge to get enough high-quality data to feed a 100B parameter version of such a mutli-modal model, but there don't seem to be any theoretically insurmountable issues towards this "dumb" way of giving the models more intuition. [1] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text, https://arxiv.org/abs/2104.11178 https://arxiv.org/abs/2104.11178 [2] Perceiver IO: A General Architecture for Structured Inputs & Outputs, https://arxiv.org/abs/2107.14795 https://arxiv.org/abs/2107.14795

dunefox 5y ago

The 'general perceiver' is such a model that is somewhat distanced from the modality of its input: https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206