4 ms·

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scali

by tux3 6y ago

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text that has all the signs and form of good prose, but the output repeatedly violates physics and common sense in funny ways. I know just enough about machine learning to have dangerously unrealistic expectations, but I'd like if I could reasonnably hope to see signs of a shared representation or shared knowledge between say image labeling and language modeling. This looks like a very concrete data point to take if you care about generality. Maybe then we can seriously talk about world-modeling.

Udik 6y ago
> I'd be surprised if this architecture scales to larger resolutions Could a different NN be used for upscaling?
- the8472 6y ago
  Upscaling NNs already exist, but that's beside the point. This is a demonstration that no domain-specific knowledge needs to be encoded in your architecture. Manually wiring different NNs together to split a task into separate subtasks would be the opposite. Far more interesting would be general training optimizations that could train GPT faster on any kind of task.
unwoundmouse 6y ago
This is the question I want answered. I feel that in the human brain, many words evoke imagery. For instance, when I hear the word bleak, I think of a grey setting
gwern 6y ago
Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels. However, if you were starting this research today, you'd use any of half-a-dozen different self-attention variants which are roughly linear, including OA's own Sparse Transformers (which they did use to generate images, just on a far smaller scale which wouldn't be adequate to show competitive performance with SimCLR etc). With those, it's perfectly possible to do self-attention over whole images at 256px or higher. (As a matter of fact, Aydao has been working on a StyleGAN which just uses self-attention every layer instead of convolutions, using one of the new attentions; you can see some generated image samples from Flowers here: https://github.com/tensorfork/tensorfork/issues/31 https://github.com/tensorfork/tensorfork/issues/31 GAN loss, not autoregressive pixel likelihood, but it makes the point.)
- gwern 6y ago
  * lucidrains, not Aydao
visarga 6y ago
It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That would be a bag of visual tokens. Then continue the sequence with text tokens and train it to solve a reasoning task. Another example: Unicoder-VL https://arxiv.org/pdf/1908.06066.pdf https://arxiv.org/pdf/1908.06066.pdf

by tux3 6y ago

Udik 6y ago

> I'd be surprised if this architecture scales to larger resolutions Could a different NN be used for upscaling?

the8472 6y ago
Upscaling NNs already exist, but that's beside the point. This is a demonstration that no domain-specific knowledge needs to be encoded in your architecture. Manually wiring different NNs together to split a task into separate subtasks would be the opposite. Far more interesting would be general training optimizations that could train GPT faster on any kind of task.

unwoundmouse 6y ago

This is the question I want answered. I feel that in the human brain, many words evoke imagery. For instance, when I hear the word bleak, I think of a grey setting

gwern 6y ago

Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels. However, if you were starting this research today, you'd use any of half-a-dozen different self-attention variants which are roughly linear, including OA's own Sparse Transformers (which they did use to generate images, just on a far smaller scale which wouldn't be adequate to show competitive performance with SimCLR etc). With those, it's perfectly possible to do self-attention over whole images at 256px or higher. (As a matter of fact, Aydao has been working on a StyleGAN which just uses self-attention every layer instead of convolutions, using one of the new attentions; you can see some generated image samples from Flowers here: https://github.com/tensorfork/tensorfork/issues/31 https://github.com/tensorfork/tensorfork/issues/31 GAN loss, not autoregressive pixel likelihood, but it makes the point.)

gwern 6y ago
* lucidrains, not Aydao

visarga 6y ago

It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That would be a bag of visual tokens. Then continue the sequence with text tokens and train it to solve a reasoning task. Another example: Unicoder-VL https://arxiv.org/pdf/1908.06066.pdf https://arxiv.org/pdf/1908.06066.pdf