4 ms·

I feel like I have a schematic understanding of how transformer models process text. But I struggle to understand how the same concept could be used with images

I feel like I have a schematic understanding of how transformer models process text. But I struggle to understand how the same concept could be used with images or other less linear data types. I understand that such data can be vectorized so it’s represented as a point or series of points in a higher dimensional space, but how does such a process capture the essential high level perceptual aspects well enough to be able to replicate large scale features? Or equivalently, how are the dimensions chosen? I have to imagine that simply looking at an image as an RGB pixel sequence would largely miss the point of what it “is” and represents.

sangnoir 3y ago
> how does such a process capture the essential high level perceptual aspects well enough to be able to replicate large scale features Layers and lots of training. Upper layers can capture/recognize large scale features
mliker 3y ago
If I understand what you're asking, the Transformer isn't initially treating the image as a sequence of pixels like p1, p2, ..., pN. Instead, you can use a convolutional neural network to respect the structure of the image to extract features. Then you use the attention mechanism to pay attention to parts of the image that aren't necessarily close together but that when viewed together, contribute to the classification of an object within the image.
- famouswaffles 3y ago
  Vision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly because doing that gets very expensive very fast. The image is split into patches and the patches have positional embeddings.
famouswaffles 3y ago
Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings. as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision. and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 It basically just comes down to lots of layers and training
- Solvency 3y ago
  Why would you feed pixels into transformers? Surely you could translate the overall image into frequencies using Fourier transforms and then into embeddings which feed the LLM?
  - birdiesanders 3y ago
    It's essentially a process if denoising a spectrogram, and as such the data is processed that way.
- mliker 3y ago
  The original commenter is trying to build intuition so my statement was simply to help the commenter understand that Transformers can operate on patches. As for > after all, the original authors didn't anticipate the use case in Vision. To quote the original "Attention is all you need" paper: "We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video" To say that the original authors did not anticipate the Transformers use in Vision is false.
QuesnayJr 3y ago
They split the image into small patches (I think 16x16 is standard), and then treat each patch as a token. The image becomes a sequence of tokens, which gets analyzed the same as if it was text. Doing it this way is obviously throwing away a lot of information, but I assume the advantages of using a transfomer outweigh the disadvantages.
- semiquaver 3y ago
  This is a very useful answer, thanks. So is every possible 16x16 grid of pixels a legal token value and they are used more or less verbatim or are they encoded in some other way? Prompted by some of the other replies I’ve read up a bit on positional embeddings. That they are used even with text transformers which otherwise lack the linearity of RNNs etc helps tremendously to clarify things.

sangnoir 3y ago

> how does such a process capture the essential high level perceptual aspects well enough to be able to replicate large scale features Layers and lots of training. Upper layers can capture/recognize large scale features

mliker 3y ago

If I understand what you're asking, the Transformer isn't initially treating the image as a sequence of pixels like p1, p2, ..., pN. Instead, you can use a convolutional neural network to respect the structure of the image to extract features. Then you use the attention mechanism to pay attention to parts of the image that aren't necessarily close together but that when viewed together, contribute to the classification of an object within the image.

famouswaffles 3y ago
Vision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly because doing that gets very expensive very fast. The image is split into patches and the patches have positional embeddings.

famouswaffles 3y ago

Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings. as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision. and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 It basically just comes down to lots of layers and training

Solvency 3y ago
Why would you feed pixels into transformers? Surely you could translate the overall image into frequencies using Fourier transforms and then into embeddings which feed the LLM?
- birdiesanders 3y ago
  It's essentially a process if denoising a spectrogram, and as such the data is processed that way.
mliker 3y ago
The original commenter is trying to build intuition so my statement was simply to help the commenter understand that Transformers can operate on patches. As for > after all, the original authors didn't anticipate the use case in Vision. To quote the original "Attention is all you need" paper: "We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video" To say that the original authors did not anticipate the Transformers use in Vision is false.

QuesnayJr 3y ago

They split the image into small patches (I think 16x16 is standard), and then treat each patch as a token. The image becomes a sequence of tokens, which gets analyzed the same as if it was text. Doing it this way is obviously throwing away a lot of information, but I assume the advantages of using a transfomer outweigh the disadvantages.

semiquaver 3y ago
This is a very useful answer, thanks. So is every possible 16x16 grid of pixels a legal token value and they are used more or less verbatim or are they encoded in some other way? Prompted by some of the other replies I’ve read up a bit on positional embeddings. That they are used even with text transformers which otherwise lack the linearity of RNNs etc helps tremendously to clarify things.