3 ms·
Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds
by famouswaffles 3y ago
Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing.
Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings.
as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision.
and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929
It basically just comes down to lots of layers and training
- Solvency 3y agoWhy would you feed pixels into transformers? Surely you could translate the overall image into frequencies using Fourier transforms and then into embeddings which feed the LLM?
- birdiesanders 3y agoIt's essentially a process if denoising a spectrogram, and as such the data is processed that way.
- mliker 3y agoThe original commenter is trying to build intuition so my statement was simply to help the commenter understand that Transformers can operate on patches. As for > after all, the original authors didn't anticipate the use case in Vision. To quote the original "Attention is all you need" paper: "We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video" To say that the original authors did not anticipate the Transformers use in Vision is false.
- famouswaffles 3y agoI'm talking about using transnformers on their own, beating SOTA CNNs. People thought you needed CNNs even with transformers...until that was shown wrong. The point is that there isn't any special intuition that makes this fit vision.