3 ms·
They split the image into small patches (I think 16x16 is standard), and then treat each patch as a token. The image becomes a sequence of tokens, which gets a
by QuesnayJr 3y ago
They split the image into small patches (I think 16x16 is standard), and then treat each patch as a token. The image becomes a sequence of tokens, which gets analyzed the same as if it was text.
Doing it this way is obviously throwing away a lot of information, but I assume the advantages of using a transfomer outweigh the disadvantages.
- semiquaver 3y agoThis is a very useful answer, thanks. So is every possible 16x16 grid of pixels a legal token value and they are used more or less verbatim or are they encoded in some other way? Prompted by some of the other replies I’ve read up a bit on positional embeddings. That they are used even with text transformers which otherwise lack the linearity of RNNs etc helps tremendously to clarify things.