3 ms·

Surprised (as a non-ML-practitioner) that a uniform tiling is used to ‘tokenize’ the image in this approach. After all, language transformers don’t operate by c

Surprised (as a non-ML-practitioner) that a uniform tiling is used to ‘tokenize’ the image in this approach. After all, language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components) Wouldn’t it make sense to have a process that selects regions of the image to focus in on, perhaps itself guided by some sort of attention model? So for a face it might produce tiles covering the whole face, the eyes, the mouth, whatever other details it finds helpful… all obviously accompanied by a vector representing the tile’s spatial relationship to the whole image. Or is the assumption that this sort of thing is effectively done within the model’s layers as it combines the input from multiple tiles?

espadrine 4y ago
> language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components) Don’t they? Most use BPE, which yields arbitrary but common sequence of characters that often are just syllables. Additionally, many use SentencePiece, which randomizes the subdivisions, which sometimes produces single characters instead. (That said, what you suggest would be an interesting research avenue.)
- jameshart 4y ago
  BPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it uses an existing semantic model for ‘what normal text looks like’ to choose potentially useful tokens (even for nonsensical text, which is why it works on stuff like code) And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.) Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…
- fxtentacle 4y ago
  In this paper https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech). In my case, the entropy-optimized tokenization also matched conserved features of the German language.
rolisz 4y ago
Byte pair encoding (used in GPT2 models) is not semantic, it's purely based on statistics of occurrences of characters groups. It's not supposed to be semantic, it's just supposed to enable a complete coverage of the whole text. Similarly, the image patches have to cover the whole image and then the transformer can learn to extract the important parts.
- jameshart 4y ago
  Image patch extraction could be statistical rather than semantic. Like if the top part of the image is just all blue sky with no high frequency detail, you don’t necessarily need to feed the model dozens of blue tiles when you could just pass one or two big blue tiles. If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well. Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not. But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.
  - rolisz 4y ago
    I think that is not done because it's hard/unclear how to handle patches that are not the same size. In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).
    - knodi123 4y ago
      > I think that is not done because it's hard/unclear how to handle patches that are not the same size. Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?
      - rolisz 4y ago
        I think that is one of the benefits of having an easily parallelizable model.
deleted 4y ago
[deleted]
satvikpendem 4y ago
Reminds me of using quadtrees for image compression. The lower detail parts would be covered by a large single color-value block while more detailed parts would use smaller and smaller blocks. https://github.com/alpharaoh/Quadtree-Compression https://github.com/alpharaoh/Quadtree-Compression
fxtentacle 4y ago
We tried doing just that and the short answer is that we couldn't fit the gradients for arbitratily-placed tiles into a DGX with 320 GB of GPU RAM. For 32x32 pixel tiles, free location placement will 1024x your memory usage.
- bl0rg 4y ago
  Perhaps hard attention could be a thing here. Possibly too tricky to train though.
astrange 4y ago
It seems to work okay without this, but an approach with less than totally free positioning would be to send in one more set of overlapping tiles shifted by half a tile size. Or better yet, image pyramids, which would help you handle different image sizes. That’s significantly more data either way though, which is probably the issue.

espadrine 4y ago

> language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components) Don’t they? Most use BPE, which yields arbitrary but common sequence of characters that often are just syllables. Additionally, many use SentencePiece, which randomizes the subdivisions, which sometimes produces single characters instead. (That said, what you suggest would be an interesting research avenue.)

jameshart 4y ago
BPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it uses an existing semantic model for ‘what normal text looks like’ to choose potentially useful tokens (even for nonsensical text, which is why it works on stuff like code) And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.) Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…
fxtentacle 4y ago
In this paper https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech). In my case, the entropy-optimized tokenization also matched conserved features of the German language.

rolisz 4y ago

Byte pair encoding (used in GPT2 models) is not semantic, it's purely based on statistics of occurrences of characters groups. It's not supposed to be semantic, it's just supposed to enable a complete coverage of the whole text. Similarly, the image patches have to cover the whole image and then the transformer can learn to extract the important parts.

jameshart 4y ago
Image patch extraction could be statistical rather than semantic. Like if the top part of the image is just all blue sky with no high frequency detail, you don’t necessarily need to feed the model dozens of blue tiles when you could just pass one or two big blue tiles. If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well. Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not. But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.
- rolisz 4y ago
  I think that is not done because it's hard/unclear how to handle patches that are not the same size. In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).
  - knodi123 4y ago
    > I think that is not done because it's hard/unclear how to handle patches that are not the same size. Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?
    - rolisz 4y ago
      I think that is one of the benefits of having an easily parallelizable model.

deleted 4y ago

[deleted]

satvikpendem 4y ago

Reminds me of using quadtrees for image compression. The lower detail parts would be covered by a large single color-value block while more detailed parts would use smaller and smaller blocks. https://github.com/alpharaoh/Quadtree-Compression https://github.com/alpharaoh/Quadtree-Compression

fxtentacle 4y ago

We tried doing just that and the short answer is that we couldn't fit the gradients for arbitratily-placed tiles into a DGX with 320 GB of GPU RAM. For 32x32 pixel tiles, free location placement will 1024x your memory usage.

bl0rg 4y ago
Perhaps hard attention could be a thing here. Possibly too tricky to train though.

astrange 4y ago

It seems to work okay without this, but an approach with less than totally free positioning would be to send in one more set of overlapping tiles shifted by half a tile size. Or better yet, image pyramids, which would help you handle different image sizes. That’s significantly more data either way though, which is probably the issue.