3 ms·

Byte pair encoding (used in GPT2 models) is not semantic, it's purely based on statistics of occurrences of characters groups. It's not supposed to be semantic,

by rolisz 4y ago

Byte pair encoding (used in GPT2 models) is not semantic, it's purely based on statistics of occurrences of characters groups. It's not supposed to be semantic, it's just supposed to enable a complete coverage of the whole text. Similarly, the image patches have to cover the whole image and then the transformer can learn to extract the important parts.

jameshart 4y ago
Image patch extraction could be statistical rather than semantic. Like if the top part of the image is just all blue sky with no high frequency detail, you don’t necessarily need to feed the model dozens of blue tiles when you could just pass one or two big blue tiles. If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well. Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not. But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.
- rolisz 4y ago
  I think that is not done because it's hard/unclear how to handle patches that are not the same size. In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).
  - knodi123 4y ago
    > I think that is not done because it's hard/unclear how to handle patches that are not the same size. Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?
    - rolisz 4y ago
      I think that is one of the benefits of having an easily parallelizable model.
  - jameshart 4y ago
    The tiles would all be the same size - they’d just be sampled from the image at different resolutions.
    - rolisz 4y ago
      Then you introduce a resizing component, which resizes differently for different parts of the image. And you still have to decide how to select the tile sizes: if it's heuristics, well, the field of ML has been moving away from that. If it's with another learnable component, maybe it's worth a shot, at the cost of a more complicated model, which probably has two stages, so the inference time is worse (and maybe even train time)

by rolisz 4y ago

jameshart 4y ago

Image patch extraction could be statistical rather than semantic. Like if the top part of the image is just all blue sky with no high frequency detail, you don’t necessarily need to feed the model dozens of blue tiles when you could just pass one or two big blue tiles. If there’s a small plane in part of the sky, an overlapping much smaller tile with that object in it could be passed in as well. Really just a matter of choosing tile sizes based on how fine the detail is in that part of the image - which is sort of similar to my mental image for how BPE decides whether to cluster characters into one token or not. But as I say, not a practitioner, and I kind of get the impression that ‘the transformer attention model just solves for all that’.

rolisz 4y ago
I think that is not done because it's hard/unclear how to handle patches that are not the same size. In text it doesn't matter because all tokens are mapped to an integer (eg. the -> 54, hacker-> 1356).
- knodi123 4y ago
  > I think that is not done because it's hard/unclear how to handle patches that are not the same size. Is that because of all the effort put into running the same algorithm in parallel on a ton of tiles and then piecing together the results?
  - rolisz 4y ago
    I think that is one of the benefits of having an easily parallelizable model.
- jameshart 4y ago
  The tiles would all be the same size - they’d just be sampled from the image at different resolutions.
  - rolisz 4y ago
    Then you introduce a resizing component, which resizes differently for different parts of the image. And you still have to decide how to select the tile sizes: if it's heuristics, well, the field of ML has been moving away from that. If it's with another learnable component, maybe it's worth a shot, at the cost of a more complicated model, which probably has two stages, so the inference time is worse (and maybe even train time)