3 ms·

Huh, are the image vector embeddings implicitly doing OCR as well? Because it seems like the meme search is pulling from the text as well as images, though it's

Huh, are the image vector embeddings implicitly doing OCR as well? Because it seems like the meme search is pulling from the text as well as images, though it's not entirely clear.

bo0tzz 3y ago
CLIP does not have explicit OCR support, but it does somewhat coincidentally have a slight understanding of text. This is explained by training captions containing (some of) the text that is in the image.
- osmarks 3y ago
  I think the SigLIP models' dataset (WebLi) includes OCRed things too, so they have very good text understanding. I tested a bunch of things for my own meme search engine.
  - osmarks 3y ago
    (https://arxiv.org/pdf/2209.06794.pdf https://arxiv.org/pdf/2209.06794.pdf page 20.)