4 ms·

You don't really need a LLM for OCR. Hell, I suppose they just run a python script in its VM and rephrase the output. At least that's what I would do. Perhaps

You don't really need a LLM for OCR. Hell, I suppose they just run a python script in its VM and rephrase the output. At least that's what I would do. Perhaps the script would be a "specialist model" in a sense.

Spivak 2y ago
It's not that you need an LLM for OCR but the fact that an LLM can do OCR (and handwriting recognition which is much harder) despite not being made specifically for that purpose is indicative of something. The jump from knowing "this is a picture of a paper with writing on it" like what you get with CLIP to being able to reproduce what's on the paper is, to me, close enough to seeing that the difference isn't meaningful anymore.
- _lvbh 2y ago
  GPT-4v is provided with OCR
  - letmevoteplease 2y ago
    No reason to believe that. Open source VLMs can do OCR.[1] [1] https://huggingface.co/spaces/opencompass/open_vlm_leaderboard https://huggingface.co/spaces/opencompass/open_vlm_leaderboa...
  - simonw 2y ago
    That's a common misconception. Sometimes if you upload an image to ChatGPT and ask for OCR it will run Python code that executes Tesseract, but that's effectively a bug: GPT-4 vision works much better than that, and it will use GPT-4 vision if you tell it "don't use Python" or similar.