5 ms·
Author claims that the most likely is that there is Tesseract running behind ChatGPT-4v/o. There is no way that this is Tesseract. -> Tesseract accuracy is ve
by rvnx 2y ago
Author claims that the most likely is that there is Tesseract running behind ChatGPT-4v/o.
There is no way that this is Tesseract.
-> Tesseract accuracy is very low, it can barely do OCR on printed documents.
- jerrygenser 2y agoEven if tesseract accuracy is low, if the tesseract result in addition to the image is then passed to the LLM, it can result in a much more accurate OCR. For example, GPT4 with some vision capability would be able to fill in the incorrect OCR with the additional word co-occurrence understanding. I've tested this approach with purely text LLM to correct OCR mistakes and it works quite well. Also note that in some newer OCR pipelines that don't involve LLMs, there is a vision component and then a text correcting model that is in some ways similar to some forms of spell check, which can further improve results.
- lyu07282 2y agoyou can tell that the OCR fails more in cases without natural language like with code/random characters. OAI seems to claim 4o is a fully end to end multimodal model, but we will never know for sure, we can't trust a single word OpenAI is saying.
- kherud 2y agoShouldn't this theory be testable? The response time for an image of the same size should remain constant (assuming a generated response of constant size). You could then try to put an increasing amount of text inside of the image. If this text is fed to the LLM using OCR, the total amount of tokens grows. You should then be able to observe an increase in response time.
- RicoElectrico 2y agoYeah, Tesseract is barely production quality.
- lyu07282 2y agoyeah it was SOTA in 2006, 18 years ago
- jascha_eng 2y agoOther than proprietary models, what is better than it today? Just asking in case I ever need OCR and don't want to pay the cloud providers for it :D
- lyu07282 2y agocheckout https://github.com/mindee/doctr https://github.com/mindee/doctr or https://github.com/VikParuchuri/surya https://github.com/VikParuchuri/surya for something practical multimodal llm would of course blow it all out the water, so some llama3-like model is probably SOTA in terms of what you can run yourself. something like https://huggingface.co/blog/idefics2 https://huggingface.co/blog/idefics2
- freedmand 2y agoAgreed. Tesseract is not able to handle handwriting or text that is distorted well, e.g. colored text over an image background — to the point that it would hurt any downstream LLM trying to make sense of the contents. It won’t even pick out bounding boxes. I doubt they are running an OCR model, but if they actually were it would likely be an in-house one trained with more modern techniques.
- llm_trw 2y agoBecause no one knows how to prep the images. With the right file type and resolution I get under a single character error per 10 pages and it's been that good since the late 00s.
- yorwba 2y agoHow do you prep the images?
- llm_trw 2y agoMay hourly rate starts at $300. If you'd like to hire me you're more than welcome to. I've done this work for a number of companies in the past.
- alach11 2y agoWith handwriting? With mixed fonts? Tesseract requires heavy customization and extension to perform reasonably on these workloads. The off-the-shelf options from major cloud providers blow it out of the water.
- llm_trw 2y agoNever had to use it with handwriting, mixed fonts and text where location carries semantic infirmation: absolutely.
- neochief 2y agoI once uploaded a giant image to chatGPT, asking to transcribe it, but the request failed, and in the error message there was a reference to some python script related to tesseract stuff. Since then I'm 100% tesserract is used there in some aspect for text recognition.