3 ms·
Yeah, I'm beginning to think detailed OCR-style extraction just isn't a good fit for these models. It's frustrating, because getting that junk out of horribly
by simonw 3y ago
Yeah, I'm beginning to think detailed OCR-style extraction just isn't a good fit for these models.
It's frustrating, because getting that junk out of horribly scanned PDFs is 90% of the need of data journalism!
The best OCR API I've used is still AWS Textract - have you seen anything better?
I ended up building a tiny CLI tool for Textract because I found the thing so hard to use: https://github.com/simonw/textract-cli https://github.com/simonw/textract-cli
- hm-nah 3y agoAzure Doc Intel is good. It creates a semantic representation (is json) of the doc. The schema is useful with page, paragraph, table, etc objects. The bounding-box for each element is also useful. It flails with images, only giving you the bounding-box of where the image exists. It’s up to you to extract any images separately then figure out how to correlate. Overall I think it’s useful rather than rolling your own (at least for small scale).
- simonw 3y agoI have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document. That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.
- larodi 3y agoHave you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
- simonw 3y agoOh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.
- c_moscardi 3y agoYeah, I think MS' is the best out there, but agree that the usability leaves something to be desired. 2 thoughts: 1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case. 2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct. If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406efc875c https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e... [1] https://mindee.github.io/doctr/latest/using_doctr/using_models.html#end-to-end-ocr https://mindee.github.io/doctr/latest/using_doctr/using_mode...
- simonw 3y agoThat does look pretty great, thanks for the tip. Sending images through that API and then using an LLM to extract data from the text result from the OCR could be worth exploring.
- mistrial9 3y agoOCR software is thirty years in the making! there must be dozens of alternatives .. interested to hear from people close to this topic
- is_true 3y agoI had a good experience using easyocr to get data out of lottery draws videos