5 ms·

Azure Doc Intel is good. It creates a semantic representation (is json) of the doc. The schema is useful with page, paragraph, table, etc objects. The bounding-

by hm-nah 2y ago

Azure Doc Intel is good. It creates a semantic representation (is json) of the doc. The schema is useful with page, paragraph, table, etc objects. The bounding-box for each element is also useful. It flails with images, only giving you the bounding-box of where the image exists. It’s up to you to extract any images separately then figure out how to correlate. Overall I think it’s useful rather than rolling your own (at least for small scale).

simonw 2y ago
I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document. That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.
- larodi 2y ago
  Have you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
  - simonw 2y ago
    Oh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.
c_moscardi 2y ago
Yeah, I think MS' is the best out there, but agree that the usability leaves something to be desired. 2 thoughts: 1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case. 2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct. If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406efc875c https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e... [1] https://mindee.github.io/doctr/latest/using_doctr/using_models.html#end-to-end-ocr https://mindee.github.io/doctr/latest/using_doctr/using_mode...
- simonw 2y ago
  That does look pretty great, thanks for the tip. Sending images through that API and then using an LLM to extract data from the text result from the OCR could be worth exploring.

by hm-nah 2y ago

simonw 2y ago

I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document. That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.

larodi 2y ago
Have you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
- simonw 2y ago
  Oh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.

c_moscardi 2y ago

Yeah, I think MS' is the best out there, but agree that the usability leaves something to be desired. 2 thoughts: 1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case. 2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct. If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406efc875c https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e... [1] https://mindee.github.io/doctr/latest/using_doctr/using_models.html#end-to-end-ocr https://mindee.github.io/doctr/latest/using_doctr/using_mode...

simonw 2y ago
That does look pretty great, thanks for the tip. Sending images through that API and then using an LLM to extract data from the text result from the OCR could be worth exploring.