4 ms·

Regarding the campaign finance documents, it's not just a vision problem with handwritten filings. Even when I feed Gemini Pro a typed PDF with normal text (the

Regarding the campaign finance documents, it's not just a vision problem with handwritten filings. Even when I feed Gemini Pro a typed PDF with normal text (the kind you can search, select, and copy in any PDF reader) it just invents things to fill in the result. I asked it to summarize the largest donors of a local campaign given a 70-page PDF and it gave me a top-10 table that seemed plausible but all the names were pulled from somewhere else in Gemini's tiny brain. None of them appeared anywhere in the filing.

simonw 3y ago
Yeah, I'm beginning to think detailed OCR-style extraction just isn't a good fit for these models. It's frustrating, because getting that junk out of horribly scanned PDFs is 90% of the need of data journalism! The best OCR API I've used is still AWS Textract - have you seen anything better? I ended up building a tiny CLI tool for Textract because I found the thing so hard to use: https://github.com/simonw/textract-cli https://github.com/simonw/textract-cli
- hm-nah 3y ago
  Azure Doc Intel is good. It creates a semantic representation (is json) of the doc. The schema is useful with page, paragraph, table, etc objects. The bounding-box for each element is also useful. It flails with images, only giving you the bounding-box of where the image exists. It’s up to you to extract any images separately then figure out how to correlate. Overall I think it’s useful rather than rolling your own (at least for small scale).
  - simonw 3y ago
    I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document. That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.
    - larodi 3y ago
      Have you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
      - simonw 3y ago
        Oh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.
  - c_moscardi 3y ago
    Yeah, I think MS' is the best out there, but agree that the usability leaves something to be desired. 2 thoughts: 1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case. 2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct. If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406efc875c https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e... [1] https://mindee.github.io/doctr/latest/using_doctr/using_models.html#end-to-end-ocr https://mindee.github.io/doctr/latest/using_doctr/using_mode...
    - simonw 3y ago
      That does look pretty great, thanks for the tip. Sending images through that API and then using an LLM to extract data from the text result from the OCR could be worth exploring.
- mistrial9 3y ago
  OCR software is thirty years in the making! there must be dozens of alternatives .. interested to hear from people close to this topic
- is_true 3y ago
  I had a good experience using easyocr to get data out of lottery draws videos
wcedmisten 3y ago
This is why I can't really trust LLMs for data tasks like this. I can't really be certain that it won't be hallucinating results
devmor 3y ago
Generative AI is just not the Leatherman Tool everyone wants it to be. There are absolutely ways to create solutions in this problem space using ML as a tool, but they are more specialized and probably not economical until the cost of training bespoke models goes down.
- anamax 3y ago
  "Leatherman Tool" is a good comparison, as pretty much every "blade" in a Leathermann is a crappy tool, better only than the Swiss Army knife equivalent. The Leatherman's reason for existence is that you have it with you for unexpected problems.

simonw 3y ago

Yeah, I'm beginning to think detailed OCR-style extraction just isn't a good fit for these models. It's frustrating, because getting that junk out of horribly scanned PDFs is 90% of the need of data journalism! The best OCR API I've used is still AWS Textract - have you seen anything better? I ended up building a tiny CLI tool for Textract because I found the thing so hard to use: https://github.com/simonw/textract-cli https://github.com/simonw/textract-cli

hm-nah 3y ago
Azure Doc Intel is good. It creates a semantic representation (is json) of the doc. The schema is useful with page, paragraph, table, etc objects. The bounding-box for each element is also useful. It flails with images, only giving you the bounding-box of where the image exists. It’s up to you to extract any images separately then figure out how to correlate. Overall I think it’s useful rather than rolling your own (at least for small scale).
- simonw 3y ago
  I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document. That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.
  - larodi 3y ago
    Have you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
    - simonw 3y ago
      Oh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.
- c_moscardi 3y ago
  Yeah, I think MS' is the best out there, but agree that the usability leaves something to be desired. 2 thoughts: 1. I believe the IR jargon for getting a JSON of this form is Key Information Extraction (KIE). MS has an out-of-the-box model for this. I just tried the screenshot and it did a pretty good (but not perfect) job. It didn't get every form field, but most. MS sort-of has a flow for fine-tuning, but it really leaves a lot to be desired IMO. Curious if this would be "good enough" to satisfy the use case. 2. In terms of just OCR (i.e. getting the text/numeric strings correct), MS is known to be the best on typed text at the moment [1]. Handwriting is a different beast... but it looks like MS is doing a very good job there (and SOTA on handwriting is very good). In particular, it got all the numbers in that screenshot correct. If you want to see the results from MS on the screenshot in this blog post, here's the entire JSON blob. A bit of a behemoth but the key/value stuff is in there: https://gist.github.com/cmoscardi/8c376094181451a49f0c62406efc875c https://gist.github.com/cmoscardi/8c376094181451a49f0c62406e... [1] https://mindee.github.io/doctr/latest/using_doctr/using_models.html#end-to-end-ocr https://mindee.github.io/doctr/latest/using_doctr/using_mode...
  - simonw 3y ago
    That does look pretty great, thanks for the tip. Sending images through that API and then using an LLM to extract data from the text result from the OCR could be worth exploring.
mistrial9 3y ago
OCR software is thirty years in the making! there must be dozens of alternatives .. interested to hear from people close to this topic
is_true 3y ago
I had a good experience using easyocr to get data out of lottery draws videos

wcedmisten 3y ago

This is why I can't really trust LLMs for data tasks like this. I can't really be certain that it won't be hallucinating results

devmor 3y ago

Generative AI is just not the Leatherman Tool everyone wants it to be. There are absolutely ways to create solutions in this problem space using ML as a tool, but they are more specialized and probably not economical until the cost of training bespoke models goes down.

anamax 3y ago
"Leatherman Tool" is a good comparison, as pretty much every "blade" in a Leathermann is a crappy tool, better only than the Swiss Army knife equivalent. The Leatherman's reason for existence is that you have it with you for unexpected problems.