4 ms·

The crucial information is missing - accuracy comparison with other OCR providers. From my experience LLM based OCR might misread the layout and hallucinate val

The crucial information is missing - accuracy comparison with other OCR providers. From my experience LLM based OCR might misread the layout and hallucinate values, it is very subtle but sometimes critically wrong. Classical OCR has more precision but doesn't get the layout at all. Combining both has other issues, no approach is 100% reliable.

agentcoops 1y ago
Have you evaluated this lately? Last year or even just earlier this year I would have mostly agreed with you. At this point, however, with at least the documents I have been working on, OCR reliability with GPT5 or Mistral OCR [1] has been much better than even domain-trained classical OCR. If the documents have even slightly complex layout (to say nothing of page numbers or page headings or an uncommon font), the accuracy of state of the art LLMs has been in my work an order of magnitude greater. The ability to have the LLM tentatively combine trailing sentences across pages, which is especially useful if you have to work with documents in German say, is invaluable. [1] https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr
- zarzavat 1y ago
  I asked GPT-5 to OCR a table for me the other day, it hallucinated perhaps 10% of the values. This was a screenshot of a spreadsheet, with large font, not challenging except for the layout. What's interesting is that I asked it to also read the background colors of the cells and it did much worse on that task. I believe these models could be useful for a first pass if you are willing to manually review everything they output, but the failure mode is unsettling.
smusamashah 1y ago
Any tool that takes a scanned PDF, then overlay's OCRed text over scan so that text becomes searchable?
- Xmd5a 1y ago
  https://github.com/ocrmypdf/OCRmyPDF https://github.com/ocrmypdf/OCRmyPDF >OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched I ... I nailed it.
  - quibono 1y ago
    Just a note that OCRmyPDF currently uses Tesseract
WithinReason 1y ago
Breaking up the page, feeding the pieces one-by-one and reassembling the output helps with that. I was expecting this project to do that but it can only feed a whole page.
worldsayshi 1y ago
Yes I tried using LLM for reading CV:s a while back and I really struggled with getting it to not omit important information.

agentcoops 1y ago

Have you evaluated this lately? Last year or even just earlier this year I would have mostly agreed with you. At this point, however, with at least the documents I have been working on, OCR reliability with GPT5 or Mistral OCR [1] has been much better than even domain-trained classical OCR. If the documents have even slightly complex layout (to say nothing of page numbers or page headings or an uncommon font), the accuracy of state of the art LLMs has been in my work an order of magnitude greater. The ability to have the LLM tentatively combine trailing sentences across pages, which is especially useful if you have to work with documents in German say, is invaluable. [1] https://mistral.ai/news/mistral-ocr https://mistral.ai/news/mistral-ocr

zarzavat 1y ago
I asked GPT-5 to OCR a table for me the other day, it hallucinated perhaps 10% of the values. This was a screenshot of a spreadsheet, with large font, not challenging except for the layout. What's interesting is that I asked it to also read the background colors of the cells and it did much worse on that task. I believe these models could be useful for a first pass if you are willing to manually review everything they output, but the failure mode is unsettling.

smusamashah 1y ago

Any tool that takes a scanned PDF, then overlay's OCRed text over scan so that text becomes searchable?

Xmd5a 1y ago
https://github.com/ocrmypdf/OCRmyPDF https://github.com/ocrmypdf/OCRmyPDF >OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched I ... I nailed it.
- quibono 1y ago
  Just a note that OCRmyPDF currently uses Tesseract

WithinReason 1y ago

Breaking up the page, feeding the pieces one-by-one and reassembling the output helps with that. I was expecting this project to do that but it can only feed a whole page.

worldsayshi 1y ago

Yes I tried using LLM for reading CV:s a while back and I really struggled with getting it to not omit important information.