3 ms·
We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Eve
by themanmaran 2y ago
We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is:
- Every document has ground truth text, a JSON schema, and the ground truth JSON.
- Run OCR on each document and pass the result to GPT-4o along with the JSON Schema
- Compare the predicted JSON against the ground truth JSON for accuracy.
In our benchmark, the ground truth text => gpt-4o was 99.7%+ accuracy. Meaning whenever gpt-4o was given the correct text, it could extract the structured JSON values ~100% of the time. So if we pass in the OCR text from Mistral and it scores 70%, that means the inaccuracies are isolated to OCR errors.
https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark
- cdolan 2y agowere you guys able to finish running the benchmark with mistral and got a 70% score? Missed that Edit - I see it on the Benchmark page now. Woof, low 70% scores in some areas! https://getomni.ai/ocr-benchmark https://getomni.ai/ocr-benchmark
- themanmaran 2y agoYup, surprising results! We were able to dig in a bit more. Main culprit is the overzealous "image extraction". Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002). And it happened with a lot of full documents as well. Ex: most receipts got classified as images, and so it didn't extract any text.
- lingjiekong 2y agowhere do you find this regarding "Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002)."?
- culi 2y agothemanmaran works at Omni so presumably they have access to the actual resulting data from this study
- cdolan 2y agoThis sounds like a real problem and hurdle for North American (US/CAN in particular) invoice and receipt processing?
- someothherguyy 2y agoWouldn't that just bias itself to the shape of the text extracted from the OCR against the shape of the raw text alone? It doesn't seem like it would be a great benchmark for estimating semantic accuracy?