3 ms·
Love to see another benchmark! We published the OmniAI OCR benchmark the other week. Thanks for adding us to the list. One question on the "Automation" score i
by themanmaran 2y ago
Love to see another benchmark! We published the OmniAI OCR benchmark the other week. Thanks for adding us to the list.
One question on the "Automation" score in the results, is this a function of extraction accuracy vs the accuracy of the LLM's "confidence score". I noticed the "accuracy" column was very tightly grouped (between 79 & 84%) but the automation score was way more variable.
And side note: is there an open source Mistral benchmark for their latest OCR model? I know they claimed it was 95% accurate, but it looks that was based on an internal evaluation.
- prats226 2y agoAutomation is combination of both, accuracy and accuracy of confidence scores. Good way to think about automation is recall at high precision which is what you need for true automation where you don't worry about documents that are very likely to have correct results and focus on manually correcting documents likely to have errors. The reason accuracies are tighly grouped but not the automation is because these models are trained to be accurate but not necessarily predictable, where there is no real way to get confidence score calibrated. Couldn't find the benchmark mistral used as well