5 ms·
Show HN: Benchmarking VLMs vs. Traditional OCR
Vision models have been gaining popularity as a replacement for traditional OCR. Especially with Gemini 2.0 becoming cost competitive with the cloud platforms.
We've been continuously evaluating different models since we released the Zerox package last year (https://github.com/getomni-ai/zerox https://github.com/getomni-ai/zerox). And we wanted to put some numbers behind it. So we’re open sourcing our internal OCR benchmark + evaluation datasets.
Full writeup + data explorer here: https://getomni.ai/ocr-benchmark https://getomni.ai/ocr-benchmark
Github: https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark
Huggingface: https://huggingface.co/datasets/getomni-ai/ocr-benchmark https://huggingface.co/datasets/getomni-ai/ocr-benchmark
Couple notes on the methodology:
1. We are using JSON accuracy as our primary metric. The end goal is to evaluate how well each OCR provider can prepare the data for LLM ingestion.
2. This methodology differs from a lot of OCR benchmarks, because it doesn't rely on text similarity. We believe text similarity measurements are heavily biased towards the exact layout of the ground truth text, and penalize correct OCR that has slight layout differences.
3. Every document goes Image => OCR => Predicted JSON. And we compare the predicted JSON against the annotated ground truth JSON. The VLMs are capable of Image => JSON directly, we are primarily trying to measure OCR accuracy here. Planning to release a separate report on direct JSON accuracy next week.
This is a continuous work in progress! There are at least 10 additional providers we plan to add to the list.
The next big roadmap items are:
- Comparing OCR vs. direct extraction. Early results here show a slight accuracy improvement, but it’s highly variable on page length.
- A multilingual comparison. Right now the evaluation data is english only.
- A breakdown of the data by type (best model for handwriting, tables, charts, photos, etc.)
- jasonjmcghee 2y agoA big takeaway for me is that Gemini Flash 2.0 is a great solution to OCR, considering accessibility, cost, accuracy, and speed. It also has a 1M token context window, though from personal experience it seems to work better the smaller the context window is. Seems like Google models have been slowly improving. It wasn't so long ago I completely dismissed them.
- shawabawa3 2y agoAnd from my personal experience with Gemini 2.0 flash Vs 2.0 pro is not even close I had gemini 2.0 pro read my entire hand written, stain covered, half English, half french family cookbook perfectly first time It's _crazy_ good. I had it output the whole thing in latex format to generate a printable document immediately too
- anon373839 2y agoI’m definitely not getting that takeaway. This wasn’t even an OCR benchmark: the task was structured data extraction, and deterministic metrics were set aside in favor of GPT-as-a-judge. VLMs are every bit as susceptible to the (unsolved) hallucination problem as regular LLMs are. I would not use them to do OCR on anything important because the failure modes are totally unbounded (unlike regular OCR).
- bayindirh 2y agoAlso, what's strange is there's no free of paid OCR engine is added to the mix for the evaluation. Tessaract is built specially for OCR'in scanned documents, and it has a both traditional and neural network based modes, to boot.
- michaelt 2y ago> Also, what's strange is there's no free of paid OCR engine is added to the mix for the evaluation. The article says they evaluated "Traditional OCR providers (Azure, AWS Textract, Google Document AI, etc.)" Are those not paid OCR engines?
- bayindirh 2y agoYou're absolutely correct. I read the article quite fast, and assumed they are AI, albeit not LLM powered systems as well. I'm using computers since I can read, and when somebody says "traditional OCR", I think about the older systems like Tessaract or ABBYY's FineReader which can be again automated for batch processing, albeit mostly locally. Sending huge amount of PDFs to a cloud server to get them processed is still a bit alien to me, since it can be done on-premises (or on a VPS with the said software) very efficiently from my perspective.
- arresin 2y agoWhat is the privacy of the documents for the cloud service? There’s nothing in the privacy policy about data sent over the api.
- dcreater 2y agoThen you have to assume by default that your data is visible to their employees, can be monetized, used to improve their models etc
- arresin 2y agoI’ll stick with the devil I know that is only slightly worse in their own benchmarks (Google)
- cess11 2y ago"San Francisco, California" There is none, because CLOUD Act.
- gbertb 2y agoHow does this compare to Marker https://github.com/VikParuchuri/marker https://github.com/VikParuchuri/marker?
- marcotac 2y agoI'm curious as well
- simonw 2y agoThe benchmark I most want to see around OCR is one that covers risks from accidental (or deliberate) prompt injection - I want to know how likely it is that a model might OCR a page and then accidentally act on instructions in that content rather than straight transcribing it as text. I'm interested in the same thing for audio transcription too, for models like Gemini or GPT-4o audio accepting audio input.
- mentalgear 2y agoAn interesting OCR aspect indeed; hence it's great that their OCR Benchmark is open source, allowing for the addition of such a category. Or maybe there are already separate OCR prompt-injection benchmarks. Also, I'd be useful to understand how an OCR context differs from standard injection attacks. One thing I can think of is potential tabular injection attacks. But also image-based, especially for VLMs, are relevant. So a OCR injection attack benchmark might just be a combination of different domain-specific benchmarks formated as images.
- magicalhippo 2y agoPlaying with local llama vision and minicpm-v models, they do seem resistant to what one might call blatant prompt injection. Ie just inserting one of the classic "ignore previous instructions" or similar. So yeah, would be curious how susceptible they are to more refined approaches. Are there some known examples?
- themanmaran 2y agoWe've tested basic prompt injections within images, but not been able to reliably trigger any adverse effects. However there are two big bugs we've found with VLMs: 1. Correcting the document. If you have an income statement, and all the line items add up to $1,001. But the total says $1000. The model will frequently correct the final output. Which would be terrible if you were trying to build a "identify mistakes in these documents" type tool. 2. Infinite loops. Sometimes the models will get hung up on a particular token and repeat that until it times out. This gets triggered a lot in markdown tables |---|---|----------------->
- th0ma5 2y ago
- banditelol 2y agoAnyone have tried comparing with Qwen VL based model? I heard good things about its performance on ocr compared to other self hostable model, but haven't really tried benchmarking its performance
- jimmySixDOF 2y agoYes I'd like to see this repeated with any of the small VLM's like IBM Granite or the HF Smols. Pretty much anything in the sub 7B range.
- blindriver 2y agoAs far as I'm concerned, all of these specialty services are dead compared to a generalized LLM like OpenAI or Gemini. I wrote in a previous post about how NLP services were dead because of LLMs and obviously people in NLP took great offense to that. But I was able to use the NLP abilities of an LLM without needing to know anything about the intricacies of NLP or any APIs and it worked great. This post on OCR pretty much shows exactly what I meant. Gemini does OCR almost as good as OmniAI (granted I've never heard of it), but at 1/10th the cost. OpenAI will only get better very quickly. Kudos to OmniAI for releasing honest data, though. Sure you might get an additional 5% accuracy from OmniAI vs Gemini but a generalized LLM can do so much more than just OCR. I've been playing with OpenAI this entire weekend and literally the sky's the limit. Not only can you OCR images, you can ask the LLM to summarize it, transform it into HTML, classify it, give a rating based on whatever parameters you want, get a lexile score, all in a single API call. Plus it will even spit out the code to do all of the above for you to use as well. And if it doesn't do what you need it to do right now, it will pretty soon. I think the future of AI is going to be pretty bleak for everyone except the extremely big players that can afford to invest hundreds of billions of dollars. I also think there's going to be a real battle of copyright in less than 5 years which will also favor the big rich players as well.
- modo_mario 2y agoWouldn't an issue be that whilst for LLM's replacing NLP's you don't often care about the super rare hickup or hallucination. Whilst where OCR's tend to be used it's often a no go.... Just saying this trying to remember all the places where I've implemented it or seen it implemented. A common one was billing stuff.
- Eridrus 2y ago5% accuracy can be worth a lot. The price of any of these services pales in comparison to getting a human involved in any fraction of cases. It is likely reasonable to expect the base LMs to keep getting better and for there to not be a moat on accuracy in the long term, but businesses are not just built on benchmark accuracy and have plenty of other ways to survive, even if the technology under the hood changes.
- alok-g 2y agoLooking at the sample documents, this seems more focused on tables and structured data extraction and not long-form texts. The ground truth JSON has so much less information than the original document image. I would love to see a similar benchmark for full contents including long-form text and tables.
- lyu07282 2y agoIndeed, from their conclusions: > They [VLMs] are generally more capable of "looking past the noise" of scan lines, creases, watermarks. Traditional models tend to outperform on high-density pages (textbooks, research papers) as well as common document formats like tax forms. Which is a bit confusing? Did they test that or what? It doesn't seem that way from their limited dataset.
- westurner 2y agoHarmonic Loss converges more efficiently on MNIST OCR: https://github.com/KindXiaoming/grow-crystals https://github.com/KindXiaoming/grow-crystals .. "Harmonic Loss Trains Interpretable AI Models" (2025) https://news.ycombinator.com/item?id=42941954 https://news.ycombinator.com/item?id=42941954
- alkh 2y agoWhat is the best solution for recognizing handwritten text that combines multiple languages, especially in cases where certain letters look the same but represent different sounds? For example, the letter 'p' in English versus 'р' in Cyrillic languages, which sounds more like the English 'r'.
- jll29 2y agoDoes anyone have good experience with a particular pipeline for OCR-ing C source code?
- danielcampos93 2y agoGPT-4o as a judge to evaluate the quality of something which gpt4o is not inherently that good at. Red flag.
- fzysingularity 2y agoWhat VLMs do you use when you're listing OmniAI - is this mostly wrapping the model providers like your zerox repo?
- betula_ai 2y agoThank you for sharing this. Some of the other public models that we can host ourselves may perform in practice better than the models listed - e.g. Qwen 2.5 VL https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file
- EarlyOom 2y agoOCR seems to be mostly solved for 'normal' text laid out according to Latin alphabet norms (left to right, normal spacing etc.), but would love to see more adversarial examples. We've seen lots of regressions around faxed or scanned documents where the text boxes may be slightly rotated (e.g. https://www.cad-notes.com/autocad-tip-rotate-multiple-texts-at-once-to-readable-orientation/ https://www.cad-notes.com/autocad-tip-rotate-multiple-texts-...) not to mention handwriting and poorly scanned docs. Then there's contextually dependent information like X-axis labels that are implicit from a legend somewhere, so its not clear even with the bounding boxes what the numbers refer to. This is where VLMs really shine: they can extract text then use similar examples from the page to map them into their output values when the bounding box doesn't provide this for free.
- codelion 2y agoThat's a great point about the limitations of traditional OCR with rotated or poorly scanned documents. I agree that VLMs really shine when it comes to understanding context and extracting information beyond just the text itself. It's pretty cool how they can map implicit relationships, like those X-axis labels you mentioned.
- deleted 2y ago[deleted]
- hasibzunair 2y agoWhat kind of VLMs are being used in OmniAI? I fine-tuned a Llama 3.2 Vision on a small dataset I created for extracting text without heavy cropping. Results are simply amazing in comparison with OCR-based approaches. It can be tried here: https://news.ycombinator.com/item?id=43192417 https://news.ycombinator.com/item?id=43192417
- default_ 2y agoWondering why Florence-2 is not on the list of models?