5 ms·
My impression is that OCR is basically solved at this point. The OmniAI benchmark that's also referenced here wasn't updated with new models since February 202
by pietz 1y ago
My impression is that OCR is basically solved at this point.
The OmniAI benchmark that's also referenced here wasn't updated with new models since February 2025. I assume that's because general purpose LLMs have gotten better at OCR than their own OCR product.
I've been able to solve a broad range of OCR tasks by simply sending each page as an image to Gemini 2.5 Flash Lite and asking it nicely to extract the content in Markdown under some additional formatting instructions. That will cost you around $0.20 for 1000 pages in batch mode and the results have been great.
I'd be interested to hear where OCR still struggles today.
- kbumsik 1y ago> My impression is that OCR is basically solved at this point. Not really in practice to me. Especially they still struggle with Table format detection.
- coulix 1y agoThis. Any complex parent table span cell relationship still has low accuracy. Try the reverse, take a complex picture table and ask Chatgpt5, claude Opus 3.1, Gemini Pro 2.5 to produce a HTML table. They will fail.
- bobsmooth 1y agoMaybe I misunderstood the assignment but it seems to work for me. https://chatgpt.com/share/68f5f9ba-d448-8005-86d2-c3fbae028bcf https://chatgpt.com/share/68f5f9ba-d448-8005-86d2-c3fbae028b... Edit: Just caught a mistake, transcribed one of the prices incorrectly.
- kbumsik 1y agoRight, I wouldn't use full table detection to VLM model because they tend to mistake with numbers in table...
- pietz 1y agoMaybe my imagination is limited or our documents aren't complex enough, but are we talking about realistic written documents? I'm sure you can take a screenshot of a very complex spreadsheet and it fails, but in that case you already have the data in structured form anyway, no?
- kbumsik 1y ago> realistic written documents? Just get a DEF 14A (Annual meeting) filing of a company from SEC EDGAR. I have seen so many mistakes when looking at the result closely. Here is a DEF 14A filing from Salseforce. You can print it to a PDF and then try converting. https://www.sec.gov/Archives/edgar/data/1108524/000110852425000009/crm-20250424.htm https://www.sec.gov/Archives/edgar/data/1108524/000110852425...
- grosswait 1y agoHistorical filings are still a problem, but hasn’t the SEC required filing in an XML format since the end of 2024?
- richardlblair 1y agoIt's not really about SEC filings, though. While we folks on HN would never think of hard copies of invoices, but much of the world still operates this way. As mentioned above I have about 200 construction invoices. They are all formatted in a way that doesn't make sense. Most fail both OCR and OpenAI
- KoolKat23 1y agoOpenAI has unusuably low image DPI. Try Gemini.
- daemonologist 1y agoNow if someone mails or faxes you that spreadsheet? You're screwed. Spreadsheets are not the biggest problem though, as they have a reliable 2-dimensional grid - at worst some cells will be combined. The form layouts and n-dimensional table structures you can find on medical and insurance documents are truly unhinged. I've seen documents that I struggled to interpret.
- richardlblair 1y agoI had mentioned this when the new QWEN model dropped - I have a stack of construction invoices that fail through both OCR and OpenAI. It's a hard (and very interesting) problem space.
- carschno 1y agoTechnically not OCR, but HTR (hand-written text/transcript recognition) is still difficult. LLMs have increased accuracy, but their mistakes are very hard to identify because they just 'hallucinate' text they cannot digitize.
- sramam 1y agoInteresting - have you tried sending the image and 'hallucinated' text together to a review LLM to fix mistakes? I don't have a use case of 100s or 1000s of hand-written notes have to be transcribed. I have only done this with whiteboard discussion snapshots and it has worked really well.
- lazide 1y agoOften, the review LLM will also say everything is okay when it’s made up too.
- mormegil 1y agoThis. I am reading old vital records in my family genealogy quest, and as those are sometimes really difficult to read, I turned to LLMs, hearing they are great in OCR. It’s been… terrible. The LLM will transcribe the record without problems, the output seems completely correct, a typical text of a vital record. Just… the transcribed text has nothing to do with my specific record. On the other hand, transkribus.eu has been fairly usable for old vital record transcription – even though the transcribed text is far from perfect, many letters and words are recognized incorrectly, it helps me a lot with the more difficult records.
- pietz 1y agoWe ran a small experiment internally on this and it looked like Gemini is better at handwriting recognition than I am. After seeing what it parsed, I was like "oh yeah, that's right". I do agree that instead of saying "Sorry, I can't read that" it just made up something.
- CraigRood 1y ago
- raincole 1y agoIf you can accept that the machine just make up what it doesn't recognize instead of saying "I don't know," then yes it's solved. (I'm not being snarky. It's acceptable in some cases.)
- jakewins 1y agoBut this was very much the case with existing OCR software as well? I guess the LLMs will end up making up plausible looking text instead of text riddled with errors, which makes it much harder to catch the mistakes, in fairness
- rkagerer 1y agoGood libraries gave results with embedded confidence levels for each unit recognized.
- wahnfrieden 1y agoExisting ocr doesn’t skip over entire (legible) paragraphs or hallucinate entire sentences
- Davidzheng 1y agorarely happens to me using LLMs to transcribe pdfs
- criddell 1y agoI usually run the image(s) through more than one converter then compare the results. They all have problems, but the parts they agree on are usually correct.
- KoolKat23 1y agoThis must be some older/smaller model.
- red75prime 1y agoJust checked it with Gemini 2.5 Flash. Instructing it to mark low-confidence words seems to work OK(ish).
- peter-m80 1y agoNo way it's solved. try to make OCR over a magazine with creative layouts. Not possible. I have a collection of vintage computer magazines and from time to time I try to OCR them whith the state of the art mechanisms. All of them requiere a lot of human intervention
- jmkni 1y agodo you have an example of a particularly tricky one?
- ekianjo 1y agoJust try old ads you will see how hard it gets
- pietz 1y agoCould you provide an example that fails? I'm interested in this.
- constantinum 1y agoI use LLMWhisperer[1] for OCR'ing old magazine ads. It preserves the layout and context. Example > https://postimg.cc/ts3vT7kG https://postimg.cc/ts3vT7kG https://pg.llmwhisperer.unstract.com/ https://pg.llmwhisperer.unstract.com/
- darkwater 1y agoSo, the mug with inspirational text says "Bountiful Potential"?
- cahaya 1y agoLot's of OCR/ LLM's (Even Gemini Pro 2.5) still struggle converting complex tables to markdown or HTML: Tables with multiple headers and merged cells that get mixed up, multiple columns with tick boxes get mixed up, multi page tables that are not understood correctly. Also Llamaindex fails miserably on those things. Curious to hear which OCR/ LLM excels with these specific issues? Example complex table: https://cdn.aviation.bot/complex-tables.zip https://cdn.aviation.bot/complex-tables.zip I can only parse this table correctly by first parsing the table headers manually into HTML as example output. However, it still mixes up tick boxes. Full table examples: https://www.easa.europa.eu/en/icao-compliance-checklist https://www.easa.europa.eu/en/icao-compliance-checklist
- pietz 1y agoI threw in the first image/table into Gemini 2.5 Pro letting it choose the output format and it looks like it extracted the data just fine. It decided to represent the checkboxes as "checked" and "unchecked" because I didn't specify preferences.
- CaptainOfCoit 1y ago> Lot's of OCR/ LLM's (Even Gemini Pro 2.5) still struggle converting complex tables to markdown or HTML: But that's something else, that's no longer just OCR ("Optical Character Recognition"). If the goal suddenly changes from "Can take letters in images and make into digital text" to "Can replicate anything seen on a screen", the problem-space gets too big. For those images you have, I'd use something like Magistral + Structured Outputs instead, first pass figure out what's the right structure to parse into, second pass to actually fetch and structure the data.
- eeixlk 1y agohtceaad t nofdnsy lyruuieo sieerrr t owcope?
- kmacdough 1y ago> But that's something else, that's no longer just OCR ("Optical Character Recognition"). Lines often blur for technologies under such rapid evolution. Not sure it's helpful to nitpick the verbal semantics. It is a fair question whether the OCR-inspired approach is the correct approach for more complex structured documents. But saying it's "not OCR" do doesn't seem meaningful from a technical perspective.
- vintermann 1y agoOCR of printed text may be one thing, but handwriting OCR (a.k.a HTR) is very, very far from solved. It's actually hard to find a practical task general historical HTR is good enough to do usefully, even for state of the art models.
- robotswantdata 1y agoVLLMs suck at complex layouts and there is a high risk of hallucination. Never use alone for contracts or health data.
- sbinnee 1y agoMaybe for English. Other languages are very much not solved.
- llm_nerd 1y agoComplex documents is where OCR struggles mightily. If you have a simple document with paragraphs of text, sure OCR is pretty solved. If you have a complex layout with figures and graphs and supporting images and asides and captions and so on (basically any paper, or even trade documents), it absolutely falls apart. And GP LLMs are heinous at OCR. If you are having success with FL, your documents must be incredibly simple. There has been enormous advances in OCR over the past 6 months, so the SoTa is a moving, rapidly advancing target.
- burpsnard 1y agoI've only used tesseract, 'recreationally', but i tried generating images of random chars to see what resolution/contrast/noise was minimally recognisable; shocked at how bad it was. heavily relies on language models of character sequences, pretty useless On 'line noise'
- baobun 1y agoChinese, especially handwritten.
- Davidzheng 1y agoI think it'll be good to have an end-to-end pdf to latex converter for old math papers. For commutative diagrams almost all models still struggle. especially very complicated commutative diagrams.
- simlevesque 1y ago> That will cost you around $0.20 for 1000 pages in batch mode and the results have been great. Can you explain more about your setup ? I have a quarter million pages I want to OCR.
- constantinum 1y agoWhy PDF parsing is Hell[1]: Fixed layout and lack of semantic structure in PDFs. Non-linear text flow due to columns, sidebars, or images. Position-based text without contextual or relational markers. Absence of standard structure tags (like in HTML). Scanned or image-based PDFs requiring OCR. Preprocessing needs for scanned PDFs (noise, rotation, skew). Extracting tables from unstructured or visually complex layouts. Multi-column and fancy layouts breaking semantic text order. Background images and watermarks interfering with text extraction. Handwritten text recognition challenges. [1] https://unstract.com/blog/pdf-hell-and-practical-rag-applications/ https://unstract.com/blog/pdf-hell-and-practical-rag-applica...
- KoolKat23 1y agoI agree, Gemini 2.5 models are excellent. The fuss around old fashioned OCR seemed strange to me initially considering the above, but I selfishly forgot to consider addressing compute/offline requirements. It would also be nice for there to be a good competitor.
- cle 1y agoThat will not work with many of the world's most important documents because of information density. For example, dense tables or tables with lots of row/col spans, or complex forms with checkboxess, complex real-world formatting and features like strikethroughs, etc. To solve this generally you need to chunk not by page, but by semantic chunks that don't exceed the information density threshold of the model, given the task. This is not a trivial problem at all. And sometimes there is no naive way to chunk documents so that every element can fit within the information density limit. A really simple example is a table that spans hundreds pages. Solving that generally is an open problem.
- Gazoche 1y agoThere is no "solved" in computer vision, there is only "good enough" and what constitutes "good enough" depends on your problem domain. Take an OCR model with 99.9% character-wise accuracy. Sounds pretty good, right? Well if your use case is, say, digitalizing old printed novels, then yeah it's probably good enough. But what if your documents are personal records with millions of names, to insert in some administrative database? Now 1 out of 1000 persons will have their name misspelled. Ooops.
- 6gvONxR4sf7o 1y agoOCR for printed documents is super robust, but handwriting, low res, and aligned recognition (not just image to "hello world" but also having "h is here in space e is here in space...) are all still well behind "basically solved."
- kelvinjps10 1y agoGoogle vision it's still better than Gemini at OCR, for example at getting bounding boxes.
- blindriver 1y agoI attempted OCR using all of the open source models available about 3 months ago, including Llama 4. These were pngs of text using a regular font. Most produced garbage except Llama 4, and even then it was only about 90% accurate. Using OpenAI or Gemini produced much better results but the open source models were really bad.
- themanmaran 1y ago> OmniAI benchmark that's also referenced here wasn't updated with new models since February 2025. I assume that's because general purpose LLMs have gotten better at OCR than their own OCR product. Benchmark author here. No, just pivoted away from OCR API as a product! Still use our API internally but have been lazy about updating benchmarks. Gemini is definitely the best model for OCR. But it has a really high rate of "recitation" errors. Where it will determine the output token is too close to its training data and cut it off. Something like 10% of the time from our testing. Also it has this hilarious hallucination when you have a blank page in the document mix and it just makes up new info. OpenAI is OK. GPT5 wasn't any better than 4o or 4.1. Main issues were: dropping content like headers/footers, loses it's mind on sideways pages, and will frequently refuse to read things like ID documents, health care forms, or things it judges to have too much PII.
- veidr 1y agoClearly-printed text to a sequence of characters is solved, for use cases that don't require 100% accuracy. But not for semantic document structure — recognizing that the grammatically incomplete phrase in a larger font is a heading, recognizing subheadings and bullet lists, tables, etc. Also not for handwritten text, text inside of images (signage and so forth), or damaged source material (old photocopies and scans created in the old days). Those areas all seem to me where an LLM-based approach could narrow the gap between machine recognition and humans. You have to sort of reason about it from the context as a human to figure it out, too.