6 ms·
OlmOCR: Open-source tool to extract plain text from PDFs
- TZubiri 2y agoIt's amazing how of these solutions exist. Such a hard problem that we create for ourselves.
- rahimnathwani 2y agoGood: - no cloud service required, can run on local Nvidia GPU - outputs a single stream of text with the correct reading order (even for multi column PDF) - recognizes handwriting and stuff Bad: - doesn't seem to extract the text within diagrams (which I guess is fine because that text would be useless to an LLM) OP is the demo page, which lets you OCR 10 pages. The code needs an Nvidia GPU to run: https://github.com/allenai/olmocr https://github.com/allenai/olmocr Not sure if the VRAM requirements because I haven't tried running locally yet.
- thelittleone 2y agoText from diagrams can be useful in LLMs. For example an LLM can understand a flow charts decision making shapes etc, but without text it could misinterpret information. I process a bunch of PDFs including procedures. Diagrams are concerted to code. The text helps in many cases.
- rahimnathwani 2y agoDiagrams are concerted to code That's cool. May I ask what your pipeline looks like? And what code format do you use for diagrams? Mermaid?
- zitterbewegung 2y agoWould like to know how this compares to https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract
- rahimnathwani 2y agoTesseract is multilingual. Tesseract extracts all text from doc, without trying to fix reading order. Tesseract runs in many more places, as it doesn't require a GPU. Tesseract's pure text output tends to have a lot of extra bits, e.g. bits of text that appear in diagrams. Good as a starting point and fine for most downstream tasks.
- jesuslop 2y agoand mathpix
- rahimnathwani 2y agoWow. The Mathpix mobile app has support for reading two column PDFs as a single column. You can't run it locally, though, right?
- kergonath 2y ago> The Mathpix mobile app has support for reading two column PDFs as a single column. Mathpix is what gave the best results when I tried a whole bunch of OCR solutions on technical PDFs (multi-column with diagrams, figures and equations). It is brilliant. > You can't run it locally, though, right? Unfortunately, no. Which is a shame because I also have confidential documents to OCR and there is no way I put them on someone else’s cloud.
- rahimnathwani 2y agoDid you try marker? https://github.com/VikParuchuri/marker https://github.com/VikParuchuri/marker I haven't tried olmocr yet and I now realize my 8GB GPU probably won't cut it, as it used a 7B param VLM model under the hood.
- kergonath 2y ago> Did you try marker? I did not, but I will. Thanks for the pointer!
- xz18r 2y agoWhy exactly does this need to be AI? OCR was a thing way before the boom and works pretty fine, usually. Seems like overkill.
- simonw 2y agoKnow of any classic OCR tools that can reliably extract tabular data from scrappy PDFs? I've been hunting for a dependable option for that for years.
- constantinum 2y agoIf you're looking for better accuracy and table layout preservation, give LLMWhisperer and Docling a try! Both keep tables tidy with a Markdown-like structure.
- cle 2y agoI've not found one either. I did this at a very large scale recently and ended up just using pdfplumber. I did POC Table Transformer but the cost was too high at my scale--there are probably better options now anyway. Most seem to focus on structure detection and then use traditional OCR for the actual content extraction. It's a very hard space in the long tail, like tables that span pages or tables with complex internal structures. I went into it thinking "eh how hard can tables be?". Very hard. Thankfully it's a pretty active research area.
- rahimnathwani 2y agoLook at pages 18-20 of the technical report. I don't know of any non-AI OCR that can do as good a job as that.
- Sophira 2y agoI think it's interesting how we call this AI, because neural networks have been used for OCR for literally decades at this point. Where does "neural networks" stop and "AI" begin?
- remram 2y ago
- chad1n 2y agoThese "OCR" tools who are actually multimodals are interesting because they can do more than just text abstraction, but their biggest flaw is hallucinations and overall the nondeterministic nature. Lately, I've been using Gemini to turn my notebooks into Latex documents, so I can see a pretty nice usecase for this project, but it's not for "important" papers or papers that need 100% accuracy.
- thelittleone 2y agoHow about building a tool which indexes ocr chunks / tokens and a confidence grading. Setting a tolerance level and defining actions where the token or chunk (s) fall below that level. Actions could include could include automated verification using another model or last resort human.
- Eisenstein 2y agoHow would you calculate the confidence? LLMs are notoriously bad at grading their own output.
- vikp 2y agoI'm a fan of the team of Allen AI and their work. Unfortunately, the benchmarking of olmocr against marker (https://github.com/VikParuchuri/marker https://github.com/VikParuchuri/marker) is quite flawed. Throughput - they benchmarked marker API cost vs local inference cost for olmocr. In our testing, marker locally gets 20 - 120 pages per second on an H100 (without custom kernels, etc). Olmocr in our testing gets between .4 (unoptimized) and 4 (sglang) pages per second on the same machine. Accuracy - their quality benchmarks are based on win rate with only 75 samples - which are different between each tool pair. The samples were filtered down from a set of ~2000 based on opaque criteria. They then asked researchers at Allen AI to judge which output was better. When we benchmarked with our existing set and LLM as a judge, we got a 56% win rate for marker across 1,107 documents. We had to filter out non-English docs, since olmocr is English-only (marker is not). Hallucinations/other problems - we noticed a lot of missing text and hallucinations with olmocr in our benchmark set. You can see sample output and llm ratings here - https://huggingface.co/datasets/datalab-to/marker_benchmark_comparison_olmocr_llm https://huggingface.co/datasets/datalab-to/marker_benchmark_... . You can see all benchmark code at https://github.com/VikParuchuri/marker/tree/master/benchmarks https://github.com/VikParuchuri/marker/tree/master/benchmark... . Happy to chat more with anyone at Allen AI who wants to discuss this. I think olmocr is a great contribution - happy to help you benchmark marker more fairly.
- KennyBlanken 2y agoAre you also a fan of the Dallas Cowboys?
- rcdwealth 2y ago[dead]
- simonw 2y agoI posted some notes on this here a couple of days ago: https://simonwillison.net/2025/Feb/26/olmocr/ https://simonwillison.net/2025/Feb/26/olmocr/
- mayoosh 2y agowhat is the cost of running on the GPU?
- brianjking 2y agoI'm using the GGUF in LMStudio found here: https://huggingface.co/allenai/olmOCR-7B-0225-preview-GGUF https://huggingface.co/allenai/olmOCR-7B-0225-preview-GGUF
- fschuett 2y agoVery impressive, it's the only AI Vision toolkit so far that actually recognizes Latin and medieval scripts. I've been trying to somehow translate public-domain medieval books (including the artwork and original layout) to PDF, so they can be re-printed, i.e pages like this: https://i.imgur.com/YLuF9sa.png https://i.imgur.com/YLuF9sa.png - I tried a Google Vision + o1 solution, which did work to some extent, but not on the first try. This even recognizes the "E" of the artwork initial (or fixes it because of the context), which many OCR or AI solutions fail at. The only think I'd need now is a way to get the original font and artwork positions (would be a great addition to OlmOCR). Potentially I could work up a solution to create the font manually (as most medieval books are written in the same writing style), then find the shape of the glyphs in the original image once I have the text and then mask out the artwork with some OpenCV magic.
- yorwba 2y agoYou might be interested in https://learnable-typewriter.github.io https://learnable-typewriter.github.io for extracting the glyph shapes once you have the OCRd text.
- constantinum 2y agoTested it with the following documents: * Loan application form: It picks up checkboxes and handwriting. But it missed a lot of form fields. Not sure why? * Edsger W. Dijkstra’s handwritten notes(from Texas univ archive) - Parsing is good.* * Badly(misaligned) scanned bill - Parsing is good. Observation: there is a name field, but it produced a synonymous name instead of the name in the bill — hallucination?? * Investment fund factsheet - It could parse the bar charts and tables, but it whimsically excluded many vital data points from the document. * Investment fund factsheet, complex tables - Bad extraction, could not extract merged tables and again whimsical elimination of rows and columns. Anyone curious, try LLMWhisperer[1] for OCR. It doesn't use LLMs, so no hallucination side effects. It also preserves the layout of the input document for more context and clarity. There's also Docling[2], which is handy for converting tables from PDFs into markdown. While it uses Tesseract/EasyOCR under the hood, which can sometimes make the OCR results a bit less accurate [1] - https://pg.llmwhisperer.unstract.com/ https://pg.llmwhisperer.unstract.com/ [2] - https://github.com/DS4SD/docling https://github.com/DS4SD/docling
- phren0logy 2y agoFYI, you can choose which OCR engine Docling uses (from a handful of predefined choices) - it doesn’t have to be Tesseract. https://ds4sd.github.io/docling/reference/pipeline_options/#docling.datamodel.pipeline_options.OcrEngine https://ds4sd.github.io/docling/reference/pipeline_options/#...
- linwangg 2y ago[dead]
- Zardoz84 2y agoI was expecting a tool to extract text from PDFs, not another LLM pretending to be a reliable OCR.
- dredmorbius 2y agoFor better or worse, LLM PDF OCR conversion is likely where state of the art / market are headed. "Reliable OCR" has been a rather phenomenal oxymoron for going on five or more decades, so if you've something more reliable in mind I'd appreciate your sharing it.
- arnestrickmann 2y agothats so cool
- deleted 2y ago[deleted]
- Krasnol 2y agoMake it an .exe file and storm the world's offices.
- KennyBlanken 2y agoWasn't this just linked to here a few days ago with tests showing it has atrocious accuracy, misses a significant amount of text, and takes an order of magnitude more time (and several orders of magnitude more energy) compared to known OCR solutions?
- mjnews 2y ago[dead]
- mjnews 2y ago[dead]
- johnthescott 2y agoa surprising number of academic pdfs do not have the Title element set in the dictionary. seems like a jobs for "ai".
- brianjking 2y agoHas anyone figured out how to load this on a Huggingface endpoint?