ritvikpandey21

4 ms·

1.
by ritvikpandey21 6mo ago
We've been building table extraction at Pulse and evaluated four benchmarks: OmniDocBench, SCORE-Bench, ParseBench, and RD-TableBench. None of them fully reflect the enterprise document workflows we've encountered in production. T
2.
PulseBench-Tab: Open-source, multilingual benchmark for table extraction(runpulse.com)
5 points by ritvikpandey21 6mo ago|1 comments
3.
by ritvikpandey21 10mo ago
Results look pretty good (with the exception of one very faint page) - check it out here! https://platform.runpulse.com/dashboard/extractions/public/f...
4.
by ritvikpandey21 10mo ago
thanks! we benchmark against all the major players (azure doc intelligence, aws textract, google doc ai, frontier llms, etc). we have some public news coming out soon on this front, but we have a very rigorous dataset using both public and
5.
by ritvikpandey21 10mo ago
yeah models are definitely improving, but we've found even the latest ones still hallucinate and infer text rather than doing pure transcription. we carry out very rigorous benchmarks against all of the frontier models. we think the di
6.
by ritvikpandey21 10mo ago
yeah models are definitely improving, but we've found even the latest ones still hallucinate and infer text rather than doing pure transcription. we carry out very rigorous benchmarks against all of the frontier models. we think the di
7.
by ritvikpandey21 10mo ago
thanks for the flag! have pointed this out will be pushing an update here shortly
8.
by ritvikpandey21 10mo ago
we disagree! we've found llms by themselves aren't enough and suffer from pretty big failure modes like hallucination and inferring text rather than pure transcription. we wrote a blog about this [1]. the right approach so far see
9.
by ritvikpandey21 10mo ago
thanks! appreciate the kind words
10.
by ritvikpandey21 10mo ago
our team has tested docling pretty extensively, works well for simpler text-heavy docs without complex layouts, but the moment you introduce tables or multi-column stuff it doesn't maintain layout well.
11.
by ritvikpandey21 10mo ago
we're more focused on the core extraction layer itself rather than workflow tooling. we train our own vision models for layout detection, ocr, and table parsing from scratch. the key thing for us is determinism and auditability, so out
12.
by ritvikpandey21 1y ago
DeepSeek AI just released DeepSeek-OCR, a new open-source model that aims to rethink text extraction through what it calls Context Optical Compression. The launch quickly caught attention on X and GitHub, with many celebrating another big s
13.
by ritvikpandey21 1y ago
interesting read
14.
by ritvikpandey21 1y ago
We processed hundreds of millions of pages and found that a single accuracy metric is misleading. A model that's 98% accurate on 1,000 pages with 200 data elements each still produces 4,000 incorrect values. The real killers are broken
15.
by ritvikpandey21 1y ago
We evaluated ByteDance's Dolphin document parsing model on enterprise document processing tasks using standardized benchmarks and real-world document sets. Our testing dataset included 847 financial documents, 312 legal forms, and 156
16.
by ritvikpandey21 1y ago
After processing nearly 500 million pages of enterprise documents, we've discovered that the biggest challenge in document AI isn't character recognition or table extraction. It's something far more fundamental: understanding
17.
Why Semantic Understanding Breaks at Page Boundaries(runpulse.com)
2 points by ritvikpandey21 1y ago|1 comments
18.
Legacy OCR Tools Are Failing the Legal Industry: Here's Why(runpulse.com)
2 points by ritvikpandey21 2y ago|0 comments
19.
by ritvikpandey21 2y ago
curious how LLM hallucinations will work on logging info - gonna be a hard problem to solve
20.
by ritvikpandey21 2y ago
as builders in this space, we decided to put it to the test on complex nested tables, pie charts, etc. to see if the same VLM hallucination issues persist, and to what degree. while results were promising, we found several critical failure
21.
Beyond the Hype: Real-World Tests of Mistral's OCR(runpulse.com)
4 points by ritvikpandey21 2y ago|3 comments
22.
by ritvikpandey21 2y ago
We put Mistral AI's new OCR to the test against complex documents that matter for real business use cases. While it outperforms a lot of frontier LLMs, we found critical limitations for finance, legal, and healthcare domains.
23.
by ritvikpandey21 2y ago
claude is definitely better than gpt -- but both have their flaws! they pretty much fall flat on their face with nested entries, low-fidelity images, etc. (we detailed this heavily in our blog post here [1]) other ocr providers are doing a
24.
by ritvikpandey21 2y ago
to not make the read extra long, we only included one example. we tried over 50 docs and found a couple with pie charts/bar graphs that weren't parsed at all. there were also a few instances with entire column entires incorrect du
25.
by ritvikpandey21 2y ago
thanks man!
26.
by ritvikpandey21 2y ago
we respect andrew a lot, as we mentioned in our blog! he's an absolute legend in the field, founded google brain, coursera, worked heavily on baidu ai. this is more to inform everyone not to blindly trust new document extraction tools
27.
by ritvikpandey21 2y ago
don't be mistaken, andrew's a legend! he's done some incredible work -- google brain, coursera, baidu ai, etc.
28.
by ritvikpandey21 2y ago
we're not the biggest believers in 'agentic' parsing! we definitely do believe there's a specific role for LLMs in the data ingestion pipeline, but this occurs more when bar graphs/charts/figures -> structur
29.
by ritvikpandey21 2y ago
good catch on the 1654, will edit that on our blog! try it multiple times, we've noticed esp for tabular data it's fairly nondeterministic. we trialed it over 10 times on many financial CIMs and observed this phenomena.
30.
Putting Andrew Ng's OCR models to the test(runpulse.com)
124 points by ritvikpandey21 2y ago|61 comments

More ›