ritvikpandey21

6 ms·

31.
by ritvikpandey21 2y ago
Today, Andrew Ng, one of the legends of the AI world, released a new document extraction service that went viral on X: https://x.com/AndrewYNg/status/1895183929977843970 At Pulse, we put the models to the test wit
32.
by ritvikpandey21 2y ago
hey -- wrote a blog post about this exact phenomena [1] (also posted on HN couple weeks back [2]). tldr: maintaining confidence from LLM nondeterministic outputs over millions of pages is a problem. especially in production environments lik
33.
by ritvikpandey21 2y ago
hey, i recommend checking out the previous HN threads [1] on why LLMs shouldn’t be used in production-grade OCR, especially if accuracy is super important (as in the audiobook case) we wrote the post and created Pulse [2] for these exact us
34.
by ritvikpandey21 2y ago
thanks for the kind words! we do have a mailing list for current users, I can add you to that
35.
by ritvikpandey21 2y ago
on raw text, LLM’s usually do not struggle. however, when you start processing low-fidelity images (receipt scans with stains, documents with marks all over it, bent corners/areas, rotated docs) these transcription issues become extrem
36.
by ritvikpandey21 2y ago
yup, the models you tried out require a lot of work to be able to run efficiently. additionally for actual decent ocr it’ll require very high quality document datasets (of PDF/excel/pptx). scanned documents are especially hard and
37.
by ritvikpandey21 2y ago
appreciate the feedback. completely agree, tuning and/or training a VLM will definitely produce better ocr extractions. however, it’s notoriously hard to accumulate a really good ground truth labeled dataset of pdf/excel/pp
38.
by ritvikpandey21 2y ago
we completely agree - mechanistic interpretability might help keep these language models in check, but it’s going to be very difficult to run this on closed source frontier models. im excited to see where that field progresses
39.
by ritvikpandey21 2y ago
yup, important clarification! the language portion of the model also works with the extraction however, and is prone to the hallucinations
40.
by ritvikpandey21 2y ago
give it a shot with a few of the examples in the blog! or better yet, find some financial statements from Goldman/morgan Stanley and run it through the model.
41.
by ritvikpandey21 2y ago
yea seems like o1-pro was able to solve a few of those variations in the paper we referenced. take a look at the rest of the examples and let me know! the paper’s (somewhat) old at this point, but if you try our more complex square variatio
42.
by ritvikpandey21 2y ago
we don’t think so - we’ve fine tuned most of the SOTA language models available today on table datasets, documents with complex layouts, and while they do perform better, seems like they’re still prone to the same hallucinations. these fron
43.
by ritvikpandey21 2y ago
yes they usually can! we delved into the mathematics behind this a bit in the blog, but tldr the LLMs are making educated guesses based on the embedding similarities - which can be detrimental for ocr systems.
44.
by ritvikpandey21 2y ago
thanks for the feedback!
45.
by ritvikpandey21 2y ago
for most (text-dense) documents without much layout differences, these small prompt eng tricks work pretty well! scaling this to complex layouts and 1000+ page docs, we found the models don’t stick to their instructions. perhaps there’s som
46.
by ritvikpandey21 2y ago
would love to give this a shot with pulse! feel free to reach out to me at ritvik [at] trypulse [dot] ai, and i’d be very curious to give these a run! in general, i’m happy to give some general advice on algos/models to fine-tune for t
47.
by ritvikpandey21 2y ago
yup, this is a pretty common occurrence in using LLMs for data extraction. For personal use (trying to load a receipt) it’s great that the LLM filled in info. For production systems which need high quality, near 100% extraction accuracy, in
48.
Why LLMs still have problems with OCR(runpulse.com)
218 points by ritvikpandey21 2y ago|145 comments
49.
by ritvikpandey21 2y ago
Document ingestion and the launch of Gemini 2.0 caused a lot of buzz this week. As a team building in this space, this is something we researched thoroughly. Here’s our take: ingestion is a multistep pipeline, and maintaining confidence fro
50.
by ritvikpandey21 2y ago
ritvik here from pulse. everyone’s pretty much made the right points here, but wanted to emphasize that due to the llm architecture, they predict “the most probable text string” that corresponds to the embedding, not necessarily the exact t