17 ms·

I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it w

• cess11 2y ago
  What hardware are you using to run it?
  • kccqzy 2y ago
    The Gemini model isn't open so it does not matter what hardware you have. You might have confused Gemini with Gemma.
    • cess11 2y ago
      OK, I see, pity. I'm interested in similar applications but in contexts where the material is proprietary and might contain PII.
• panarky 2y ago
  This is a big aha moment for me. If Gemini can do semantic chunking at the same time as extraction, all for so cheap and with nearly perfect accuracy, and without brittle prompting incantation magic, this is huge.
  • potatoman22 2y ago
    Small point but is it doing semantic chunking, or loading the entire pdf into context? I've heard mixed results on semantic chunking.
    • panarky 2y ago
      It loads the entire PDF into context, but then it would be my job to chunk the output for RAG, and just doing arbitrary fixed-size blocks, or breaking on sentences or paragraphs is not ideal. So I can ask Gemini to return chunks of variable size, where each chunk is a one complete idea or concept, without arbitrarily chopping a logical semantic segment into multiple chunks.
      • thelittleone 2y ago
        Fixed size chunks is holding back a bunch of RAG projects on my backlog. Will be extremely pleased if this semantic chunking solves the issue. Currently we're getting around an 78-82% success on fixed size chunked RAG which is far too low. Users assume zero results on a RAG search equates to zero results in the source data.
        • refulgentis 2y ago
          FWIW, you might be doing it / ruled it out already: - BM25 to eliminate the 0 results in source data problem - Longer term, a peek at Gwern's recent hierarchical embedding article. Got decent early returns even with fixed size chunks
          • thelittleone 2y ago
• faxmeyourcode 2y ago
  I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings. This is giving me hope that it's possible.
  • otoburb 2y ago
    >>I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings. For this specific use case you can also try edgartools[1] which is a library that was relatively recently released that ingests SEC submissions and filings. They don't use OCR but (from what I can tell) directly parse the XBRL documents submitted by companies and stored in EDGAR, if they exist. [1] https://github.com/dgunning/edgartools https://github.com/dgunning/edgartools
    • faxmeyourcode 2y ago
      I'll definitely be looking into this, thanks for the recommendation! Been playing around with it this afternoon and it's very promising.
  • barrenko 2y ago
    If you'd kindly tl;dr the chunking strategies you have tried and what works best, I'd love to hear.
  • anirudhb99 2y ago
    (from the gemini team) we're working on it! semantic chunking & extraction will definitely be possible in the coming months.
  • jgalt212 2y ago
    isn't everyone on iXBRL now? Or are you struggling with historical filings?
    • faxmeyourcode 2y ago
      XBRL is what I'm using currently, but it's still kind of a mess (maybe I'm just bad at it) for some of the non-standard information that isn't properly tagged.
• yzydserd 2y ago
  How do today’s LLM’s like Gemini compare with the Document Understanding services google/aws/azure have offered for a few years, particularly when dealing with known forms? I think Google’s is Document AI.
  • zacmps 2y ago
    I've found the highest accuracy solution is to OCR with one of the dedicated models then feed that text and the original image into an LLM with a prompt like: "Correct errors in this OCR transcription".
    • bradfox2 2y ago
      This is what we do today. Have you tried it against Gemini 2.0?
    • therein 2y ago
      How does it behave if the body of text is offensive or what if it is talking about a recipe to purify UF-6 gas at home? Will it stop doing what it is doing and enter lecturing mode? I am asking not to be cynical but because of my limited experience with using LLMs for any task that may operate on offensive or unknown input seems to get triggered by all sorts of unpredictable moral judgements and dragged into generating not the output I wanted, at all. If I am asking this black box to give me a JSON output containing keywords for a certain text, if it happens to be offensive, it refuses to do that. How does one tackle with that?
      • zacmps 2y ago
        It's not something I've needed to deal with personally. We have run into added content filters in Azure OpenAI on a different application, but we just put in a request to tune them down for us.
      • xnx 2y ago
        There are many settings for changing the safety level in Gemini API calls: https://ai.google.dev/gemini-api/docs/safety-settings https://ai.google.dev/gemini-api/docs/safety-settings
• depr 2y ago
  So are you mostly processing PDFs with data? Or PDFs with just text, or images, graphs?
  • thelittleone 2y ago
    Not the parent, but we process PDFs with text, tables, diagrams. Works well if the schema is properly defined.
• sensecall 2y ago
  Out of interest, did you parse into any sort of defined schema/structure?
  • gnat 2y ago
    Parent literally said so … > Our prompt is currently very simple: "OCR this PDF into this format as specified by this json schema" and didn't require some fancy "prompt engineering" to contort out a result.
• bionhoward 2y ago
  The Gemini api has a customer noncompete, so it’s not an option for AI, what are you working on that doesn’t compete with AI?
  • B-Con 2y ago
    You do realize most people aren't working on AI, right? Also, OP mentioned fintech at the outset.
  • novaleaf 2y ago
    what doesn't compete with ai?
• xnx 2y ago
  Your OCR vendor would be smart to replace their own system with Gemini.
  • _the_inflator 2y ago
    With “Next generation, extremely sophisticated AI” to be precise, I wait say. ;) Marketing joke aside, maybe a hybrid approach could serve the vendor well. Best of both worlds if it reaps benefits or even have a look at hugging face for even more specialized aka better LLMs.
  • TeMPOraL 2y ago
    They will, and they'll still have a solid product to sell, because their value proposition isn't accurate OCR per se, but putting an SLA on it. Reaching reliability with LLM OCR might involve some combination of multiple LLMs (and keeping track of how they change), perhaps mixed with old-school algorithms, and random sample reviews by humans. They can tune this pipeline however they need at their leisure to eke out extra accuracy, and then put written guarantees on top, and still be cheaper for you long-term.
• itissid 2y ago
  Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations? One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"? > accuracy was like 96% of that of the vendor and price was significantly cheaper. I would like to know how this 96% was tested. If you use a human to do random sample based testing, well how do you adjust the random sample for variations in distribution of errors that vary like a small set of documents could have 90% of the errors and yet they are only 1% of the docs?
  • itissid 2y ago
    For an OCR company I imagine it is unconscionable to do this because if you would say OCR for an Oral History project for a library and you made hallucination errors, well you've replaced facts with fiction. Rewriting history? What the actual F.
    • phatfish 2y ago
      Probaly totally fine for a "fintech" (Crypto?) though. Most of them are just burning VC money anyway. Maybe a lucky customer gets a windfall because Gemini added some zeros.
      • rapind 2y ago
        I think you can just ask DeepSeek to create a coin for you at this point, and with the recent elimination of any oversight, you can automate your rug pulls...
    • threecheese 2y ago
      Normal OCR (like Tesseract) can be wrong as well (and IMO this happens frequently). It won’t hallucinate/straight make shit up like an LLM, but a human needs to review OCR results if the workload requires accuracy. Even across multiple runs of the same image an OCR can give different results (in some scenarios). No OCR system is perfectly accurate, they all use some kind of machine learning/floating point/potentially nondeterministic tech.
• ForHackernews 2y ago
  This is great, I just want to highlight out how nuts it is that we have spun up whole industries around extracting text that was typically printed from a computer, back into a computer. There should be laws that mandates that financial information be provided in a sensible format: even Office Open XML would be better than this insanity. Then we can redirect all this wasted effort into digging ditches and filling them back in again.
• makeitdouble 2y ago
  > After trial and error with different models As a mere occasional customer I've been scanning 4 to 5 pages of the same document layout every week in gemini for half a year, and every single week the results were slightly different. To note the docs are bilingual so it could affect the results, but what stroke me is the lack of consistency, and even with the same model, running it two or three times in a row gives different results. That's fine for my usage, but that sounds like a nightmare if everytime Google tweaks their model, companies have to reajust their whole process to deal with the discrepancies. And sticking with the same model for multiple years also sound like a captive situation where you'd have to pay premium for Google to keep it available for your use.
  • tomrod 2y ago
    Consider turning down the temperature in the configuration? LLMs have a bit of randomness in them. Gemini 2.0 Flash seems better than 1.5 - https://deepmind.google/technologies/gemini/flash/ https://deepmind.google/technologies/gemini/flash/
  • iandanforth 2y ago
    At temperature zero, if you're using the same API/model, this really should not be the case. None of the big players update their APIs without some name / version change.
    • pigscantfly 2y ago
      This isn't really true unfortunately -- mixture of experts routing seems to suffer from batch non-determinism. No one has stated publicly exactly why this is, but you can easily replicate the behavior yourself or find bug reports / discussion with a bit of searching. The outcome and observed behavior of the major closed-weight LLM APIs is that a temperature of zero no longer corresponds to deterministic greedy sampling.
      • brookst 2y ago
        If temperature is zero, and weights are weights, where is the non-deterministic behavior coming from?
• j_timberlake 2y ago
  This sounds extremely like my old tax accounting job. OCR existed and "worked" but it was faster to just enter the numbers manually than fix all the errors. Also, the real solution to the problem should have been for the IRS to just pre-fill tax returns with all the accounting data that they obviously already have. But that would require the government to care.
  • Andrex 2y ago
    They finally made filing free. So, maybe this century?
    • kennyloginz 2y ago
      Check again, Elon and his Doge team killed that.
      • happyopossum 2y ago
        No they didn’t, that claim is ridiculously easy to debunk but it has been going around because it fits the narrative.
        • djeastm 2y ago
          It'd be nicer if you wouldn't presume to know the reasons people might believe erroneous information. In this case, the reason for the misinformation is do to the lack of communication from the DOGE entity regarding their actions. Mr. Musk wrote via Tweet that he had "deleted" the digital services agency "18F" that develops the IRS Free File program and also deleted their X account. https://apnews.com/article/irs-direct-file-musk-18f-6a4dc35a92f9f29c310721af53f58b16 https://apnews.com/article/irs-direct-file-musk-18f-6a4dc35a... If indeed he did cut the agency, it remains to be see how long the application will be operational.
  • eternauta3k 2y ago
    Germany (not exactly the cradle of digitalization) already auto-fills salary tax fields with data from the employer.
• kbyatnal 2y ago
  This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" You could improve your accuracy further by adding some chain-of-thought to your prompt btw. e.g. Make each field in your json schema have a `reasoning` field beforehand so the model can CoT how it got to its answer. If you want to take it to the next level, `citations` in our experience also improves performance (and when combined with bounding boxes, is powerful for human-in-the-loop tooling). Disclaimer: I started an LLM doc processing infra company (https://extend.app/ https://extend.app/)
  • TeMPOraL 2y ago
    > The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "validate its performance, and deploy it with confidence into prod", but also sell that confidence with an SLA on top of it.
    • Cumpiler69 2y ago
      >A smart vendor will shift into that space - they'll use that LLM themselves It's a bit late to start shifting now since it takes time. Ideally they should already have a product on the market.
      • TeMPOraL 2y ago
        There's still time. The situation in which you can effectively replace your OCR vendor with hitting LLM APIs via a half-assed Python script ChatGPT wrote for you, has existed for maybe few months. People are only beginning to realize LLMs got good enough that this is an option. An OCR vendor that starts working on the shift today, should easily be able to develop, tune, test and productize an LLM-based OCR pipeline way before most of their customers realize what's been happening. But it is a good opportunity for a fast-moving OCR service to steal some customers from their competition. If I were working in this space, I'd be worried about that, and also about the possibility some of the LLM companies realize they could actually break into this market themselves right now, and secure some additional income. EDIT: I get the feeling that the main LLM suppliers are purposefully sticking to general-purpose APIs and refraining from competing with anyone on specific services, and that this goes beyond just staying focused. Some of potential applications, like OCR, could turn into money printers if they moved on them now, and they all could use some more cash to offset what they burn on compute. Is it because they're trying to avoid starting an "us vs. them" war until after they made everyone else dependent on them?
• _blk 2y ago
  Is privacy a concern?
  • jack_pp 2y ago
    In fintech I'd suspect the PDFs are public knowledge
  • surfingdino 2y ago
    Why would it be? Their only concern is IPO.
• bdd8f1df777b 2y ago
  What if you prompt Gemini that mistaking LLC for IIC is a common mistake? Will Gemini auto correct it?
  • tomrod 2y ago
    With lower temperature, it seems to work okay for me. A _killer_ awesome thing it does too is allow code specification in the config instead of through repeated attempts at prompts.
• mnky9800n 2y ago
  What are the pdfs containing? I’ve been wanting to build a system that ingests pdf reports that reference other types of data like images, csv, etc. that can also be ingested to ultimately build an analytics database from the stack of unsorted data AB’s meta data but I have not found any time to do anything like that yet. What kind of tooling do you use to build your data pipelines?
• octonaut 2y ago
  I'm interested to hear what your experience has been dealing with optional data. For example if the input pdf has fields which are sometimes not populated or nonexistent, is Gemini smart enough to leave those fields blank in the output schema? Usually the LLM tries to please you and makes up values here.
• smohare 2y ago
  [dead]
• eru 2y ago
  > For the 4% inaccuracies a lot of them are things like the text "LLC" handwritten would get OCR'd as "IIC" which I would say is somewhat "fair". I'm actually somewhat surprised Gemini didn't guess from context that LLC is much more likely? I guess the OCR subsystem is intentionally conservative? (Though I'm sure you could do a second step on your end, take the output from the conservative OCR pass, and sent it through Gemini and ask it to flag potential OCR problems? I bet that would flag most of them with very few false positives and false negatives.)
• eitally 2y ago
  It's great to hear it's this good, and it makes sense since Google has had several years of experience creating document-type-specific OCR extractors as components of their Document AI product in Cloud. What most heartening is to hear that the legwork they did for that set of solutions has made it into Gemini for consumers (and businesses).
• RamblingCTO 2y ago
  Successful document processing vendors to use LLMs already. I know this at least of klippa. They have (apparently) fine-tuned models, prompts etc. The biggest issue with using LLMs directly is error handling, validation and "parameter drift"/randomness. This is the typical "I'll build it myself but worse" thing
• faebi 2y ago
  Did you try other vision models such as ChatGPT and Grok? I'm doing something similar but struggled to find good comparisons in between the vision models in terms OCR and document understanding.
• ensocode 2y ago
  Just to make sure: you are talking about your experiences with Gemini 1.5 Flash here, right?
• andai 2y ago
  If the documents have the same format, maybe you could include an example document in the prompt, so the boilerplate stuff (like LLC) gets handled properly.
• martingoodson 2y ago
  I work in financial data and our customers would not accept 96% accuracy in the data points we supply. Maybe 99.96%. For most use cases in financial services, accurate data is very important.
  • riku_iki 2y ago
    so, what solution are you using to extract data with 99.96% accuracy?
• surfingdino 2y ago
  You could ingest them with AWS Textract and have predictability and formatting in the format of your choice. Using LLMs for this is lazy and generates unpredictable and non-deterministic results.
• rco8786 2y ago
  “LLC” to “IIC” is one thing. But wouldn’t that also make it just as easy to to mistake something like “$100” for “$700”?
• mring33621 2y ago
  Hi! Any guesstimate for pages/minute from your Gemini OCR experience? Thanks!
• PaulHoule 2y ago
  Where I work we've had great success at using LLMs to OCR paper documents that look like https://static.foxnews.com/foxnews.com/content/uploads/2023/12/Fox_December-10-13-2023_National_Topline_December-17-Release.pdf https://static.foxnews.com/foxnews.com/content/uploads/2023/... but were often written with typewriters long ago to get nice structured tabular output. Deals with text being split across lines and across pages just fine.
• lebimas 2y ago
  You could probably take this a step further and pipe the OCR'ed text into Claude 3.5 Sonnet and get it to fix any OCR errors
• chickenWing 2y ago
  It is cheaper now, but I wonder if it will continue to be cheaper when companies like Google and OpenAI decide they want to make a profit off of AI, instead of pouring billions of dollars of investment funds into it. By the time that happens, many of the specialized service providers will be out of business and Google will be free to jack up the price.
  • kbaker 2y ago
    I use Claude through OpenRouter (with Aider), and was pretty amazed to see that it routes the requests during the same session almost round-robin through Amazon Bedrock, sometimes through Google Vertex, sometimes through Anthropic themselves, all of course using the same underlying model. Literally whoever has the cheapest compute. With the speed that AI models are improving these days, it seems like the 'moat' of a better model is only a few months before it is commoditized and goes to the cheapest provider.
• chewkokwah 2y ago
  How about the comparison with traditional proprietary on premise software like ONMIPage or ABBYY or those listed below: https://en.wikipedia.org/wiki/Comparison_of_optical_character_recognition_software https://en.wikipedia.org/wiki/Comparison_of_optical_characte...
• llm_bot 2y ago
  I'm curious to hear about your experience with this. Which solution were you using before (the one that took 12 minutes)? If it was a self-hosted solution, what hardware were you using? How does Gemini handle PDFs with an unknown schema, and how does it compare to other general PDF parsing tools like Amazon Textract or Azure Document Intelligence? In my initial test, tables and checkboxes weren't well recognized.