18 ms·

Replace OCR with Vision Language Models

292 points by EarlyOom 2y ago

gfiorav 2y ago
I wonder what the speed of this approach vs traditional ocr techniques. Also, curious if this could be used for text detection (find a bounding box containing text within an image).
- vunderba 2y ago
  Was just coming here to say this, there does not yet exist a multimodal vision LLM approach that is capable of identifying bounding boxes of where the text occurs. I suppose you could manually cut the image up and send each part separately to the LLM but that feels like an kludge and it's still in-exact.
  - EarlyOom 2y ago
    We can do bounding boxes too :) we just call it visual grounding https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb https://github.com/vlm-run/vlmrun-cookbook/blob/main/noteboo...
    - vunderba 2y ago
      Wait what? That's pretty neat. I'm on my phone right now, so I can't really view the notebook very easily. How does this work? Are you using some kind of continual partitioning of the image and refeeding that back into the LLM to sort of pseudo-zoom in/out on the parts that contain non-cut off text until you can resolve that into rough coordinates?
    - deleted 2y ago
      [deleted]
    - what 2y ago
      Kind of skeptical since you also provide a “confidence” value, which has to be entirely made up. Do you have an example that isn’t a sample drivers license? Something that is unlikely to have appeared in an LLM’s training data?
  - chpatrick 2y ago
    qwen 2.5 vl was specifically trained to produce bounding boxes I believe.
submeta 2y ago
Can I use this to convert flowcharts to yaml representations?
- EarlyOom 2y ago
  We convert to a JSON schema, but it would be trivial to convert this to yaml. There are some minor differences in e.g. tokens required to output JSON vs yaml which is why we've opted for our strategy.
orliesaurus 2y ago
I think OCR tools are good at what they say on the box, recognizing characters on a piece of paper etc. If I understand this right, the advantage of using a vision language model is the added logic that you can say things like: "Clearly this is a string, but does it look like a timestamp or something else?"
- EarlyOom 2y ago
  VLMs are able to take context into account when filling in fields, following either a global or field specific prompt. This is great for e.g. unlabeled axes, checking a legend for units to be suffixed after a number, etc. Also, you catch lots of really simple errors with type hints (e.g. dates, addresses, country codes etc.).
- vintermann 2y ago
  You can also use it for robustness. Looking at e.g. historical censuses, it's amazing how many ways people found to not follow the written instructions for filling them out. Often the information you want is still there, but woe to you if you look at the columns one by one and assume the information in them to be accurate and neatly within its bounding box.
- raxxorraxor 2y ago
  This has always been part of the complete OCR package as far as I know. The raw result of an OCR constantly fails to differentiate 1 l I i | or other similar symbols/letters. Maybe this necessary step can be improved and altered with a VLM. There is also the preprocessing where the image get its perspective corrected. Not sure how well a VLM performs here. As you said, I think combining these techniques will be the most efficient way forward.
ekidd 2y ago
I've been experimenting with vlm-run (plus custom form definitions), and it works surprisingly well with Gemini 2.0 Flash. Costs, as I understand, are also quite low for Gemini. You'll have best results with simple to medium-complexity forms, roughly the same ones you could ask a human to process with less than 10 minutes of training. If you need something like this, it's definitely good enough that you should consider kicking the tires.
- fzysingularity 2y ago
  Very cool! If you have more examples / schemas you'd be interested in sharing, feel free to add to the `contrib` section.
- fzysingularity 2y ago
  BTW Check out the Gemini qualitative results here in our hub: https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qualitative-results https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qu.... It gives you an idea of where today's models fail (Gemini Flash, OpenAI gpt4o+mini, open-source ones like Llama 3.2 Vision, Qwen VL 2.5 etc).
Eisenstein 2y ago
If you just want to play with using a vision model to do OCR, I made a little script that uses KoboldCpp to do it locally. * https://github.com/jabberjabberjabber/LLMOCR https://github.com/jabberjabberjabber/LLMOCR
LeoPanthera 2y ago
What's the characters-per-Wh of an LLM compared to traditional OCR?
- fzysingularity 2y ago
  That's a tough one to answer right now, but to be perfectly honest, we're off by 2-3 orders of magnitude in terms of chars/W. That said, VLMs are extremely powerful visual learners with LLM-like reasoning capabilities making them more versatile than OCR for practically all imaging domains. In a matter of a few years, I think we'll essentially see models that are more cost-performant via distillation, quantization and the multitude of tricks you can do to reduce the inference overhead.
- mlyle 2y ago
  A lot worse. But, higher quality OCR will reduce the amount of human post-processing needed, and, in turn will allow us to reduce the number of humans. Since humans are relatively expensive in energy use, this can be expected to save a lot of energy.
  - rafram 2y ago
    > Since humans are relatively expensive in energy use Are they? I'm seeing figures around 80 watts at rest, and 150 when exercising. The brain itself only uses about 20 watts [1]. That's 1/35 of a single H100's power consumption (700 watts - which doesn't even take into account the energy required to cool the data center, the humans who build and maintain it, ...). [1]: https://www.humanbrainproject.eu/en/follow-hbp/news/2023/09/04/learning-brain-make-ai-more-energy-efficient/#:~:text=It%20is%20estimated%20that%20a%20human%20brain,hydroelectric%20plant%20if%20they%20were%20done%20artificially. https://www.humanbrainproject.eu/en/follow-hbp/news/2023/09/...
    - mlyle 2y ago
      The PUE of humans for that 80 watts is terrible, though. Ridiculous multiples of additional energy needed to convert solar power to a form of a energy that they can use, and even the manufacturing lifecycle and transport of humans to the datacenter is energy inefficient.
tgtweak 2y ago
Not really interested until this can run locally without api keys :\
- EarlyOom 2y ago
  You can! it works with Ollama https://github.com/vlm-run/vlmrun-hub https://github.com/vlm-run/vlmrun-hub At the end of the day its just schemas. You can decide for yourself if its work upgrading to a larger, more expensive model.
beebaween 2y ago
What's the best way to run this is I prefer to use local GPUs?
- EarlyOom 2y ago
  You can try out some of our schemas with Ollama if you want: https://github.com/vlm-run/vlmrun-hub https://github.com/vlm-run/vlmrun-hub (instructions in Readme)
- fzysingularity 2y ago
  We’re adding this as we speak. Ollama support is already there, and here’s vLLM inference: https://github.com/vlm-run/vlmrun-hub/pull/120 https://github.com/vlm-run/vlmrun-hub/pull/120
mmusson 2y ago
Lol. The resume includes expert in Mia Khalifa easter egg.
gunian 2y ago
replaced it with real humans -> nano tech in their brain -> transmit to server getting almost 99% accuracy
intalentive 2y ago
What's the value-add here? The schemas?
- fzysingularity 2y ago
  We've seen so many different schemas and ways of prompting the VLMs. We're just standardizing it here, and making it dead-simple to try it out across model providers.
- vlmrunadmin007 2y ago
  Basically there is no model schema combination. IF you go ahead and prompt a open source model with the schema it doesn't produce the results in the expected format. The main contribution is how to make these model conform to your specific needs and in a structured format.
  - idiliv 2y ago
    Wait, but we're doing that already, and it works well (Qwen 2.5 VL)? If need be, you can always resort to structured generation to enforce schema conformity?
TZubiri 2y ago
Wow thanks! There's a client who had a startup idea that involved analyzing pdfs, I used textract, but it was too cumbersome and unreliable. Maybe I can reach out to see if he wants to give it anothee go with this!
- fzysingularity 2y ago
  Let us know, I think >70% of OCR tasks today can be done with VLMs with a little bit of guidance ;). Ping us at contact "at" vlm.run
deleted 2y ago
[deleted]
skbjml 2y ago
This is awesome!
rafram 2y ago
It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sources, and it made up names and dates that sort of fit the vibe of the document when it couldn’t read the text! They sounded right for the ethnicity and time period but were entirely fake. There’s no way to ground the model using the source text when the model is your OCR.
- EarlyOom 2y ago
  This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.
  - hashta 2y ago
    An effective way that usually increases accuracy is to use an ensemble of capable models that are trained independently (e.g., gemini, gpt-4o, qwen). If >x% of them have the same output, accept it, otherwise reject and manually review
    - rafram 2y ago
      There’s a very low chance that three separate models will come up with the same result. There are always going to be errors, small or large. Even if you find a way around that, running the process three times on every page is going to be prohibitively expensive, especially if you want to finetune.
      - vintermann 2y ago
        No, running it two or three times for every page isn't prohibitive. In fact, one of the arguments for using modern general-purpose multimodal models for historical HTR is that it is cheaper and faster than Transkribus. What you can do is for instance to ask one model for a transcription, and ask a second model to compare the transcription to the image and correct any errors it finds. You actually have a lot of budget to try things like these if the alternative is to fine-tune your own model.
syntaxing 2y ago
Maybe I’m being greedy but is it possible to have a vLLM detect when a portion is an image? I want to convert some handwritten notes into markdown but some portion are diagrams. I want the vLLM to extract the diagrams to embed into the markdown output
- vlmrunadmin007 2y ago
  We have successfully tested the model with vLLM and plan to release it across multiple inference server frameworks, including vLLM and OLAMA.
themanmaran 2y ago
We recently published an open source benchmark [1] specifically for evaluating VLM vs OCR. And generally the VLMs did much better than the traditional OCR models. VLM highlights: - Handwriting. Being contextually aware helps here. i.e. they read the document like a human would, interpreting the whole word/sentence instead of character by character - Charts/Infographics. VLMs can actually interpret charts or flow diagrams into a text format. Including things like color coded lines. Traditional OCR highlights: - Standardized documents (e.x. US tax forms that they've been trained on) - Dense text. Imagine textbooks and multi column research papers. This is the easiest OCR use case, but VLMS really struggle as the number of output tokens increase. - Bounding boxes. There still isn't really a model that gives super precise bounding boxes. Supposedly Gemini and Qwen were trained for it, but they don't perform as well as traditional models. There's still a ton of room for improvement, but especially with models like Gemini the accuracy/cost is really competitive. [1] https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark
- fzysingularity 2y ago
  Saw your benchmark, looks great. Will run our models against those benchmark and share some of our learnings. As you mentioned there are a few caveats to VLMs that folks are typically unaware of (not at all exhaustive, but the ones you highlighted): 1. Long-form text (dense): Token limits of 4/8K mean that dense pages may go over limits of the LLM outputs. This requires some careful work to make them work as seamlessly as OCR. 2. Visual grounding a.k.a. bounding boxes are definitely one of those things that VLMs aren't natively good at (partly because the cross-entropy losses used aren't really geared for bounding box regression). We're definitely making some strides here [1] to improve that so you're going to get an experience that is almost as good as native bounding box regression (all within the same VLM). [1] [1] https://colab.research.google.com/github/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb https://colab.research.google.com/github/vlm-run/vlmrun-cook...
BrannonKing 2y ago
What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixels are just noise).
- NoMoreNicksLeft 2y ago
  A big difficulty there is typeface detection, some of these were never digital fonts. But, even if it could detect them, you likely don't have those fonts on your computer to be able to put it back together as a digital typesetting for any but the most trivial fonts.
  - retrorangular 2y ago
    The tool could include all known open source fonts, and for the rest, maybe could have a model recreate missing fonts for non-patented fonts, as while font files (.ttf, .otf, .woff, etc.) are copyrighted, styles usually do not have design patents, so tracing and re-creating them is usually not an issue as far as I'm aware (not a lawyer.) [1] Though if it accidentally "traces" one of the few exceptions, then you've potentially committed a crime, and the big difficulty in typeface detection you mention increases those odds. That said, there are so few exceptions that even if the model couldn't properly identify a font, it might be able to identify whether a font is likely to have a design patent. I do think getting an AI to create a high quality vector font from a potentially low-res raster graphic is going to be quite challenging though. Raster to vector tools I've tried in the past left a bit to be desired. 1. https://www.copyright.gov/comp3/chap900/ch900-visual-art.pdf https://www.copyright.gov/comp3/chap900/ch900-visual-art.pdf > As a general rule, typeface, typefont, lettering, calligraphy, and typographic ornamentation are not registrable. 37 C.F.R. § 202.1(a), (e). These elements are mere variations of uncopyrightable letters or words, which in turn are the building blocks of expression. See id. The Office typically refuses claims based on individual alphabetic or numbering characters, sets or fonts of related characters, fanciful lettering and calligraphy, or other forms of typeface. This is true regardless of how novel and creative the shape and form of the typeface characters may be. > There are some very limited cases where the Office may register some types of typeface, typefont, lettering, or calligraphy, such as the following: > • Pictorial or graphic elements that are incorporated into uncopyrightable characters or used to represent an entire letter or number may be registrable. Examples include original pictorial art that forms the entire body or shape of the typeface characters, such as a representation of an oak tree, a rose, or a giraffe that is depicted in the shape of a particular letter. > • Typeface ornamentation that is separable from the typeface characters is almost always an add-on to the beginning and/or ending of the characters. To the extent that such flourishes, swirls, vector ornaments, scrollwork, borders and frames, wreaths, and the like represent works of pictorial or graphic authorship in either their individual designs or patterned repetitions, they may be protected by copyright. However, the mere use of text effects (including chalk, popup papercraft, neon, beer glass, spooky-fog, and weathered-and-worn), while potentially separable, is de minimis and not sufficient to support a registration. > The Office may register a computer program that creates or uses certain typeface or typefont designs, but the registration covers only the source code that generates these designs, not the typeface, typefont, lettering, or calligraphy itself. For a general discussion of computer programs that generate typeface designs, see Chapter 700, Section 723.
fl0under 2y ago
Looks cool! May also be interested in Allen AI's OCR tool olmOCR they just released too [1][2]. They say "convert a million PDF pages for only $190 USD". [1] https://github.com/allenai/olmocr https://github.com/allenai/olmocr [2] https://arxiv.org/abs/2502.18443 https://arxiv.org/abs/2502.18443
- TZubiri 2y ago
  The issue with that promise is that anyone can convert pdfs, the question is whether the conversions are correct or whether you have Income Expenses 200 100 On one document, and Income Expenses 20 0100 On others. There's no shortage of products that tried to solve this problem from scratch (or by piggybacking on other projects) and called it a day without worrying about the huge problem that is quality and parseability. The most robust players just give you the coordinates of a glyph and you are on your own: Textract, PDFBox.
rasz 2y ago
I rather see machine learning used to help OCR by - recognizing/recreating exact font used - helping align/rotate source Not to hallucinate gibberish when source lacks enough data.
erulabs 2y ago
You sort of have to use both. OCR and LLM and then correlate the two results. They are bad at very different things, but a subsequent call to a 2nd LLM to pair together the results does improve quality significantly, plus you get both document understanding and context as well as bounding boxes, etc. I'm building a "never fill out paperwork again" app, if anyone is interested, would be happy to chat!
- cpursley 2y ago
  Any tips on how to prompt that second pairing step? And what sort of things to ask the llm to extract in step 1?
- fzysingularity 2y ago
  We think VLMs would outperform most OCR+LLM solutions in due time. I get that there’s need for these hybrid solutions today, but we’re comparing 20+ year mature tech vs something that’s roughly 1.5 years old. Also, VLMs are end-to-end trainable, unlike OCR+LLM solutions (that are trained separately), so it’s clear that these approaches scale much better for domain-specific use cases or verticals.
- K0balt 2y ago
  A VLM that invokes ocr tool use is a compelling idea that could result in pretty good results, I would expect.
cyp0633 2y ago
Existing solutions like Tesseract already can embed text into the image, but I'm wondering if there's a way to combine LLM with Tesseract, so that LLMs can help correcting results and finding unidentified text, and finally still embed text back to the image
serjester 2y ago
Good to see more work being done here, but I don't understand why this is tied to someone's proprietary API. Swapping model providers and adding some basic logging is not remotely painful enough to justify onboarding yet another vendor. Especially one that's handling something as sensitive as LLM prompts.
temp0826 2y ago
I've been looking for a solution to translate a dictionary for me. It is a Shipibo-Conibo (indigenous Peruvian language) to Spanish dictionary- I'd like to translate the Spanish to English (and leave the Shipibo intact). Curious for any thoughts here. I have the dictionary as a PDF (already searchable so I don't think it would need to be re-OCR'd...though that's possible too, it's not clearest scan).
- wrs 2y ago
  I wouldn’t be surprised to find that Claude/ChatGPT/etc. can just…do that. With the prompt you just gave. The output could be in Markdown, which is easily turned into a PDF. You would have to break up the input PDF into pages to avoid running out of output window.
  - rafram 2y ago
    I would!
  - temp0826 2y ago
    I didn't consider that would actually work and am giving it a try now...but by its own estimate it's going to take several days to finish (I'm not paying for plus or whatever).
- zzleeper 2y ago
  By any chance, would it be possible to share the PDF? I haven't heard shipibo language in a long while, and am quite curious about it.
  - temp0826 2y ago
    Here you go- https://archive.org/details/shipibodiccionario https://archive.org/details/shipibodiccionario
iLemming 2y ago
What's the fastest and accurate CLI OCR tool? My use case is simple - I want to be able to grab a piece of screen (Flameshot is great for that), and OCR it. I need this for note-taking during pair-programming over Zoom. Currently I'm using tesseract - it works, it's fast, but it also makes mistakes; it would be also great if it could discern tabular data and put them in ascii or markdown tables. I've tried docling, but it feels like a bit of an overkill. It seems to be slower - remember, I need to be able to grab the text from the screenshot very quickly. I have only tried default settings, maybe tweaking it would improve things. Can anyone share some thoughts on this? Thanks!
- acdha 2y ago
  Anything using the Apple Vision framework is fast and surprisingly accurate: https://github.com/bytefer/macos-vision-ocr https://github.com/bytefer/macos-vision-ocr
  - cdolan 2y ago
    Cool to see, may use this locally for OCR in some cases. But I think the "handwriting" example is a little misleading. Thats a font, not a scan of hand written material
  - wahnfrieden 2y ago
    This uses the old APIs that are less accurate than the new Swift-only LiveText ones
- ANighRaisin 2y ago
  The AI OCR build into snipping tool in windows is better than tesseract, albeit more inconvenient than something like powertoys or Capture2Text, which use a quick shortcut.
ritvikpandey21 2y ago
hey -- wrote a blog post about this exact phenomena [1] (also posted on HN couple weeks back [2]). tldr: maintaining confidence from LLM nondeterministic outputs over millions of pages is a problem. especially in production environments like healthcare, finance, etc. we've noticed decently high hallucination rates, even in finetuned LLMs. [1]: https://www.runpulse.com/blog/why-llms-suck-at-ocr https://www.runpulse.com/blog/why-llms-suck-at-ocr [2]: https://news.ycombinator.com/item?id=42966958#42977527 https://news.ycombinator.com/item?id=42966958#42977527
duckb 2y ago
Does this support table detection and extraction?
- fzysingularity 2y ago
  Yes, it's experimental at the moment: https://docs.vlm.run/guides/doc-ai/guide-visual-grounding https://docs.vlm.run/guides/doc-ai/guide-visual-grounding
Inviz 2y ago
Service doesnt inspire confidence. Openai-compatible api doesnt work (expects `content.str` in message to be a string - ???). Getting 500s on non-openai compatible endpoint - seems like timeouts(?). When it did work it missed a lot, and hallucinated a lot too on custom documents/schemas.
egorfine 2y ago
I had a need to scan serial numbers from Apple's product boxes out of pictures taken by a clueless person on their phone. All OCR tools failed. Vision model did the trick so well it's not even funny to discuss anything further. "This is a picture of Apple product box. Find and return only the serial number of the product as found on a label. Return 'none' if no serial number can be found".
- ptx 2y ago
  Did you check if all the numbers were correct?
  - egorfine 2y ago
    Of course. There was a little piece of code to query Apple for S/N data and it validated whether it was correct.
rendaw 2y ago
Why do all these OCR services only show examples with flawless screenshots of digital documents? Are there that many people trying to OCR digital data? Why not just copy the HTML? If it's not intended for digital documents, where are the screenshots with fold marks, slipping lines, lighting gradients, thumbs, etc etc.
leecarraher 2y ago
maybe it was my prompt, but there seems to be far too much interpretation after the image embedding. In my examples it implicitly started to summarize parts of the text, unfortunately incorrectly. On an invoice with typed lettering it summarized that payments submitted would not post for 2-3 business days, when in reality the text said if you submitted after 2p on a friday, the payment would not post until the following monday. Which is significantly different. I'd be curious if you could ablate those layers in some way, because the one-shot structured text detection recognition was much better than vanilla ocr.
cytocync 2y ago
[dead]
Cassyrobert 2y ago
[dead]
htrp 2y ago
VLM's can't replace ocr one to one.. most hosted multimodal models seem to have a classical OCR (tesseract-based) step in their inference loop
rasguanabana 2y ago
Wouldn’t VLM be susceptible to prompt injection?
wantlotsofcurry 2y ago
I'll definitely be trying this out on my current side project! Question: What tools/libs are people using to accurately detect square/rectangle objects in images? I've used VNDetectRectangle [1] in Swift but it's not as accurate as I'd like it to be, even with preprocessing. [1]: https://developer.apple.com/documentation/vision/vndetectrectanglesrequest https://developer.apple.com/documentation/vision/vndetectrec...

292 points by EarlyOom 2y ago

I wonder what the speed of this approach vs traditional ocr techniques. Also, curious if this could be used for text detection (find a bounding box containing text within an image).

vunderba 2y ago
Was just coming here to say this, there does not yet exist a multimodal vision LLM approach that is capable of identifying bounding boxes of where the text occurs. I suppose you could manually cut the image up and send each part separately to the LLM but that feels like an kludge and it's still in-exact.
- EarlyOom 2y ago
  We can do bounding boxes too :) we just call it visual grounding https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb https://github.com/vlm-run/vlmrun-cookbook/blob/main/noteboo...
  - vunderba 2y ago
    Wait what? That's pretty neat. I'm on my phone right now, so I can't really view the notebook very easily. How does this work? Are you using some kind of continual partitioning of the image and refeeding that back into the LLM to sort of pseudo-zoom in/out on the parts that contain non-cut off text until you can resolve that into rough coordinates?
  - deleted 2y ago
    [deleted]
  - what 2y ago
    Kind of skeptical since you also provide a “confidence” value, which has to be entirely made up. Do you have an example that isn’t a sample drivers license? Something that is unlikely to have appeared in an LLM’s training data?
- chpatrick 2y ago
  qwen 2.5 vl was specifically trained to produce bounding boxes I believe.

submeta 2y ago

Can I use this to convert flowcharts to yaml representations?

EarlyOom 2y ago
We convert to a JSON schema, but it would be trivial to convert this to yaml. There are some minor differences in e.g. tokens required to output JSON vs yaml which is why we've opted for our strategy.

orliesaurus 2y ago

I think OCR tools are good at what they say on the box, recognizing characters on a piece of paper etc. If I understand this right, the advantage of using a vision language model is the added logic that you can say things like: "Clearly this is a string, but does it look like a timestamp or something else?"

EarlyOom 2y ago
VLMs are able to take context into account when filling in fields, following either a global or field specific prompt. This is great for e.g. unlabeled axes, checking a legend for units to be suffixed after a number, etc. Also, you catch lots of really simple errors with type hints (e.g. dates, addresses, country codes etc.).
vintermann 2y ago
You can also use it for robustness. Looking at e.g. historical censuses, it's amazing how many ways people found to not follow the written instructions for filling them out. Often the information you want is still there, but woe to you if you look at the columns one by one and assume the information in them to be accurate and neatly within its bounding box.
raxxorraxor 2y ago
This has always been part of the complete OCR package as far as I know. The raw result of an OCR constantly fails to differentiate 1 l I i | or other similar symbols/letters. Maybe this necessary step can be improved and altered with a VLM. There is also the preprocessing where the image get its perspective corrected. Not sure how well a VLM performs here. As you said, I think combining these techniques will be the most efficient way forward.

ekidd 2y ago

I've been experimenting with vlm-run (plus custom form definitions), and it works surprisingly well with Gemini 2.0 Flash. Costs, as I understand, are also quite low for Gemini. You'll have best results with simple to medium-complexity forms, roughly the same ones you could ask a human to process with less than 10 minutes of training. If you need something like this, it's definitely good enough that you should consider kicking the tires.

fzysingularity 2y ago
Very cool! If you have more examples / schemas you'd be interested in sharing, feel free to add to the `contrib` section.
fzysingularity 2y ago
BTW Check out the Gemini qualitative results here in our hub: https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qualitative-results https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qu.... It gives you an idea of where today's models fail (Gemini Flash, OpenAI gpt4o+mini, open-source ones like Llama 3.2 Vision, Qwen VL 2.5 etc).

Eisenstein 2y ago

If you just want to play with using a vision model to do OCR, I made a little script that uses KoboldCpp to do it locally. * https://github.com/jabberjabberjabber/LLMOCR https://github.com/jabberjabberjabber/LLMOCR

LeoPanthera 2y ago

What's the characters-per-Wh of an LLM compared to traditional OCR?

fzysingularity 2y ago
That's a tough one to answer right now, but to be perfectly honest, we're off by 2-3 orders of magnitude in terms of chars/W. That said, VLMs are extremely powerful visual learners with LLM-like reasoning capabilities making them more versatile than OCR for practically all imaging domains. In a matter of a few years, I think we'll essentially see models that are more cost-performant via distillation, quantization and the multitude of tricks you can do to reduce the inference overhead.
mlyle 2y ago
A lot worse. But, higher quality OCR will reduce the amount of human post-processing needed, and, in turn will allow us to reduce the number of humans. Since humans are relatively expensive in energy use, this can be expected to save a lot of energy.
- rafram 2y ago
  > Since humans are relatively expensive in energy use Are they? I'm seeing figures around 80 watts at rest, and 150 when exercising. The brain itself only uses about 20 watts [1]. That's 1/35 of a single H100's power consumption (700 watts - which doesn't even take into account the energy required to cool the data center, the humans who build and maintain it, ...). [1]: https://www.humanbrainproject.eu/en/follow-hbp/news/2023/09/04/learning-brain-make-ai-more-energy-efficient/#:~:text=It%20is%20estimated%20that%20a%20human%20brain,hydroelectric%20plant%20if%20they%20were%20done%20artificially. https://www.humanbrainproject.eu/en/follow-hbp/news/2023/09/...
  - mlyle 2y ago
    The PUE of humans for that 80 watts is terrible, though. Ridiculous multiples of additional energy needed to convert solar power to a form of a energy that they can use, and even the manufacturing lifecycle and transport of humans to the datacenter is energy inefficient.

tgtweak 2y ago

Not really interested until this can run locally without api keys :\

EarlyOom 2y ago
You can! it works with Ollama https://github.com/vlm-run/vlmrun-hub https://github.com/vlm-run/vlmrun-hub At the end of the day its just schemas. You can decide for yourself if its work upgrading to a larger, more expensive model.

beebaween 2y ago

What's the best way to run this is I prefer to use local GPUs?

EarlyOom 2y ago
You can try out some of our schemas with Ollama if you want: https://github.com/vlm-run/vlmrun-hub https://github.com/vlm-run/vlmrun-hub (instructions in Readme)
fzysingularity 2y ago
We’re adding this as we speak. Ollama support is already there, and here’s vLLM inference: https://github.com/vlm-run/vlmrun-hub/pull/120 https://github.com/vlm-run/vlmrun-hub/pull/120

mmusson 2y ago

Lol. The resume includes expert in Mia Khalifa easter egg.

gunian 2y ago

replaced it with real humans -> nano tech in their brain -> transmit to server getting almost 99% accuracy

intalentive 2y ago

What's the value-add here? The schemas?

fzysingularity 2y ago
We've seen so many different schemas and ways of prompting the VLMs. We're just standardizing it here, and making it dead-simple to try it out across model providers.
vlmrunadmin007 2y ago
Basically there is no model schema combination. IF you go ahead and prompt a open source model with the schema it doesn't produce the results in the expected format. The main contribution is how to make these model conform to your specific needs and in a structured format.
- idiliv 2y ago
  Wait, but we're doing that already, and it works well (Qwen 2.5 VL)? If need be, you can always resort to structured generation to enforce schema conformity?

TZubiri 2y ago

Wow thanks! There's a client who had a startup idea that involved analyzing pdfs, I used textract, but it was too cumbersome and unreliable. Maybe I can reach out to see if he wants to give it anothee go with this!

fzysingularity 2y ago
Let us know, I think >70% of OCR tasks today can be done with VLMs with a little bit of guidance ;). Ping us at contact "at" vlm.run

deleted 2y ago

[deleted]

skbjml 2y ago

This is awesome!

rafram 2y ago

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sources, and it made up names and dates that sort of fit the vibe of the document when it couldn’t read the text! They sounded right for the ethnicity and time period but were entirely fake. There’s no way to ground the model using the source text when the model is your OCR.

EarlyOom 2y ago
This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.
- hashta 2y ago
  An effective way that usually increases accuracy is to use an ensemble of capable models that are trained independently (e.g., gemini, gpt-4o, qwen). If >x% of them have the same output, accept it, otherwise reject and manually review
  - rafram 2y ago
    There’s a very low chance that three separate models will come up with the same result. There are always going to be errors, small or large. Even if you find a way around that, running the process three times on every page is going to be prohibitively expensive, especially if you want to finetune.
    - vintermann 2y ago
      No, running it two or three times for every page isn't prohibitive. In fact, one of the arguments for using modern general-purpose multimodal models for historical HTR is that it is cheaper and faster than Transkribus. What you can do is for instance to ask one model for a transcription, and ask a second model to compare the transcription to the image and correct any errors it finds. You actually have a lot of budget to try things like these if the alternative is to fine-tune your own model.

syntaxing 2y ago

Maybe I’m being greedy but is it possible to have a vLLM detect when a portion is an image? I want to convert some handwritten notes into markdown but some portion are diagrams. I want the vLLM to extract the diagrams to embed into the markdown output

vlmrunadmin007 2y ago
We have successfully tested the model with vLLM and plan to release it across multiple inference server frameworks, including vLLM and OLAMA.

themanmaran 2y ago

We recently published an open source benchmark [1] specifically for evaluating VLM vs OCR. And generally the VLMs did much better than the traditional OCR models. VLM highlights: - Handwriting. Being contextually aware helps here. i.e. they read the document like a human would, interpreting the whole word/sentence instead of character by character - Charts/Infographics. VLMs can actually interpret charts or flow diagrams into a text format. Including things like color coded lines. Traditional OCR highlights: - Standardized documents (e.x. US tax forms that they've been trained on) - Dense text. Imagine textbooks and multi column research papers. This is the easiest OCR use case, but VLMS really struggle as the number of output tokens increase. - Bounding boxes. There still isn't really a model that gives super precise bounding boxes. Supposedly Gemini and Qwen were trained for it, but they don't perform as well as traditional models. There's still a ton of room for improvement, but especially with models like Gemini the accuracy/cost is really competitive. [1] https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark

fzysingularity 2y ago
Saw your benchmark, looks great. Will run our models against those benchmark and share some of our learnings. As you mentioned there are a few caveats to VLMs that folks are typically unaware of (not at all exhaustive, but the ones you highlighted): 1. Long-form text (dense): Token limits of 4/8K mean that dense pages may go over limits of the LLM outputs. This requires some careful work to make them work as seamlessly as OCR. 2. Visual grounding a.k.a. bounding boxes are definitely one of those things that VLMs aren't natively good at (partly because the cross-entropy losses used aren't really geared for bounding box regression). We're definitely making some strides here [1] to improve that so you're going to get an experience that is almost as good as native bounding box regression (all within the same VLM). [1] [1] https://colab.research.google.com/github/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb https://colab.research.google.com/github/vlm-run/vlmrun-cook...

BrannonKing 2y ago

What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixels are just noise).

NoMoreNicksLeft 2y ago
A big difficulty there is typeface detection, some of these were never digital fonts. But, even if it could detect them, you likely don't have those fonts on your computer to be able to put it back together as a digital typesetting for any but the most trivial fonts.
- retrorangular 2y ago
  The tool could include all known open source fonts, and for the rest, maybe could have a model recreate missing fonts for non-patented fonts, as while font files (.ttf, .otf, .woff, etc.) are copyrighted, styles usually do not have design patents, so tracing and re-creating them is usually not an issue as far as I'm aware (not a lawyer.) [1] Though if it accidentally "traces" one of the few exceptions, then you've potentially committed a crime, and the big difficulty in typeface detection you mention increases those odds. That said, there are so few exceptions that even if the model couldn't properly identify a font, it might be able to identify whether a font is likely to have a design patent. I do think getting an AI to create a high quality vector font from a potentially low-res raster graphic is going to be quite challenging though. Raster to vector tools I've tried in the past left a bit to be desired. 1. https://www.copyright.gov/comp3/chap900/ch900-visual-art.pdf https://www.copyright.gov/comp3/chap900/ch900-visual-art.pdf > As a general rule, typeface, typefont, lettering, calligraphy, and typographic ornamentation are not registrable. 37 C.F.R. § 202.1(a), (e). These elements are mere variations of uncopyrightable letters or words, which in turn are the building blocks of expression. See id. The Office typically refuses claims based on individual alphabetic or numbering characters, sets or fonts of related characters, fanciful lettering and calligraphy, or other forms of typeface. This is true regardless of how novel and creative the shape and form of the typeface characters may be. > There are some very limited cases where the Office may register some types of typeface, typefont, lettering, or calligraphy, such as the following: > • Pictorial or graphic elements that are incorporated into uncopyrightable characters or used to represent an entire letter or number may be registrable. Examples include original pictorial art that forms the entire body or shape of the typeface characters, such as a representation of an oak tree, a rose, or a giraffe that is depicted in the shape of a particular letter. > • Typeface ornamentation that is separable from the typeface characters is almost always an add-on to the beginning and/or ending of the characters. To the extent that such flourishes, swirls, vector ornaments, scrollwork, borders and frames, wreaths, and the like represent works of pictorial or graphic authorship in either their individual designs or patterned repetitions, they may be protected by copyright. However, the mere use of text effects (including chalk, popup papercraft, neon, beer glass, spooky-fog, and weathered-and-worn), while potentially separable, is de minimis and not sufficient to support a registration. > The Office may register a computer program that creates or uses certain typeface or typefont designs, but the registration covers only the source code that generates these designs, not the typeface, typefont, lettering, or calligraphy itself. For a general discussion of computer programs that generate typeface designs, see Chapter 700, Section 723.

fl0under 2y ago

Looks cool! May also be interested in Allen AI's OCR tool olmOCR they just released too [1][2]. They say "convert a million PDF pages for only $190 USD". [1] https://github.com/allenai/olmocr https://github.com/allenai/olmocr [2] https://arxiv.org/abs/2502.18443 https://arxiv.org/abs/2502.18443

TZubiri 2y ago
The issue with that promise is that anyone can convert pdfs, the question is whether the conversions are correct or whether you have Income Expenses 200 100 On one document, and Income Expenses 20 0100 On others. There's no shortage of products that tried to solve this problem from scratch (or by piggybacking on other projects) and called it a day without worrying about the huge problem that is quality and parseability. The most robust players just give you the coordinates of a glyph and you are on your own: Textract, PDFBox.

rasz 2y ago

I rather see machine learning used to help OCR by - recognizing/recreating exact font used - helping align/rotate source Not to hallucinate gibberish when source lacks enough data.

erulabs 2y ago

You sort of have to use both. OCR and LLM and then correlate the two results. They are bad at very different things, but a subsequent call to a 2nd LLM to pair together the results does improve quality significantly, plus you get both document understanding and context as well as bounding boxes, etc. I'm building a "never fill out paperwork again" app, if anyone is interested, would be happy to chat!

cpursley 2y ago
Any tips on how to prompt that second pairing step? And what sort of things to ask the llm to extract in step 1?
fzysingularity 2y ago
We think VLMs would outperform most OCR+LLM solutions in due time. I get that there’s need for these hybrid solutions today, but we’re comparing 20+ year mature tech vs something that’s roughly 1.5 years old. Also, VLMs are end-to-end trainable, unlike OCR+LLM solutions (that are trained separately), so it’s clear that these approaches scale much better for domain-specific use cases or verticals.
K0balt 2y ago
A VLM that invokes ocr tool use is a compelling idea that could result in pretty good results, I would expect.

cyp0633 2y ago

Existing solutions like Tesseract already can embed text into the image, but I'm wondering if there's a way to combine LLM with Tesseract, so that LLMs can help correcting results and finding unidentified text, and finally still embed text back to the image

serjester 2y ago

Good to see more work being done here, but I don't understand why this is tied to someone's proprietary API. Swapping model providers and adding some basic logging is not remotely painful enough to justify onboarding yet another vendor. Especially one that's handling something as sensitive as LLM prompts.

temp0826 2y ago

I've been looking for a solution to translate a dictionary for me. It is a Shipibo-Conibo (indigenous Peruvian language) to Spanish dictionary- I'd like to translate the Spanish to English (and leave the Shipibo intact). Curious for any thoughts here. I have the dictionary as a PDF (already searchable so I don't think it would need to be re-OCR'd...though that's possible too, it's not clearest scan).

wrs 2y ago
I wouldn’t be surprised to find that Claude/ChatGPT/etc. can just…do that. With the prompt you just gave. The output could be in Markdown, which is easily turned into a PDF. You would have to break up the input PDF into pages to avoid running out of output window.
- rafram 2y ago
  I would!
- temp0826 2y ago
  I didn't consider that would actually work and am giving it a try now...but by its own estimate it's going to take several days to finish (I'm not paying for plus or whatever).
zzleeper 2y ago
By any chance, would it be possible to share the PDF? I haven't heard shipibo language in a long while, and am quite curious about it.
- temp0826 2y ago
  Here you go- https://archive.org/details/shipibodiccionario https://archive.org/details/shipibodiccionario

iLemming 2y ago

What's the fastest and accurate CLI OCR tool? My use case is simple - I want to be able to grab a piece of screen (Flameshot is great for that), and OCR it. I need this for note-taking during pair-programming over Zoom. Currently I'm using tesseract - it works, it's fast, but it also makes mistakes; it would be also great if it could discern tabular data and put them in ascii or markdown tables. I've tried docling, but it feels like a bit of an overkill. It seems to be slower - remember, I need to be able to grab the text from the screenshot very quickly. I have only tried default settings, maybe tweaking it would improve things. Can anyone share some thoughts on this? Thanks!

acdha 2y ago
Anything using the Apple Vision framework is fast and surprisingly accurate: https://github.com/bytefer/macos-vision-ocr https://github.com/bytefer/macos-vision-ocr
- cdolan 2y ago
  Cool to see, may use this locally for OCR in some cases. But I think the "handwriting" example is a little misleading. Thats a font, not a scan of hand written material
- wahnfrieden 2y ago
  This uses the old APIs that are less accurate than the new Swift-only LiveText ones
ANighRaisin 2y ago
The AI OCR build into snipping tool in windows is better than tesseract, albeit more inconvenient than something like powertoys or Capture2Text, which use a quick shortcut.

ritvikpandey21 2y ago

hey -- wrote a blog post about this exact phenomena [1] (also posted on HN couple weeks back [2]). tldr: maintaining confidence from LLM nondeterministic outputs over millions of pages is a problem. especially in production environments like healthcare, finance, etc. we've noticed decently high hallucination rates, even in finetuned LLMs. [1]: https://www.runpulse.com/blog/why-llms-suck-at-ocr https://www.runpulse.com/blog/why-llms-suck-at-ocr [2]: https://news.ycombinator.com/item?id=42966958#42977527 https://news.ycombinator.com/item?id=42966958#42977527

duckb 2y ago

Does this support table detection and extraction?

fzysingularity 2y ago
Yes, it's experimental at the moment: https://docs.vlm.run/guides/doc-ai/guide-visual-grounding https://docs.vlm.run/guides/doc-ai/guide-visual-grounding

Inviz 2y ago

Service doesnt inspire confidence. Openai-compatible api doesnt work (expects `content.str` in message to be a string - ???). Getting 500s on non-openai compatible endpoint - seems like timeouts(?). When it did work it missed a lot, and hallucinated a lot too on custom documents/schemas.

egorfine 2y ago

I had a need to scan serial numbers from Apple's product boxes out of pictures taken by a clueless person on their phone. All OCR tools failed. Vision model did the trick so well it's not even funny to discuss anything further. "This is a picture of Apple product box. Find and return only the serial number of the product as found on a label. Return 'none' if no serial number can be found".

ptx 2y ago
Did you check if all the numbers were correct?
- egorfine 2y ago
  Of course. There was a little piece of code to query Apple for S/N data and it validated whether it was correct.

rendaw 2y ago

Why do all these OCR services only show examples with flawless screenshots of digital documents? Are there that many people trying to OCR digital data? Why not just copy the HTML? If it's not intended for digital documents, where are the screenshots with fold marks, slipping lines, lighting gradients, thumbs, etc etc.

leecarraher 2y ago

maybe it was my prompt, but there seems to be far too much interpretation after the image embedding. In my examples it implicitly started to summarize parts of the text, unfortunately incorrectly. On an invoice with typed lettering it summarized that payments submitted would not post for 2-3 business days, when in reality the text said if you submitted after 2p on a friday, the payment would not post until the following monday. Which is significantly different. I'd be curious if you could ablate those layers in some way, because the one-shot structured text detection recognition was much better than vanilla ocr.

cytocync 2y ago

[dead]

Cassyrobert 2y ago

[dead]

htrp 2y ago

VLM's can't replace ocr one to one.. most hosted multimodal models seem to have a classical OCR (tesseract-based) step in their inference loop

rasguanabana 2y ago

Wouldn’t VLM be susceptible to prompt injection?

wantlotsofcurry 2y ago

I'll definitely be trying this out on my current side project! Question: What tools/libs are people using to accurately detect square/rectangle objects in images? I've used VNDetectRectangle [1] in Swift but it's not as accurate as I'd like it to be, even with preprocessing. [1]: https://developer.apple.com/documentation/vision/vndetectrectanglesrequest https://developer.apple.com/documentation/vision/vndetectrec...