5 ms·

Run structured extraction on documents/images locally with Ollama and Pydantic

170 points by EarlyOom 2y ago

Inviz 2y ago
What are the most promising ways to extract information from picture like this, if the domain has strict time constraints? What's the second best way that is still fast?
- fzysingularity 2y ago
  You can always distill VLMs into much smaller / faster models that’s specific to your domain or use-case. What’s the use-case and what kind of latency do you require?
EarlyOom 2y ago
We put together an open-source collection of Pydantic schemas for a variety of document categories (W2 filings, invoices etc.), including instructions for how to get structured JSON responses from any visual input with the model of your choosing. Run everything locally.
jbmsf 2y ago
Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.
- jgalt212 2y ago
  Our customers insist we run everything on their docs locally.
  - fzysingularity 2y ago
    Absolutely, we’ve been hearing the same from our customers - which is why we thought it makes sense to open source a bunch of schemas so that they’re reusable and compatible across various inference providers (esp. Ollama/local ones).
- fzysingularity 2y ago
  Cool, what types of documents do you currently handle? We could share some of our learnings/schemas here too.
  - andrewinardeer 2y ago
    Different commenter; Here I'm extracting data from commerical invoices, POs and bills of lading.
    - fzysingularity 2y ago
      Ah cool, care to share a few examples? We can probably add those schemas in the next few days if there's enough folks who could benefit from this. A basic invoice schema is already there: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/schemas/document/invoice.py https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/s... You can see some of the qualitative results on GPT4o, Gemini, Llama 3.2 11B, Phi-4 here: https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qualitative-results https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qu...
  - jbmsf 2y ago
    Mostly tax forms, state-specific formations documents (Articles of X), and state-specific payroll registration documents.
jasonjmcghee 2y ago
I've used "structured output" (with supplied schema) on Google and openai, and function calling / tool use on those, anthropic and others- and afaict they are functionally the same (if you force a specific function / schema). Has someone had a different experience?
- fzysingularity 2y ago
  They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc). So you end up hitting roadblocks for seemingly simple Pydantic schemas.
  - jasonjmcghee 2y ago
    I meant between "structured output" and "function calling". Afaict one is outputting according to a schema and the other is outputting according to a schema... which will be used as the parameters to a function. But they seem to be considered disparate concepts. So I'm trying to understand if there's some additional nuance I'm missing.
    - guntars 2y ago
      With function calls the model may or may not output something that matches the schema, with structured output the schema is enforced at the logit level.
      - jasonjmcghee 2y ago
        At least in the case of openai, you can set "strict" to "true" and function calling / tool use must / is enforced to follow the schema too.
    - fzysingularity 2y ago
      Ah ok, I misunderstood. As far as I've seen, structured outputs is essentially "json-mode" with some constraints (i.e. guided decoding over a known schema) - so the model effectively emits valid JSON that conforms to the schema. In function calling, the model is asked to emit "code" that conforms to some function parameter spec. You could use json-mode for function-calling, but probably not the other way around. I've generally found json-mode to be more useful than function-calling, even though the latter is what everyone fixates on because of it's obvious use in agents.
kaushikbokka 2y ago
Have you folks tried finetuning models for data extraction from visual data?
- EarlyOom 2y ago
  That's one of our main focuses, yes: https://docs.vlm.run/api-reference/v1/fine-tuning/post-finetuning-create#create-finetuning-job https://docs.vlm.run/api-reference/v1/fine-tuning/post-finet...
jauntywundrkind 2y ago
I'd really like to play with Qwen2.5-VL at some point, perhaps for reading data-sheets for microchips. Nicely for some applications, it's also very good at reporting position of what it finds, which many ML tools are pretty mediocre at. https://qwenlm.github.io/blog/qwen2.5-vl/ https://qwenlm.github.io/blog/qwen2.5-vl/ Not really this application, but QvQ for visual reasoning is also impressive. https://qwenlm.github.io/blog/qvq-72b-preview/ https://qwenlm.github.io/blog/qvq-72b-preview/ Meta has used Qwen as the basis for their Apollo research. https://arxiv.org/abs/2412.10360 https://arxiv.org/abs/2412.10360
- fzysingularity 2y ago
  Is Qwen2.5-VL on Ollama? Could give it a try with a few of the schemas we have. We’ve locally tested with Llama 3.2 11B Vision on Ollama: https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchmarks/2025-01-10-llama3.2-vision:11b-instructor-results.md https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchm... FWIW I think Ollama structured outputs API is quite buggy compared to the HF transformers variant.
  - fzysingularity 2y ago
    Just ran them for Qwen2.5-VL: https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchmarks/2025-02-20-bsahane-Qwen2.5-VL-7B-Instruct-Q4_K_M_benxh-ollama-results.md https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchm...
youknowwhentous 2y ago
This seems to work for videos as well. Pretty cool demo and very nice interface for the pydantic types.
- fzysingularity 2y ago
  Yes, good catch. We'll be adding several more schemas for videos in the next few weeks. A few video schemas are already added to the main catalog: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/catalog.yaml https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/c...
18chetanpatel 2y ago
This is something I was searching for..Thanks for creating!
joatmon-snoo 2y ago
Super cool! We at BAML had been thinking about doing something like this for our ecosystem as well - we’d love to add BAML models to this repo! If you haven’t heard of us, we provide a language and runtime that enable defining your schemas in a simpler syntax, and allow usage with _any_ model, not just those that implement tool calling or json mode, by by relying on schema-aligned parsing. Check it out! https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml
- EarlyOom 2y ago
  Would love to chat! reach out scott@vlm.run
peterhadlaw 2y ago
When making a new repo, reset your initial branch back to master with the following command: git config --global init.defaultBranch master There's the equivalent setting in GitHub.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

170 points by EarlyOom 2y ago

What are the most promising ways to extract information from picture like this, if the domain has strict time constraints? What's the second best way that is still fast?

fzysingularity 2y ago
You can always distill VLMs into much smaller / faster models that’s specific to your domain or use-case. What’s the use-case and what kind of latency do you require?

EarlyOom 2y ago

We put together an open-source collection of Pydantic schemas for a variety of document categories (W2 filings, invoices etc.), including instructions for how to get structured JSON responses from any visual input with the model of your choosing. Run everything locally.

jbmsf 2y ago

Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.

jgalt212 2y ago
Our customers insist we run everything on their docs locally.
- fzysingularity 2y ago
  Absolutely, we’ve been hearing the same from our customers - which is why we thought it makes sense to open source a bunch of schemas so that they’re reusable and compatible across various inference providers (esp. Ollama/local ones).
fzysingularity 2y ago
Cool, what types of documents do you currently handle? We could share some of our learnings/schemas here too.
- andrewinardeer 2y ago
  Different commenter; Here I'm extracting data from commerical invoices, POs and bills of lading.
  - fzysingularity 2y ago
    Ah cool, care to share a few examples? We can probably add those schemas in the next few days if there's enough folks who could benefit from this. A basic invoice schema is already there: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/schemas/document/invoice.py https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/s... You can see some of the qualitative results on GPT4o, Gemini, Llama 3.2 11B, Phi-4 here: https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qualitative-results https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qu...
- jbmsf 2y ago
  Mostly tax forms, state-specific formations documents (Articles of X), and state-specific payroll registration documents.

jasonjmcghee 2y ago

I've used "structured output" (with supplied schema) on Google and openai, and function calling / tool use on those, anthropic and others- and afaict they are functionally the same (if you force a specific function / schema). Has someone had a different experience?

fzysingularity 2y ago
They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc). So you end up hitting roadblocks for seemingly simple Pydantic schemas.
- jasonjmcghee 2y ago
  I meant between "structured output" and "function calling". Afaict one is outputting according to a schema and the other is outputting according to a schema... which will be used as the parameters to a function. But they seem to be considered disparate concepts. So I'm trying to understand if there's some additional nuance I'm missing.
  - guntars 2y ago
    With function calls the model may or may not output something that matches the schema, with structured output the schema is enforced at the logit level.
    - jasonjmcghee 2y ago
      At least in the case of openai, you can set "strict" to "true" and function calling / tool use must / is enforced to follow the schema too.
  - fzysingularity 2y ago
    Ah ok, I misunderstood. As far as I've seen, structured outputs is essentially "json-mode" with some constraints (i.e. guided decoding over a known schema) - so the model effectively emits valid JSON that conforms to the schema. In function calling, the model is asked to emit "code" that conforms to some function parameter spec. You could use json-mode for function-calling, but probably not the other way around. I've generally found json-mode to be more useful than function-calling, even though the latter is what everyone fixates on because of it's obvious use in agents.

kaushikbokka 2y ago

Have you folks tried finetuning models for data extraction from visual data?

EarlyOom 2y ago
That's one of our main focuses, yes: https://docs.vlm.run/api-reference/v1/fine-tuning/post-finetuning-create#create-finetuning-job https://docs.vlm.run/api-reference/v1/fine-tuning/post-finet...

jauntywundrkind 2y ago

I'd really like to play with Qwen2.5-VL at some point, perhaps for reading data-sheets for microchips. Nicely for some applications, it's also very good at reporting position of what it finds, which many ML tools are pretty mediocre at. https://qwenlm.github.io/blog/qwen2.5-vl/ https://qwenlm.github.io/blog/qwen2.5-vl/ Not really this application, but QvQ for visual reasoning is also impressive. https://qwenlm.github.io/blog/qvq-72b-preview/ https://qwenlm.github.io/blog/qvq-72b-preview/ Meta has used Qwen as the basis for their Apollo research. https://arxiv.org/abs/2412.10360 https://arxiv.org/abs/2412.10360

fzysingularity 2y ago
Is Qwen2.5-VL on Ollama? Could give it a try with a few of the schemas we have. We’ve locally tested with Llama 3.2 11B Vision on Ollama: https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchmarks/2025-01-10-llama3.2-vision:11b-instructor-results.md https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchm... FWIW I think Ollama structured outputs API is quite buggy compared to the HF transformers variant.
- fzysingularity 2y ago
  Just ran them for Qwen2.5-VL: https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchmarks/2025-02-20-bsahane-Qwen2.5-VL-7B-Instruct-Q4_K_M_benxh-ollama-results.md https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchm...

youknowwhentous 2y ago

This seems to work for videos as well. Pretty cool demo and very nice interface for the pydantic types.

fzysingularity 2y ago
Yes, good catch. We'll be adding several more schemas for videos in the next few weeks. A few video schemas are already added to the main catalog: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/catalog.yaml https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/c...

18chetanpatel 2y ago

This is something I was searching for..Thanks for creating!

joatmon-snoo 2y ago

Super cool! We at BAML had been thinking about doing something like this for our ecosystem as well - we’d love to add BAML models to this repo! If you haven’t heard of us, we provide a language and runtime that enable defining your schemas in a simpler syntax, and allow usage with _any_ model, not just those that implement tool calling or json mode, by by relying on schema-aligned parsing. Check it out! https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml

EarlyOom 2y ago
Would love to chat! reach out scott@vlm.run

peterhadlaw 2y ago

When making a new repo, reset your initial branch back to master with the following command: git config --global init.defaultBranch master There's the equivalent setting in GitHub.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]