4 ms·

It's shocking how much our industry fails to see past its own nose. Not a single example on that page is a Purchase Order, Invoice etc. Not a single example sh

by kiratp 2y ago

It's shocking how much our industry fails to see past its own nose. Not a single example on that page is a Purchase Order, Invoice etc. Not a single example shown is relevant to industry at scale.

kashnote 2y ago
Fwiw, they have an example of a parking receipt in a cookbook: https://colab.research.google.com/github/mistralai/cookbook/blob/main/mistral/ocr/structured_ocr.ipynb https://colab.research.google.com/github/mistralai/cookbook/...
guiomie 2y ago
Agreed. In general I've had such bad performance for complex table based invoice parsing, that every few months I try the latest models to see if its better. It does say "96.12" on top-tier benchmark under the Table category.
mtillman 2y ago
We find CV models to be better (higher midpoint on an ROC curve) for the types of docs you mention.
simpaticoder 2y ago
Another good example would be contracts of any kind. Imagine photographing a contract (like a car loan) and on the spot getting an AI to read it, understand it, forecast scenarious, highlight red flags, and do some comparison shopping for you.
- JBiserkov 2y ago
  ... imagining ... ... hallucinating during read ... ... hallucinating during understand ... ... hallucinating during forecast ... ... highlighting a hallucination as red flag ... ... missing an actual red flag ... ... consuming water to cool myself... Phew, being an AI is hard!
  - simpaticoder 2y ago
    Your points are well-taken, but I think that contracts are a small enough, and well represented enough in the corpus, to actually be pretty solid. This is especially true with good prompting and some sort of feedback loop.
arpinum 2y ago
Businesses at scale use EDI to handle purchase orders and invoices, no OCR needed.
- cdolan 2y ago
  Thats simply not a factual statement. Scaled businesses do USE edi, but they still receive hundreds of thousands of PDF documents a month source: built a saas product that handles pdfs for a specific industry
merb 2y ago
Mistral is Europe based where invoices are more or less sent digitally in like 95% of all the cases anyway. Some are even digital invoices, which will at some point in the eu be mandatory. For orders there are proposals for that, too. And basically invoice data extraction is a different beast.
- wolfi1 2y ago
  even in Europe this is still a thing, I know of systems which still are unable to read items having more than one line (costing s sh*tload of money)
- napolux 2y ago
  Can confirm, in Italy electronic invoicing is mandatory since 2019
- revnode 2y ago
  So an invoice attached to an email as a PDF is sent digitally ... those unfamiliar with PDF will think text and data extraction is trivial then, but this isn't true. You can have a fully digital, non-image PDF that is vector based and has what looks like text, but doesn't have a single piece of extractable text in it. It's all about how the PDF was generated. Tables can be formatted in a million ways, etc. Your best bet is to always convert it to an image and OCR it to extract structured data.
  - merb 2y ago
    This is simply not true. Maybe it’s easier and you do not need 100% precision. But it is actually possible to extract text and layout of digital pdfs. Else it would be impossible to display it. Of course some people still add image fragments to a pdf, but that practice is basically dying. I did not see a single pdf the last year we‘re it was impossible to extract the layout.
- codetrotter 2y ago
  One use-case is digitising receipts from business related travels for expenses that employees paid for out of their own pocket and which they are submitting pictures to the business for reimbursement. Bus travels, meals including dinners and snacks, etc. for which the employee has receipts on paper.
deleted 2y ago
[deleted]
mentalgear 2y ago
To be fair: Reading the blog post, the main objective seems to have been to enable information extraction with high confidence for the academic sector (e.g. unlocking all these paper pdfs), and not necessarily to be another receipt scanner.
- kiratp 2y ago
  It hilarious that the academic sector 1. publishes as PDF 2. spends all this energy on how to extract that info back from PDF 3. publishes that research as PDF as well. Receipt scanning is a multiple orders of magnitude more valuable business. Mistral at this point is looking for a commercial niche (like how Claude is aiming at software development)
sha16 2y ago
I wanted to apply OCR to my company's invoicing since they basically did purchasing for a bunch of other large companies, but the variability in the conversion was not tolerable. Even rounding something differently could catch an accountant's eye, let alone detecting a "8" as a "0" or worse.
dotnetkow 2y ago
Agreed, though in this case, they are going for general-purpose OCR. That's fine in some cases, but purpose-built models trained on receipts, invoices, tax documents, etc., definitely perform better. We've got a similar API solution coming out soon (https://digital.abbyy.com/code-extract-automate-your-new-must-have-ocr-api-coming-soon https://digital.abbyy.com/code-extract-automate-your-new-mus...) that should work better for businesses automating their docs at scale.

by kiratp 2y ago

It's shocking how much our industry fails to see past its own nose. Not a single example on that page is a Purchase Order, Invoice etc. Not a single example shown is relevant to industry at scale.

kashnote 2y ago

Fwiw, they have an example of a parking receipt in a cookbook: https://colab.research.google.com/github/mistralai/cookbook/blob/main/mistral/ocr/structured_ocr.ipynb https://colab.research.google.com/github/mistralai/cookbook/...

guiomie 2y ago

Agreed. In general I've had such bad performance for complex table based invoice parsing, that every few months I try the latest models to see if its better. It does say "96.12" on top-tier benchmark under the Table category.

mtillman 2y ago

We find CV models to be better (higher midpoint on an ROC curve) for the types of docs you mention.

simpaticoder 2y ago

Another good example would be contracts of any kind. Imagine photographing a contract (like a car loan) and on the spot getting an AI to read it, understand it, forecast scenarious, highlight red flags, and do some comparison shopping for you.

JBiserkov 2y ago
... imagining ... ... hallucinating during read ... ... hallucinating during understand ... ... hallucinating during forecast ... ... highlighting a hallucination as red flag ... ... missing an actual red flag ... ... consuming water to cool myself... Phew, being an AI is hard!
- simpaticoder 2y ago
  Your points are well-taken, but I think that contracts are a small enough, and well represented enough in the corpus, to actually be pretty solid. This is especially true with good prompting and some sort of feedback loop.

arpinum 2y ago

Businesses at scale use EDI to handle purchase orders and invoices, no OCR needed.

cdolan 2y ago
Thats simply not a factual statement. Scaled businesses do USE edi, but they still receive hundreds of thousands of PDF documents a month source: built a saas product that handles pdfs for a specific industry

merb 2y ago

Mistral is Europe based where invoices are more or less sent digitally in like 95% of all the cases anyway. Some are even digital invoices, which will at some point in the eu be mandatory. For orders there are proposals for that, too. And basically invoice data extraction is a different beast.

wolfi1 2y ago
even in Europe this is still a thing, I know of systems which still are unable to read items having more than one line (costing s sh*tload of money)
napolux 2y ago
Can confirm, in Italy electronic invoicing is mandatory since 2019
revnode 2y ago
So an invoice attached to an email as a PDF is sent digitally ... those unfamiliar with PDF will think text and data extraction is trivial then, but this isn't true. You can have a fully digital, non-image PDF that is vector based and has what looks like text, but doesn't have a single piece of extractable text in it. It's all about how the PDF was generated. Tables can be formatted in a million ways, etc. Your best bet is to always convert it to an image and OCR it to extract structured data.
- merb 2y ago
  This is simply not true. Maybe it’s easier and you do not need 100% precision. But it is actually possible to extract text and layout of digital pdfs. Else it would be impossible to display it. Of course some people still add image fragments to a pdf, but that practice is basically dying. I did not see a single pdf the last year we‘re it was impossible to extract the layout.
codetrotter 2y ago
One use-case is digitising receipts from business related travels for expenses that employees paid for out of their own pocket and which they are submitting pictures to the business for reimbursement. Bus travels, meals including dinners and snacks, etc. for which the employee has receipts on paper.

deleted 2y ago

[deleted]

mentalgear 2y ago

To be fair: Reading the blog post, the main objective seems to have been to enable information extraction with high confidence for the academic sector (e.g. unlocking all these paper pdfs), and not necessarily to be another receipt scanner.

kiratp 2y ago
It hilarious that the academic sector 1. publishes as PDF 2. spends all this energy on how to extract that info back from PDF 3. publishes that research as PDF as well. Receipt scanning is a multiple orders of magnitude more valuable business. Mistral at this point is looking for a commercial niche (like how Claude is aiming at software development)

sha16 2y ago

I wanted to apply OCR to my company's invoicing since they basically did purchasing for a bunch of other large companies, but the variability in the conversion was not tolerable. Even rounding something differently could catch an accountant's eye, let alone detecting a "8" as a "0" or worse.

dotnetkow 2y ago

Agreed, though in this case, they are going for general-purpose OCR. That's fine in some cases, but purpose-built models trained on receipts, invoices, tax documents, etc., definitely perform better. We've got a similar API solution coming out soon (https://digital.abbyy.com/code-extract-automate-your-new-must-have-ocr-api-coming-soon https://digital.abbyy.com/code-extract-automate-your-new-mus...) that should work better for businesses automating their docs at scale.