3 ms·

From a tweet: https://x.com/i/status/2001821298109120856 https://x.com/i/status/2001821298109120856 > can someone help folks at Mistral find more weak baseline

by Tiberium 10mo ago

From a tweet: https://x.com/i/status/2001821298109120856 https://x.com/i/status/2001821298109120856 > can someone help folks at Mistral find more weak baselines to add here? since they can't stomach comparing with SoTA.... > (in case y'all wanna fix it: Chandra, dots.ocr, olmOCR, MinerU, Monkey OCR, and PaddleOCR are a good start)

belval 10mo ago
I've worked on document extraction a lot and while the tweet is too flippant for my taste, it's not wrong. Mistral is comparing itself to non-VLM computer vision services. While not necessarily what everyone needs, they are a very different beasts compared to VLM based extraction because it gives you precise bounding boxes, usually at the cost of larger "document understanding". Its failure mode are also vastly different. VLM-based extraction can misread entire sentences or miss entire paragraphs. Sonnet 3 had that issue. Computer vision models instead will make in-word typos.
- wills_forward 10mo ago
  Why not use both? I just built a pipeline for document data extraction that uses PaddleOCR, then Gemini 3 to check + fix errors. It gets close to 99.9% on extraction from financial statements finally on par with humans.
  - jadbox 10mo ago
    This is The Way. Remember AI doesn't have to replace existing solutions but can tactfully supplement it.
  - vrc 10mo ago
    I did the opposite. Tesseract to get bboxes, words, and chars and then mistral on the clips with some reasonable reflow to preserve geometry. Paddle wasn’t working on my local machine (until I found RapidOCR). Surya was also very good but because you can’t really tweak any knobs, when it failed it just kinda failed. But Surya > Rapid w/ Paddle > DocTr > Tesseract while the latter gave me the most granularity when I needed it. Edit: Gemini 2.0 was good enough for VLM cleanup, and now 2.5 or above with structured output make reconstruction even easier.
- zerocrates 10mo ago
  Is DeepSeek's not VLM?
logicprog 10mo ago
I'd want to see a comparison with Qwen 3 VL 235B-A22B, which is IME significantly better than MinerU.
vinckr 10mo ago
after clicking on your link I browsed twitter for a minute and damn that place has become weird (or maybe it always was?)
- crystal_revenge 10mo ago
  As someone who has been on Twitter since 2007, it’s radically changed in the last few years to the point of being unrecognizable.
ozgune 10mo ago
Also, do you know if their benchmarks are available? In their website, the benchmarks say “Multilingual (Chinese), Multilingual (East-asian), Multilingual (Eastern europe), Multilingual (English), Multilingual (Western europe), Forms, Handwritten, etc.” However, there’s no reference to the benchmark data.
nerbert 10mo ago
On the OP link, they compare themselves to the capabilities of leaderboard AI's and beat them.

belval 10mo ago

I've worked on document extraction a lot and while the tweet is too flippant for my taste, it's not wrong. Mistral is comparing itself to non-VLM computer vision services. While not necessarily what everyone needs, they are a very different beasts compared to VLM based extraction because it gives you precise bounding boxes, usually at the cost of larger "document understanding". Its failure mode are also vastly different. VLM-based extraction can misread entire sentences or miss entire paragraphs. Sonnet 3 had that issue. Computer vision models instead will make in-word typos.

wills_forward 10mo ago
Why not use both? I just built a pipeline for document data extraction that uses PaddleOCR, then Gemini 3 to check + fix errors. It gets close to 99.9% on extraction from financial statements finally on par with humans.
- jadbox 10mo ago
  This is The Way. Remember AI doesn't have to replace existing solutions but can tactfully supplement it.
- vrc 10mo ago
  I did the opposite. Tesseract to get bboxes, words, and chars and then mistral on the clips with some reasonable reflow to preserve geometry. Paddle wasn’t working on my local machine (until I found RapidOCR). Surya was also very good but because you can’t really tweak any knobs, when it failed it just kinda failed. But Surya > Rapid w/ Paddle > DocTr > Tesseract while the latter gave me the most granularity when I needed it. Edit: Gemini 2.0 was good enough for VLM cleanup, and now 2.5 or above with structured output make reconstruction even easier.
zerocrates 10mo ago
Is DeepSeek's not VLM?

logicprog 10mo ago

I'd want to see a comparison with Qwen 3 VL 235B-A22B, which is IME significantly better than MinerU.

vinckr 10mo ago

after clicking on your link I browsed twitter for a minute and damn that place has become weird (or maybe it always was?)

crystal_revenge 10mo ago
As someone who has been on Twitter since 2007, it’s radically changed in the last few years to the point of being unrecognizable.

ozgune 10mo ago

Also, do you know if their benchmarks are available? In their website, the benchmarks say “Multilingual (Chinese), Multilingual (East-asian), Multilingual (Eastern europe), Multilingual (English), Multilingual (Western europe), Forms, Handwritten, etc.” However, there’s no reference to the benchmark data.

nerbert 10mo ago

On the OP link, they compare themselves to the capabilities of leaderboard AI's and beat them.