4 ms·

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appea

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a very hard problem for computers of any sort. I'm assuming you've given this some thought—how did you arrive at using an LLM to benchmark OCR vs other LLMs? What limitations with your benchmark have you seen/are you aware of? [0] https://github.com/VikParuchuri/marker/blob/master/benchmarks/overall/scorers/llm.py https://github.com/VikParuchuri/marker/blob/master/benchmark...

vikp 2y ago
Benchmarking is hard for markdown because of the slight formatting variations between different providers. With HTML, you can use something like TEDS (although there are issues with this, too), but with markdown, you don't have a great notion of structure, so you're left with edit distance. I think blockwise edit distance is better than full page (find the ground truth blocks, then infer each block separately and compare), but many providers only do well on full pages, which doesn't make it fair. There are a few different benchmark types in the marker repo: - Heuristic (edit distance by block with an ordering score) - LLM judging against a rubric - LLM win rate (compare two samples from different providers) None of these are perfect, but LLM against a rubric has matched visual inspection the best so far. I'll continue to iterate on the benchmarks. It may be possible to do a TEDS-like metric for markdown. Training a model on the output and then benchmarking could also be interesting, but it gets away from measuring pure extraction quality (the model benchmarking better is only somewhat correlated with better parse quality). I haven't seen any great benchmarking of markdown quality, even at research labs - it's an open problem.
arthurcolle 2y ago
You can use structured outputs, or something like my https://arthurcolle--dynamic-schema.modal.run/ https://arthurcolle--dynamic-schema.modal.run/ to extract real data from unstructured text (like that producted from an LLM) to make benchmarks slightly easier if you have a schema
- cdolan 2y ago
  What is the project? It just returns a vanilla html page saying: Dynamic Schema API API is running. See documentation for available endpoints.
  - arthurcolle 2y ago
    It's just a FastAPI app with endpoints that I developed and deployed before OpenAI released structured outputs that used a custom grammar to enforce a pydantic-like schema for Chain of Thought rollouts / structured data extraction from unstructured text. I also use it for a video transcription knowledge base generation API https://arthurcolle--dynamic-schema.modal.run/docs https://arthurcolle--dynamic-schema.modal.run/docs
themanmaran 2y ago
We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Every document has ground truth text, a JSON schema, and the ground truth JSON. - Run OCR on each document and pass the result to GPT-4o along with the JSON Schema - Compare the predicted JSON against the ground truth JSON for accuracy. In our benchmark, the ground truth text => gpt-4o was 99.7%+ accuracy. Meaning whenever gpt-4o was given the correct text, it could extract the structured JSON values ~100% of the time. So if we pass in the OCR text from Mistral and it scores 70%, that means the inaccuracies are isolated to OCR errors. https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark
- cdolan 2y ago
  were you guys able to finish running the benchmark with mistral and got a 70% score? Missed that Edit - I see it on the Benchmark page now. Woof, low 70% scores in some areas! https://getomni.ai/ocr-benchmark https://getomni.ai/ocr-benchmark
  - themanmaran 2y ago
    Yup, surprising results! We were able to dig in a bit more. Main culprit is the overzealous "image extraction". Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002). And it happened with a lot of full documents as well. Ex: most receipts got classified as images, and so it didn't extract any text.
    - lingjiekong 2y ago
      where do you find this regarding "Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002)."?
      - culi 2y ago
        themanmaran works at Omni so presumably they have access to the actual resulting data from this study

vikp 2y ago

Benchmarking is hard for markdown because of the slight formatting variations between different providers. With HTML, you can use something like TEDS (although there are issues with this, too), but with markdown, you don't have a great notion of structure, so you're left with edit distance. I think blockwise edit distance is better than full page (find the ground truth blocks, then infer each block separately and compare), but many providers only do well on full pages, which doesn't make it fair. There are a few different benchmark types in the marker repo: - Heuristic (edit distance by block with an ordering score) - LLM judging against a rubric - LLM win rate (compare two samples from different providers) None of these are perfect, but LLM against a rubric has matched visual inspection the best so far. I'll continue to iterate on the benchmarks. It may be possible to do a TEDS-like metric for markdown. Training a model on the output and then benchmarking could also be interesting, but it gets away from measuring pure extraction quality (the model benchmarking better is only somewhat correlated with better parse quality). I haven't seen any great benchmarking of markdown quality, even at research labs - it's an open problem.

arthurcolle 2y ago

You can use structured outputs, or something like my https://arthurcolle--dynamic-schema.modal.run/ https://arthurcolle--dynamic-schema.modal.run/ to extract real data from unstructured text (like that producted from an LLM) to make benchmarks slightly easier if you have a schema

cdolan 2y ago
What is the project? It just returns a vanilla html page saying: Dynamic Schema API API is running. See documentation for available endpoints.
- arthurcolle 2y ago
  It's just a FastAPI app with endpoints that I developed and deployed before OpenAI released structured outputs that used a custom grammar to enforce a pydantic-like schema for Chain of Thought rollouts / structured data extraction from unstructured text. I also use it for a video transcription knowledge base generation API https://arthurcolle--dynamic-schema.modal.run/docs https://arthurcolle--dynamic-schema.modal.run/docs

themanmaran 2y ago

We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Every document has ground truth text, a JSON schema, and the ground truth JSON. - Run OCR on each document and pass the result to GPT-4o along with the JSON Schema - Compare the predicted JSON against the ground truth JSON for accuracy. In our benchmark, the ground truth text => gpt-4o was 99.7%+ accuracy. Meaning whenever gpt-4o was given the correct text, it could extract the structured JSON values ~100% of the time. So if we pass in the OCR text from Mistral and it scores 70%, that means the inaccuracies are isolated to OCR errors. https://github.com/getomni-ai/benchmark https://github.com/getomni-ai/benchmark

cdolan 2y ago
were you guys able to finish running the benchmark with mistral and got a 70% score? Missed that Edit - I see it on the Benchmark page now. Woof, low 70% scores in some areas! https://getomni.ai/ocr-benchmark https://getomni.ai/ocr-benchmark
- themanmaran 2y ago
  Yup, surprising results! We were able to dig in a bit more. Main culprit is the overzealous "image extraction". Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002). And it happened with a lot of full documents as well. Ex: most receipts got classified as images, and so it didn't extract any text.
  - lingjiekong 2y ago
    where do you find this regarding "Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002)."?
    - culi 2y ago
      themanmaran works at Omni so presumably they have access to the actual resulting data from this study