5 ms·

> you might get some skipped bits, but that's not a dealbreaker for folks looking to feed PDFs into AI systems Unless it is. We have a few hundred PDF per mont

> you might get some skipped bits, but that's not a dealbreaker for folks looking to feed PDFs into AI systems Unless it is. We have a few hundred PDF per month (mostly tables) where we need 100% accuracy. Currently we feed them into an OCR and have humans check the result. I do not win anything if I have to check the LLM output, too.

llm_trw 2y ago
I'm currently solving this problem for work and thinking of a spin out, what's a ballpark figure you'd be willing to pay per 1000 pages for 99.999% character level accuracy?
- Lerc 2y ago
  I guess it depends on the use case, but if it surpasses the error rate that exists in the source document then it would be difficult to argue against. Specific things like evidentiary use would want 100% but that's at a level where any document processing would be suspect. What is the the typical range for error rate in PDF generation in various fields? Even robust technical documents have the occasional typo.
  - llm_trw 2y ago
    I'm not using generative models to fill in details not present in the original document. If there's a typo there then there will be a typo in the transcript. If you want to fix that then you can run another model on top of it.
    - Lerc 2y ago
      I realise that. The point is that a user is implicitly committing to the baseline error rate that exists in whatever means by which the document was created. If any additional loss was insignificant in proportion to that error rate then it would be unreasonable to reject it on that basis.
    - yigitkonur35 2y ago
      You're right. For my API that prepares PDFs for LLMs, fixing typos makes sense. But yeah, keeping original text is crucial for most OCR tasks.
- rescbr 2y ago
  At least for my use case, which is Layout processing (i.e. must output tables in some kind of table format), the OCR part (Azure Document AI or AWS Textract) dominates the cost factor. Running OCR on a document is twice more expensive than processing the output on the most expensive GPT offering. Intuitively, this was kind of unexpected for me. Only when I did some calculations on Excel that I realized it. If you’re able to halve the pricing for Layout output then you’re unblocking lots of use cases out there.
- Propelloni 2y ago
  > I'm currently solving this problem for work and thinking of a spin out, what's a ballpark figure you'd be willing to pay per 1000 pages for 99.999% character level accuracy? I guess anything up to 5 ¢ per page would be acceptable. But I'm afraid my company wouldn't be a customer. We are in Germany and we deal with particularly protected private data, there is no chance that we would exfiltrate this data to a cloud service.
  - llm_trw 2y ago
    What's the total spend per quarter? For a margin that fat I'd be willing to jump through a lot of hoops if you're doing enough pages. The models (currently) fit in 24gb vram sequentially with small enough batch sizes, so a local server with consumer grade gpus wouldn't be impossible.
    - Propelloni 2y ago
      I'll check and get back to you. How can I reach you?
      - llm_trw 2y ago
        Email at omni_vision_ai@proton.me

llm_trw 2y ago

I'm currently solving this problem for work and thinking of a spin out, what's a ballpark figure you'd be willing to pay per 1000 pages for 99.999% character level accuracy?

Lerc 2y ago
I guess it depends on the use case, but if it surpasses the error rate that exists in the source document then it would be difficult to argue against. Specific things like evidentiary use would want 100% but that's at a level where any document processing would be suspect. What is the the typical range for error rate in PDF generation in various fields? Even robust technical documents have the occasional typo.
- llm_trw 2y ago
  I'm not using generative models to fill in details not present in the original document. If there's a typo there then there will be a typo in the transcript. If you want to fix that then you can run another model on top of it.
  - Lerc 2y ago
    I realise that. The point is that a user is implicitly committing to the baseline error rate that exists in whatever means by which the document was created. If any additional loss was insignificant in proportion to that error rate then it would be unreasonable to reject it on that basis.
  - yigitkonur35 2y ago
    You're right. For my API that prepares PDFs for LLMs, fixing typos makes sense. But yeah, keeping original text is crucial for most OCR tasks.
rescbr 2y ago
At least for my use case, which is Layout processing (i.e. must output tables in some kind of table format), the OCR part (Azure Document AI or AWS Textract) dominates the cost factor. Running OCR on a document is twice more expensive than processing the output on the most expensive GPT offering. Intuitively, this was kind of unexpected for me. Only when I did some calculations on Excel that I realized it. If you’re able to halve the pricing for Layout output then you’re unblocking lots of use cases out there.
Propelloni 2y ago
> I'm currently solving this problem for work and thinking of a spin out, what's a ballpark figure you'd be willing to pay per 1000 pages for 99.999% character level accuracy? I guess anything up to 5 ¢ per page would be acceptable. But I'm afraid my company wouldn't be a customer. We are in Germany and we deal with particularly protected private data, there is no chance that we would exfiltrate this data to a cloud service.
- llm_trw 2y ago
  What's the total spend per quarter? For a margin that fat I'd be willing to jump through a lot of hoops if you're doing enough pages. The models (currently) fit in 24gb vram sequentially with small enough batch sizes, so a local server with consumer grade gpus wouldn't be impossible.
  - Propelloni 2y ago
    I'll check and get back to you. How can I reach you?
    - llm_trw 2y ago
      Email at omni_vision_ai@proton.me