6 ms·

> And 6 months from now it'll be fast, cheap, and probably more reliable! I like the optimism. I've needed to include human review when using previous generat

> And 6 months from now it'll be fast, cheap, and probably more reliable! I like the optimism. I've needed to include human review when using previous generation OCR software; when I needed the results to be accurate. It's painstaking, but the OCR offered a speedup over fully-manual transcription. Have you given any thought to human-in-the-loop processes?

themanmaran 2y ago
I've been surprised so far by llms capability, so I hope it continues. On the human in loop side, it's really use case specific. For a lot of my company's work, it's focused on getting trends from large sets of documents. Ex: "categorize building permits by municipality". If the OCR was wrong on a few documents, it's still going to capture the general trend. If the use case was "pull bank account info from wire forms" I would want a lot more double checking. But that said, humans also have a tendency to transpose numbers incorrectly.
- 8organicbits 2y ago
  Hmm, sounds like different goals. I don't work on that project any longer but it was a very small set of documents and they needed to be transcribed perfectly. Every typo in the original needed to be preserved. That said, there's huge value in lossy transcription elsewhere, as long as you can account for the errors they introduce.
- raisedbyninjas 2y ago
  Our human in the loop process with traditional OCR uses confidence scores from regions of interest and the page coordinates to speed-up the review process. I wish the LLM could provide that, but both seem far off on the horizon.
throwthrowuknow 2y ago
Have you tried using the GraphRAG approach of just rerunning the same prompts multiple times and then giving the results along with a prompt to the model telling it to extract the true text and fix any mistakes? With mini this seems like a very workable solution. You could even incorporate one or more attempts from whatever OCR you were using previously. I think that is one of the key findings from GraphRAG paper: the gpt can replace the human in the loop.

themanmaran 2y ago

I've been surprised so far by llms capability, so I hope it continues. On the human in loop side, it's really use case specific. For a lot of my company's work, it's focused on getting trends from large sets of documents. Ex: "categorize building permits by municipality". If the OCR was wrong on a few documents, it's still going to capture the general trend. If the use case was "pull bank account info from wire forms" I would want a lot more double checking. But that said, humans also have a tendency to transpose numbers incorrectly.

8organicbits 2y ago
Hmm, sounds like different goals. I don't work on that project any longer but it was a very small set of documents and they needed to be transcribed perfectly. Every typo in the original needed to be preserved. That said, there's huge value in lossy transcription elsewhere, as long as you can account for the errors they introduce.
raisedbyninjas 2y ago
Our human in the loop process with traditional OCR uses confidence scores from regions of interest and the page coordinates to speed-up the review process. I wish the LLM could provide that, but both seem far off on the horizon.

throwthrowuknow 2y ago

Have you tried using the GraphRAG approach of just rerunning the same prompts multiple times and then giving the results along with a prompt to the model telling it to extract the true text and fix any mistakes? With mini this seems like a very workable solution. You could even incorporate one or more attempts from whatever OCR you were using previously. I think that is one of the key findings from GraphRAG paper: the gpt can replace the human in the loop.