4 ms·
I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a l
by simonw 3y ago
I have to admit I've been having trouble figuring out what to do with bounding boxes of elements - I've got those out of Textract, but it feels like there's a lot of custom code needed to get from a bunch of bounding boxes to a useful JSON structure of the document.
That's why the idea of having an LLM like GPT-4 Vision or Gemini Pro or Claude process these things is so tempting - I want to be able to tell it what to do and get back JSON. And I can! It's just not reliable enough (yet?) to be particularly useful.
- larodi 3y agoHave you considered using a sort of SAM (segment anything) for the bounds and then OCR for the text and finally run it through LLM, which is a good predictor of text, to figure missing words or typos (wrong chars).
- simonw 3y agoOh that's an interesting idea! One challenge I've had with OCR is that things like multiple columns frequently confuse it - pulling out the regions first, OCRIng them independently and then using an LLM to try and piece everything back together could be a neat direction.