5 ms·
> This means you can now use gpt-4-turbo vision to extract structured data from an image! How consistent and reliable is the extracted structure? Did they add
by MuffinFlavored 3y ago
> This means you can now use gpt-4-turbo vision to extract structured data from an image!
How consistent and reliable is the extracted structure?
Did they add any kind of concept "for whatever token you generate/think it is next, "unit test it" / make sure it passes some sort of rules?"
- maciejgryka 3y agoMy experience is that it's pretty good at reading text and pretty bad at understanding layouts. So e.g. asking it to work with tables is asking for trouble.
- dontupvoteme 3y agoYeah it's absolutely horrible at layouts. I'm not 100% sure it's related but if you ask it to draw bounding boxes around things it's always off by quite a bit.
- a_wild_dandan 3y agoYou restrict the model's next prediction to valid JSON tokens. (If you mean format reliability.)
- rockwotj 3y agoI'm waiting for being able to restrict to a specific JSON Schema
- brianjking 3y agoProviding examples generally helps me. Have you checked out the Instructor library too?
- jgalt212 3y ago> How consistent and reliable is the extracted structure? That's the $100B question.
- szundi 3y agoOr $T these days
- simonw 3y agoIt's pretty good, but it's not reliable enough to exclude the need to check everything it does. Same story as basically everything relating to LLMs to be honest.
- brianjking 3y ago^ This. I've found success generally better with Textract and then passing the OCR'ed text to OpenAI and using Pydantic to get it structured.
- _flux 3y agollama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs. https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
- simonw 3y agollama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.
- _flux 3y agoIndeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it. But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.
- joshstrange 3y agoIn my testing I was better off running the image through AWS Textract then taking the output and feeding it to OpenAI. It was also much cheaper. Of course if all you are looking for is extraction then maybe you don't need OpenAI at all. I used it to clean up the OCR'd data and reformat it.
- Lucasoato 3y agoIt's very consistent, check this guy, he was able to structure LLM output using Pydantic in an elegant solution: https://www.youtube.com/watch?v=yj-wSRJwrrc https://www.youtube.com/watch?v=yj-wSRJwrrc
- mahithefish 3y agoEven if the GitHub link they just posted above. It made up the year at 2022 and hallucinated the start and end time. https://github.com/datasette/datasette-extract/issues/19 https://github.com/datasette/datasette-extract/issues/19