4 ms·
They also added both JSON and function support to the vision model - previously it didn't have those. This means you can now use gpt-4-turbo vision to extract
by simonw 3y ago
They also added both JSON and function support to the vision model - previously it didn't have those.
This means you can now use gpt-4-turbo vision to extract structured data from an image!
I was previously using a nasty hack where I'd run the image through the vision model to extract just the text, then run that text through regular gpt-4-turbo to extract structured data. I ditched that hack just now: https://github.com/datasette/datasette-extract/issues/19 https://github.com/datasette/datasette-extract/issues/19
- maciejgryka 3y agoHe, cool to hear you're doing something like this too! We ended up close, but we also need good spatial relationships, which GPT4V isn't great at, so we're using other OCR system and adding the result to the context.
- MuffinFlavored 3y ago> This means you can now use gpt-4-turbo vision to extract structured data from an image! How consistent and reliable is the extracted structure? Did they add any kind of concept "for whatever token you generate/think it is next, "unit test it" / make sure it passes some sort of rules?"
- maciejgryka 3y agoMy experience is that it's pretty good at reading text and pretty bad at understanding layouts. So e.g. asking it to work with tables is asking for trouble.
- dontupvoteme 3y agoYeah it's absolutely horrible at layouts. I'm not 100% sure it's related but if you ask it to draw bounding boxes around things it's always off by quite a bit.
- a_wild_dandan 3y agoYou restrict the model's next prediction to valid JSON tokens. (If you mean format reliability.)
- rockwotj 3y agoI'm waiting for being able to restrict to a specific JSON Schema
- brianjking 3y agoProviding examples generally helps me. Have you checked out the Instructor library too?
- jgalt212 3y ago> How consistent and reliable is the extracted structure? That's the $100B question.
- szundi 3y agoOr $T these days
- simonw 3y agoIt's pretty good, but it's not reliable enough to exclude the need to check everything it does. Same story as basically everything relating to LLMs to be honest.
- brianjking 3y ago^ This. I've found success generally better with Textract and then passing the OCR'ed text to OpenAI and using Pydantic to get it structured.
- _flux 3y agollama.cpp has the feature to enforce a certain structure from the LLM (but you still need to have the LLM be able to produce that structure and it's beneficial to prompt it towards the exact result). So this particular story doesn't need to be the same among all LLMs. https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
- simonw 3y agollama.cpp grammars will get you results that definitely conform to your grammar, but that doesn't guarantee that they'll be semantically correct - the model could still hallucinate details incorrectly while returning valid JSON.
- _flux 3y agoIndeed it's not the solution to LLM hallucinations—as far as I know nobody knows a solution to it. But it is the solution to needing to run the model again and checking the format of the output to ensure that it conforms to you expectations.
- joshstrange 3y agoIn my testing I was better off running the image through AWS Textract then taking the output and feeding it to OpenAI. It was also much cheaper. Of course if all you are looking for is extraction then maybe you don't need OpenAI at all. I used it to clean up the OCR'd data and reformat it.
- Lucasoato 3y agoIt's very consistent, check this guy, he was able to structure LLM output using Pydantic in an elegant solution: https://www.youtube.com/watch?v=yj-wSRJwrrc https://www.youtube.com/watch?v=yj-wSRJwrrc
- mahithefish 3y agoEven if the GitHub link they just posted above. It made up the year at 2022 and hallucinated the start and end time. https://github.com/datasette/datasette-extract/issues/19 https://github.com/datasette/datasette-extract/issues/19
- deleted 3y ago[deleted]
- geepytee 3y agoOne of the OpenAI PM's was also saying the model got substantially better at math: https://x.com/owencm/status/1777770827985150022 https://x.com/owencm/status/1777770827985150022 I'm trying it for coding and have added it to my VS Code Copilot extension Overall I'd say it's better at coding than the previous GPT-4 Turbo. https://double.bot https://double.bot if anyone wants to try it :)
- g9yuayon 3y agoIs being good math that important to the ChatGPT users, though? The ChatGPT's ability to do math is so limited that I'm not sure what math problems we have to ask ChatGPT to solve.
- DiggyJohnson 3y agoIt means you don’t have to be as sketched out if you’re looking for something that requires basic math. Imagine generating the correct result of a unit test or something. I wouldn’t trust it either way, but I think this is a believable example.
- pama 3y agoMath is not just arithmetic and yes better math does help at least some GPT-4 users.
- g9yuayon 3y agoDo you have examples? The other commenter's example of generating correct unit test results is indeed interesting, and I was wondering if there are other cases.
- pama 3y agoNot sure what you are looking for, but perhaps these thoughts can help. Better geometry understanding helps with spatial tasks. Better probability theory, discrete math, and linear algebra background helps with algo development. More broadly, being better at things that a mathematician is good at, could help tackle complicated tasks in finance, science, or engineering.
- abrichr 3y agoWe've been using `gpt-4-1106-vision-preview` and simply prompting the model to return json, with excellent results: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress).
- Zambyte 3y agoWhat time is the event? I think 18:00 - 22:00 is completely fabricated.
- brianjking 3y agoGreat work, thank you, Simon. Have you noticed GPT-4 Vision do weird selections of dates for 2019? In doing some processing of some data for work, I ended up switching to textract and then passing the OCR'ed text to GPT-4 and mapping to a Pydantic schema due to this issue.
- pathikghugare 3y agoDoes anyone know if we can access the same on Azure OpenAI? It still shows vision-preview at my end in West US region with no JSON mode