17 ms·
GPT-4 Turbo with Vision Generally Available
- jxaphx 2y agoThis has made a huge difference in the way we extract structured data from images. Previously we had to perform a number of steps to ensure the JSON result was what we were looking for. Now we just get the function call exactly where we expect it. If you're a C# developer looking to take advantage of this improvement, check out our free open-source implementation at https://github.com/BackslashDev-LLC/img-to-json https://github.com/BackslashDev-LLC/img-to-json and our sample application at https://github.com/BackslashDev-LLC/med-intake-demo https://github.com/BackslashDev-LLC/med-intake-demo
- tucnak 3y agoThis is just responding to Anthropic, right? Funny how it took competition for them to make Vision-class models available.
- chimney 3y agoProbably the Gemini 1.5 Pro GA that was announced hours earlier.
- choilive 3y agoThat's typically how competition works right? Your competitors push you to do better
- maciejgryka 3y agoCompetition is great! But in this case, I don't know, adding JSON mode and function calling was a pretty obvious next steps for the vision model - I bet it'd have happened anyway.
- minimaxir 3y agoGPT-4-Vision has been around for awhile in beta, it's just GA now. It's expensive though: Anthropic's Claude Haiku can process images significantly cheaper.
- dvfjsdhgfv 3y agoSomeone could explain why on their documentation page (https://help.openai.com/en/articles/8555496-gpt-4-vision-api https://help.openai.com/en/articles/8555496-gpt-4-vision-api) they link to https://web.archive.org/web/20240324122632/https://platform.openai.com/docs/guides/vision https://web.archive.org/web/20240324122632/https://platform.... instead of https://platform.openai.com/docs/guides/vision https://platform.openai.com/docs/guides/vision? As a parson who pays OpenAI a lot of money each month, I see this as a bit parasitic (unless they donate substantially to the Internet Archive).
- blowski 3y agoI would assume a mistake, albeit one that raises questions about their QA processes. A static HTML page is hardly going to break their bank.
- SushiHippie 3y agoSeems to be changed to the correct link now
- simonw 3y agoThey also added both JSON and function support to the vision model - previously it didn't have those. This means you can now use gpt-4-turbo vision to extract structured data from an image! I was previously using a nasty hack where I'd run the image through the vision model to extract just the text, then run that text through regular gpt-4-turbo to extract structured data. I ditched that hack just now: https://github.com/datasette/datasette-extract/issues/19 https://github.com/datasette/datasette-extract/issues/19
- maciejgryka 3y agoHe, cool to hear you're doing something like this too! We ended up close, but we also need good spatial relationships, which GPT4V isn't great at, so we're using other OCR system and adding the result to the context.
- MuffinFlavored 3y ago> This means you can now use gpt-4-turbo vision to extract structured data from an image! How consistent and reliable is the extracted structure? Did they add any kind of concept "for whatever token you generate/think it is next, "unit test it" / make sure it passes some sort of rules?"
- maciejgryka 3y agoMy experience is that it's pretty good at reading text and pretty bad at understanding layouts. So e.g. asking it to work with tables is asking for trouble.
- dontupvoteme 3y agoYeah it's absolutely horrible at layouts. I'm not 100% sure it's related but if you ask it to draw bounding boxes around things it's always off by quite a bit.
- a_wild_dandan 3y agoYou restrict the model's next prediction to valid JSON tokens. (If you mean format reliability.)
- nuz 3y agoPeople like to say AI is moving blazingly fast these days but this has been like a year in the waiting que. Guessing sora will take equally long if not way longer before the general audience gets to touch it.
- brandon272 3y agoI could be misjudging the situation entirely, but Sora seems like it is on a much longer "general availability" timeline.
- freedomben 3y agoYeah OpenAI's head of something (CTO?) said late fall, if it's safe enough. Gonna be awhile until us normal people get our hands on it.
- CuriouslyC 3y agoI think with Sora, "general availability" will be a much more expensive higher tiered sub with a limited number of gens per day, and I have my doubts that you'll just be able to sign up for this sub through the web, I wouldn't be surprised if it's an invite only partners thing.
- belter 3y agoSounds like a complaint about a chair in the sky... https://youtu.be/8r1CZTLk-Gk https://youtu.be/8r1CZTLk-Gk
- dbbk 3y agoHaven't seen this in years, love this video
- arcanemachiner 3y agoI think about this bit a lot, mostly while I'm being mildly inconvenienced by something.
- throwup238 3y ago
- philip1209 3y ago`Moderation` doesn't support images yet, I believe. Does anybody have a good image moderation API they are using?
- maciejgryka 3y agoIt's cool to see this update and JSON mode and function calling will both be useful with vision. I wonder, though, if there were any other specific changes to the models since the `preview` versions besides that?
- jessenaser 3y agoWhy does the production model merge vision with the base turbo model such that the output tokens remains 2048 instead of 4096? Because if we are using just text, why is the extra output size reduced still?
- tedsanders 3y agoThat's our mistake - it's still 4096. Assuming you saw this on the Playground, we'll fix it shortly. If you saw it somewhere else, please let me know. https://platform.openai.com/playground/chat?model=gpt-4-turbo https://platform.openai.com/playground/chat?model=gpt-4-turb...
- jessenaser 3y agoYes I saw it in the playground. I will keep checking for updates for 4096. Thank you for the clarification. Edit (4:57 ET): "gpt-4-turbo" shows the updated 4096 in playground. "gpt-4-turbo-2024-04-09" remains 2048 in playground.
- jessenaser 3y ago(9:32 PM ET) Update: "gpt-4-turbo-2024-04-09" now shows the updated 4096 in the playground too. Thank you for the fix! :)
- greatpostman 3y agoApparently Gemini by google has a 20% LLM market share.
- ceejayoz 3y agoAccording to whom? How would you assess it?
- chimney 3y agoProbably saw this tweet from nat friedman https://twitter.com/natfriedman/status/1777739863678386268 https://twitter.com/natfriedman/status/1777739863678386268
- ceejayoz 3y agoThis methodology would ignore every API-driven use of the models that doesn't go through the first-party web interfaces.
- brcmthrowaway 3y agoKPIs going down
- ilaksh 3y agoFor all of these posts that says function calling is now available, I feel like it's actually more of an optimization than a new capability. All of the leading edge models will output JSON in any format you ask for, such as {"fn_name": {"arg1":10}}. I think this is about making it more accurate and having a standard input/output format.
- minimaxir 3y agoYou have to specify the schema for that regardless. Function calling has been available for text inputs for awhile, now it's also available for image inputs. OpenAI's function calling/structured data mode is much more strict and reliable at following the schema than just putting "return the output in JSON" in a system prompt.
- hansonw 3y agoYes. But also note that the new function calling is actually “tool calling” where the model is also fine-tuned to expect and react to the output of the function (and there are various other nuances like being able to call multiple functions in parallel and matching up the outputs to function calls precisely). When used in multi-turn “call/response” mode it actually does start to unlock some new capabilities.
- yieldcrv 3y agoI've used LLaVa a little bit through LM Studio, but its really subpar mostly due to the GUI I think, is there a model and GUI that's better than LM Studio for vision adapters? GPT-4V in Open AI's chat interface is so seamless. LLM, Text, Speech input, Speech output with tones and emotion, Image Generation, Vision input, and soon its going to be outputting video with Sora... its kind of amusing that GPT-4 early 2023 is still the benchmark of competition on both closed source and open source models when the lead is just expanding
- robterrell 3y agoWhat do you dislike about it?
- yieldcrv 3y agothe interface, that I'm needing to load multiple adapters, and that I don't have a language model at the same time. LM Studio is way better for language models only at the moment.
- htrp 3y agoIs gpt-4-turbo-2024-04-09 basically an updated version of the gpt-4-1106-vision-preview ?
- IshKebab 3y agoSlightly OT question - can I use GTP-4 vision to drive a web browser, e.g. to automate tasks like "sign up for this website using this email and password; don't subscribe to promotional emails"?
- wewtyflakes 3y agoI believe tying it together might be a challenge. For example, if you were to use the model to get the text of buttons, you would still have to write code to find the HTML elements for those buttons and drive the click/fill actions.
- peterleiser 3y agoI've used GPT-4 to help with selenium and it gets the answer eventually, but almost never on the first try. So automating this without human intervention sounds tricky.
- illnewsthat 3y agoYes, I played around with Skyvern and it works really well! https://www.skyvern.com/ https://www.skyvern.com/ https://github.com/Skyvern-AI/Skyvern https://github.com/Skyvern-AI/Skyvern
- hubraumhugo 3y agoTheir naming and versioning choices have definitely created some confusion. Apparently GPT-4 Turbo doesn't just come with the the new vision capabilities, but also with improved non-vision capabilities: > "I’m hoping we can get evals out shortly to help quantify this. Until then - it’s a new model with various data and training improvements, resulting in better reasoning."[0] > "- major improvements across the board in our evals (especially math)"[1] [0] https://twitter.com/owencm/status/1777784000712761430 https://twitter.com/owencm/status/1777784000712761430 [1] https://twitter.com/stevenheidel/status/1777789577438318625 https://twitter.com/stevenheidel/status/1777789577438318625
- MyFirstSass 3y agoAs a regular user of ChatGPT-4 i this press release makes little sense. What does Vision mean? I've already been able to both upload docs, create images etc. for a while now? What are the "other improvements" and what is Turbo, what is 4.5, what is this new one called? How do i even see what version of the model i'm using in their interface when it just says "4"?
- tedsanders 3y agoWorth noting that GPT can be accessed both through ChatGPT and via the OpenAI API. The link in this thread is pointing to documentation for the OpenAI API. Vision means the model can see image inputs. In the API, GPT with vision was previously available in a limited capacity (only some people, and didn't work with all features, like JSON mode and function calling). Now this model is available to everyone and it works with JSON mode and function calling. It also should be smarter at some tasks. This model is now available in the API, and will roll out to users in ChatGPT. In the API, it's named `gpt-4-turbo-2024-04-09`. In ChatGPT, it will be under the umbrella of GPT-4.
- qazadex 3y agoIn my experience the turbo models are significantly worse than the old GPT-4 models, especially in structured output and following instructions. I'm guessing its due to the cheapened attention mechanism, but its a bit disappointing that OpenAI tries and hides these limitations.
- andrewstuart 3y ago"a fix for a bug" <laugh emoji> gpt-3.5-turbo-0125 New Updated GPT 3.5 Turbo The latest GPT-3.5 Turbo model with higher accuracy at responding in requested formats and a fix for a bug which caused a text encoding issue for non-English language function calls. Returns a maximum of 4,096 output tokens. Learn more.
- ugh123 3y agoOver Easter I asked GPT4 to count a mess of colored eggs on the floor that I was preparing for an egg hunt. They were mostly evenly separated and clearly visible (there were just 36). I gave it two tries to respond and it wasn't even close to the correct answer. Was it confused on colored eggs vs. "natural" eggs it might have been expecting? Should it have understood what I meant?
- ravenstine 3y agoI imagine it would be better at describing an image in a general sense, but probably isn't processing it in a sense where it would actually be counting individual features. I could be wrong about that, but it seems like a combination of traditional CV and an LLM might be what's needed for more precise feature identification.
- 0x008 3y agoThe use case of “counting objects” is basically completely solved already by yolov8. There is no need to use an LLM for that.
- ugh123 3y agoSeems like we're pretty far away from "AGI" or even anything resembling a legit "multi model"
- ugh123 3y agomulti modal
- philipwhiuk 3y agoI need an AI model to tell me which AI model/tool to use.
- 0x008 2y agoThen you might want to look into query routing or function calling.
- bilsbie 3y agoIs it worth still paying the $20?
- ndr_ 3y agoThis news piece is, first and foremost, about the Model, not the ChatGPT System. (More about the difference between “Model” and “System”: https://ndurner.github.io/antropic-claude-amazon-bedrock https://ndurner.github.io/antropic-claude-amazon-bedrock). Not sure what their upgrade policy/process for ChatGPT is like, though.
- ChrisLTD 3y agoIs it possible to upload an image to the OpenAI Chat Playground to try it out?
- ndr_ 3y agoApparently not. But here: https://huggingface.co/spaces/ndurner/oai_chat https://huggingface.co/spaces/ndurner/oai_chat
- brianjking 3y agoNo, but you can use the API via Postman.
- brianjking 3y agocurl https://api.openai.com/v1/chat/completions https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4-turbo", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What'\''s in this image?" }, { "type": "image_url", "image_url": { "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gf..." } } ] } ], "max_tokens": 300 }'
- anonymousDan 3y agoCan I ask, how are people affording to play around with GPT4? Are you all doing it at work? Or is there some way I am unaware of to keep the costs down enough to play around with it for experimenting? It's so expensive!
- ShamelessC 3y agoHow much are you using it? I have been accessing gpt-4-turbo via API in a small discord with a few friends using it as well. Have never gone above 5$/month in usage.
- shmoogy 3y agoWork for 4 api calls, for personal turbo 3.5 and ollama mixtral
- hiddencost 3y agoHow expensive? Plenty of folks here can afford to burn $1000 / mo to play with a hobby, I think that's how.
- jasonjmcghee 3y agoDepends what "so expensive means". Roughly, $100 will get you 5-10 novels worth of output, assuming 100-200k words. That's a lot.
- abhimanyue1998 3y agoapply for microsoft startup founders hub, you get 2.5k worth of credits for free
- debian3 3y agoIf you want to use over API you need to pay for what you use. If you simply want to chat with it, currently the cheapest way is cody by sourcegraph ($9/month for unlimited gpt4 and opus with a limit of 7k token). Phind is $20 per month, same models, 32k token limit.
- anonymousDan 3y ago
- alexpogosyan 3y agoWhat software people use to interact with these models via chat?
- ndr_ 3y agohttps://huggingface.co/spaces/ndurner/oai_chat https://huggingface.co/spaces/ndurner/oai_chat (bring your own API key)
- jiggawatts 3y agoRubs crystal ball: Widespread availability in Azure will take four months. No wait, it’s just a software change, I’m being silly… six months.
- 7thpower 3y agoI bet they’ll bump you up in the queue if you buy some sweet sweet PTUs
- jiggawatts 3y agoEven the previous model version is unavailable in my entire region.
- simonw 3y agoHere's a blog post about what I've been building with GPT-4 Turbo + Vision for structured data extraction into SQLite database tables from unstructured text and images: https://www.datasette.cloud/blog/2024/datasette-extract/ https://www.datasette.cloud/blog/2024/datasette-extract/ YouTube video demo here: https://www.youtube.com/watch?v=g3NtJatmQR0 https://www.youtube.com/watch?v=g3NtJatmQR0 (3m43s)
- sireat 3y agoChatGPT Plus was able to tell me what type of coffee capsules I should buy based on a picture with multiple objects with text markings. The sole coffeemaker had a sideways NS -> Nespresso compatible ones. Is this basically the same functionality except through API?
- traspler 3y agoAre there any additional improvements to function calling/json mode? In the non-turbo models it really struggled with enums.
- djmips 3y agoI've had this for a while, was I in a test bucket?