4 ms·
GPT-4V is mind blowing, it surprising to me that it gets so little attention here on HN, because after playing around with it I get the same sense of excitement
by M4v3R 3y ago
GPT-4V is mind blowing, it surprising to me that it gets so little attention here on HN, because after playing around with it I get the same sense of excitement I got when I tried the original ChatGPT. The level of understanding of what is going on in an image is leagues ahead of what we had until this point, ahead of Bard and basically everything else I already saw.
I tested it with a bunch of photos I made and images it could not have seen in its training data, and most of the time it nailed them perfectly. Its OCR capabilities are top notch, but this is combined with a spatial understanding of how text relates to other parts of the image. It can take a photo of a wall monthly calendar with hand scribbles on it and give you a list of events for each day. It can guess where a specific photo was taken just by analysing the elements present on the photo like the foliage, architecture, car license plates, etc (without being specifically prompted to do so). It can correctly identify multiple plants from a same photo. Gave it a photo of a Montessori set for teaching math (some wooden blocks with numbers and dots on them, no branding on them) and it guessed exactly what it was. And all of that just from two days of testing.
Here are just few examples:
[1] https://i.imgur.com/cV3dVOf.png https://i.imgur.com/cV3dVOf.png - Gave it a screenshot from Final Fantasy VII from a boss battle. It correctly identified the party members, and their stats even though the text and labels are a bit all over the place.
[2] https://i.imgur.com/WeXhP7V.png https://i.imgur.com/WeXhP7V.png - A photo I shot on my vacation, that didn't really contain any major landmarks, and yet it still somehow figured out from the architecture (and house colors) the exact location of it. I tried this game with several photos and it is very good at it, far better than I could ever be if I saw these photos for the first time.
[3] https://i.imgur.com/HgwYv6q.png https://i.imgur.com/HgwYv6q.png - A screenshot of a worksheet from the Human Shader Project. I just asked it to solve it for given X/Y values and it did, its answer was 100% correct (here's the second part of its answer: https://i.imgur.com/RZF2r7v.png https://i.imgur.com/RZF2r7v.png)
[4] https://i.imgur.com/12xg4qU.png https://i.imgur.com/12xg4qU.png - A photo of a highly reflective microwave inside a shopping mall. This was given to my by a friend who shot this personally and to be honest I didn't catch at first that this is a microwave, and yet GPT-4V figured that out.
[5] https://i.imgur.com/qSifni5.png https://i.imgur.com/qSifni5.png - a good old fashioned "find the path connecting one object to the other" puzzle. Correctly identified the right path (this one was taken from the internet so there is a slight chance it saw it in the training data and somehow got the solution for it from the accompanying text, although I couldn't find any instance of it).
Edit: To confirm that [5] was not a fluke I hand drew my own version of this puzzle, took a picture and uploaded it, GPT-4V nailed this one too: https://i.imgur.com/8NgWhzw.png https://i.imgur.com/8NgWhzw.png
- meowkit 3y agoFor [2] there could be meta data in the photo (depending on what device you took it on) that GPTV is extracting which would help dramatically with identifying the location.
- gear54rus 3y agoI just did a screenshot of that and tested on my account. It still figured things out.
- M4v3R 3y agoGood call, but I did make sure to strip any metadata from the photos before I uploaded them :)
- postexitus 3y agoThat smiley face implies that you surely did not. Yes, I have kids.
- adroitboss 3y agoI just think the tech has been out for so long it's not as big of a deal. Mini-Gpt4 has been out for 6 months! Of course the descriptions aren't exactly gpt-4 grade, but with mistral 7b being used as the language model instead of llama 7b, the reasoning ability will improve noticeably. [1] https://github.com/Vision-CAIR/MiniGPT-4 https://github.com/Vision-CAIR/MiniGPT-4
- M4v3R 3y agoSure, the tech was out there for quite some time but never before the quality of the output was so good, it's almost (not 100%, there still are mistakes and hallucinations ocassionally) on par with a human, which to me is really stunning. I've tried these kind of queries with other models (including Mini-GPT4) and never got any meaningful results until now. It’s the same thing with GPT 3.5/4 - sure, transformer models existed for few years already but ChatGPT crossed some kind of threshold in the quality of its output where finally people took notice.