3 ms·

I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had ask

I'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that are tiny, gets obscured text like half moon bay (is it using a combination of the scene letters and knowledge in its language model of a pumpkin contest in HMB and synthesizing that knowledge into recognizing the location!!!!???). It's damn detailed.

simonw 3y ago
Yeah, I think it's spectacular - GPT4-V is so much better then I expected it to be.
- beebmam 3y ago
  It's fun to ask it to try to draw ascii art based on an image. It barely draws anything resembling the original image, but usually gets the concepts in the original image accurately.
jcims 3y ago
I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf’s hands in this image and it did a great job. https://imgur.com/a/orpDszj https://imgur.com/a/orpDszj I’m super curious. I’ve done some basic attempts to get it to spit out the actual raw data but it won’t. I’m sure folks that are good at that could.
- simonw 3y ago
  I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt. I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?
  - jcims 3y ago
    Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.
    - simonw 3y ago
      When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf https://cdn.openai.com/papers/GPTV_System_Card.pdf
      - GaggiX 3y ago
        I don't think they have trained the model from scratch on images too, but more like Flamingo or similar methods, and the LLM model is at most finetuned but not trained from scratch. If the model was trained from scratch then I would expect it to be able to develop new emergent behavior like being much better at ascii art, SVG, OpenSCAD scripting at least in the case the model was trained on the usual masked objective but not only on text tokens but image tokens too.
  - mk_stjames 3y ago
    The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar. [1] https://arxiv.org/pdf/2304.10592.pdf https://arxiv.org/pdf/2304.10592.pdf

simonw 3y ago

Yeah, I think it's spectacular - GPT4-V is so much better then I expected it to be.

beebmam 3y ago
It's fun to ask it to try to draw ascii art based on an image. It barely draws anything resembling the original image, but usually gets the concepts in the original image accurately.

jcims 3y ago

I’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf’s hands in this image and it did a great job. https://imgur.com/a/orpDszj https://imgur.com/a/orpDszj I’m super curious. I’ve done some basic attempts to get it to spit out the actual raw data but it won’t. I’m sure folks that are good at that could.

simonw 3y ago
I don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt. I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?
- jcims 3y ago
  Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.
  - simonw 3y ago
    When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf https://cdn.openai.com/papers/GPTV_System_Card.pdf
    - GaggiX 3y ago
      I don't think they have trained the model from scratch on images too, but more like Flamingo or similar methods, and the LLM model is at most finetuned but not trained from scratch. If the model was trained from scratch then I would expect it to be able to develop new emergent behavior like being much better at ascii art, SVG, OpenSCAD scripting at least in the case the model was trained on the usual masked objective but not only on text tokens but image tokens too.
- mk_stjames 3y ago
  The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar. [1] https://arxiv.org/pdf/2304.10592.pdf https://arxiv.org/pdf/2304.10592.pdf