8 ms·
How Does GPT-4o Encode Images?
- simonw 2y agoThe way this tests GPT-4o performance by feeding in a 7x7 grid of colored shapes and requesting them back as JSON (about half way down the page) is really clever.
- blixt 2y agoI did something similar when GPT-4V came out, partially with the goal to figure out the input format (I did not get anywhere other than "magic vectors"), but also to roughly estimate the amount of data you can get back out of a 512x512 (the low quality option) image. What I found is that you can sometimes get more text out of 85-token image than you can out of 85 tokens of text! That said, I think there will be plenty of edge cases where it actually loses some information, and maybe you could argue that if you remove every other word in the text, it could still restore the text. I never went deeper on this, but I believe there's something clever to be done in the context window with the fact that images are relatively cheap tokens-wise.
- _dark_matter_ 2y agoThe author mentions this in the article, that more than 170 tokens of text can be pulled from an image.
- blixt 2y agoAh, you're right! My bad!
- GaggiX 2y agoAn important aspect that is not considered in the article is that GPT-4o can generate images by itself (even though the feature is not enable to the public) meaning that it's very likely trained on sequential image tokens and the images are quantized using a VQGAN, my guess is that the VQGAN takes 512x512 images and outputs 13x13 tokens (169 image tokens + special token), the VQGAN can be a convolutional network like shown in the article, for a transformer-based VQGAN I cannot think of a configuration with overlapping patches where it would output 13x13 tokens on a 512x512 image (unless they just added a padding of 4 on the entire image and the patches are not overlapping).
- edude03 2y agoHow do we know it generates the images itself and isn’t passing the text to dalle? It’s supposedly how the current gpt4 model does listen mode (with whisper but same idea)
- GaggiX 2y agoGo to the "Explorations of capabilities" and explore all the capabilities: https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ You cannot have this level of control by prompting Dalle, also GPT-4o isn't using Whisper (older GPT-4s yes).
- ec109685 2y agoAt least ChatGPT 4o still looks like it is using dalle. https://x.com/krishnanrohit/status/1755123169353236848?s=46 https://x.com/krishnanrohit/status/1755123169353236848?s=46
- hackerlight 2y agoTwo reasons - the shown capabilities are way beyond what dalle is capable of, and they've been clear that this "omni" model by the "omni team" is natively multimodal
- simonw 2y agoSomething I don't get is why OpenAI don't provide clear, comprehensive documentation as to how this actually works, I get that there's competition from other providers now so they have an instinct to keep implementation details secret, but as someone building on their APIs this lack of documentation really holds me back. To make good judgements about how to use this stuff I need to know how it works! I had a hilarious bug a few weeks ago where I loaded in a single image representing multiple pages of a PDF and GPT-4 vision effectively hallucinated the contents of the document when asked to OCR it, presumably because the image was too big and was first resized to a point where the text was illegible: https://simonwillison.net/2024/Apr/17/ai-for-data-journalism/#llm-mistakes https://simonwillison.net/2024/Apr/17/ai-for-data-journalism... If OpenAI had clear documentation about how their image handling works I could avoid those kinds of problems much more effectively.
- Onawa 2y agoI was trying to figure out this exact same issue. OCR on a PDF worked great, up until a certain point when it just started hallucinating like crazy. I was working on a whole pipeline to just feed in a PDF one page at a time to try and get around this issue. Otherwise, the OCR works absolutely fantastic compared to all other other tools I've been trying lately. These include OCRmyPDF (Tesseract), SuryaOCR, and some of the models on the Visual LLM Leaderboard. I've also seen some people recommend Paddle OCR, but I find their documentation to be lacking and I haven't got that one working yet to evaluate.
- infecto 2y agoFor document text/table extraction, nothing beats the quality from the cloud providers. It can get costly but the accuracy is much higher than what you will find using an openai API.
- raybb 2y agoSimon wilson recently had a thread going through some of the options here https://x.com/simonw/status/1797526667797442773 https://x.com/simonw/status/1797526667797442773
- tantalor 2y ago> CLIP embeds the entire image as a single vector, not 170 of them. Single token? > GPT-4o must be using a different, more advanced strategy internally Why
- freediver 2y agoThe embeddings do not offer the level of fidelity to recognize fine details on an image, hand writing for example.
- blixt 2y agoI went through a similar journey back when GPT-4V came out. Here's an additional puzzle for you: GPT-4V knows the exact pixel dimensions of the image (post-resize since there is a max size for images in the pipeline, besides 512x512), but I'm 99% sure it's not provided as text tokens. How am I so sure? It's easy to get GPT to divulge everything from system prompt to tool details, etc. but I've tried every trick in the book and then some, multiple times over, and there is no way to get it to quote the dimensions as text. The only way to get it to give you the dimensions is to tell it to output a structure that contains width and height and just pick something reasonable, and they will "randomly" be the correct values: https://x.com/blixt/status/1722298733470024076 https://x.com/blixt/status/1722298733470024076
- dannyw 2y agoPerhaps images aren’t tokens at all… and 170 tokens is just an approximation of the compute cost.
- qarl 2y agoThey address this question in the article.
- blixt 2y agoI think that would have pretty serious implications for the transformer architecture though. If they're not embedded like text tokens, how would attention, etc work? And a conversation with multiple images back and forth? Not to mention with GPT-4o now having audio support as well. I would assume it does become tokens.
- deleted 2y ago[deleted]
- llm_trw 2y ago> It's easy to get GPT to divulge everything from system prompt to tool details, It's easy enough to get it to hallucinate those things. It doesn't actually tell them to you.
- alach11 2y agoI really hope we see improvements to the resolutions large multimodal models can handle. Right now this patchwork approach leads to lots of unwieldly workarounds in applications.
- eminence32 2y agoI'm assuming that the tokens used to encode an image are entirely distinct from the tokens used to encode text. Does anyone know if this is actually the case?
- tempusalaria 2y agoIt’s probable that there is a separate vision encoder which projects the image tiles into the distribution space of the text tokenizer a la CLIP/LLava
- blixt 2y agoI would assume it has a "mode" token where it switches between text/image (and now audio), or you'd have to try to maximize the number of reserved tokens between multiple modes. GPT-4o did go from 100K to 200K vocabulary, but as far as I understand all of that vocabulary is in use for text (reducing the token cost for non-English).
- rvnx 2y agoAuthor claims that the most likely is that there is Tesseract running behind ChatGPT-4v/o. There is no way that this is Tesseract. -> Tesseract accuracy is very low, it can barely do OCR on printed documents.
- jerrygenser 2y agoEven if tesseract accuracy is low, if the tesseract result in addition to the image is then passed to the LLM, it can result in a much more accurate OCR. For example, GPT4 with some vision capability would be able to fill in the incorrect OCR with the additional word co-occurrence understanding. I've tested this approach with purely text LLM to correct OCR mistakes and it works quite well. Also note that in some newer OCR pipelines that don't involve LLMs, there is a vision component and then a text correcting model that is in some ways similar to some forms of spell check, which can further improve results.
- lyu07282 2y agoyou can tell that the OCR fails more in cases without natural language like with code/random characters. OAI seems to claim 4o is a fully end to end multimodal model, but we will never know for sure, we can't trust a single word OpenAI is saying.
- kherud 2y agoShouldn't this theory be testable? The response time for an image of the same size should remain constant (assuming a generated response of constant size). You could then try to put an increasing amount of text inside of the image. If this text is fed to the LLM using OCR, the total amount of tokens grows. You should then be able to observe an increase in response time.
- RicoElectrico 2y agoYeah, Tesseract is barely production quality.
- lyu07282 2y ago
- surfingdino 2y agoOCR is hard https://www.vice.com/en/article/gvy4gb/one-mans-david-and-goliath-battle-to-get-xerox-to-fix-a-major-bug https://www.vice.com/en/article/gvy4gb/one-mans-david-and-go...
- joelburget 2y agoVision transformers should be our default guess as to how GPT-4o works, yet this article never mentions them.
- valine 2y agoLlava1.6, IntenVL, CogVLM2 can all do OCR with nothing but tiled image embeddings and an LLM. Feeding in OCR results from tesseract improves the reliability of the transcript, especially for long strings of random characters, but it’s not strictly necessary for the model to read the text out of the image. Clip embeddings can absolutely “read” text if the text is large enough. Tiling enables the model to read small text.
- Onawa 2y agoDo you know of any guides or tutorials to doing this? I tried using the MiniCPM model for this task, but it just OCRed a tiny bit of information then told me that it couldn't extract the rest.
- pwillia7 2y agoI bet you could get this working in https://github.com/comfyanonymous/ComfyUI https://github.com/comfyanonymous/ComfyUI I have done some other LLava stuff in it
- tictacttoe 2y agoI found llava to be disappointing, but Claude Haiku is quite good
- iknownothow 2y agoI'm probably wrong but the author may have have misunderstood input embeddings. Input embeddings are just dictionary lookup tables. The tokenizer generates tokens and for each token you find its embedding from the lookup. The author is speculating about an embedding model but in reality they're speculating about the image-tokenizer. If I'm not wrong the text tokenizer Tiktoken has a dictionary size of 50k. The image tokenizer could have a very large dictionary size or a very small dictionary size. The 170 tokens this image tokenizer generates might actually have repeating tokens! EDIT: PS. What I meant to say was that input embeddings do not come from another trained model. Tokens come from other trained models. The input embedding matrix undergoes back propagation (learning). This is very important. This allows the model to move the embeddings of the tokens together or apart as it sees fit. If you use embeddings from another model as input embeddings, you're basically adding noise.
- iknownothow 2y agoI've pondered a bit more about it and I was the one who was mistaken. I think the author made great observations. It's just that I don't want to go back to non token thinking. I don't want there to be a 13x13xE final output from the CNN. I really want there to be a visual vocabulary from which tokens are chosen. And I want this visual vocabulary to be fixed/untrainable/deterministic. That'd be very cool. But why only choose 13x13 + 1? :( I'm willing to bet that the author's conclusion of embeddings coming from CNNs is wrong. However, I cannot get the 13x13 + 1 observation out my head though. He's definitely hit on something there. I'm with them that there is very likely a CNN involved. And I'm going to put my bet on the final filters and kernel are the visual vocabulary. And how do you go from 50k convolutional kernels (think tokens) to always 170 chosen tokens for any image? I don't know...
- kolinko 2y agoInput embeddings are taken from a dictionary in case of text tokens, but they don’t need to be - they can be any vector really.
- iknownothow 2y agoBut don't input embeddings need to undergo backprop during training? Won't the external-model's embeddings just be noise since they don't share embedding space with the model that is being trained? If the external-model also undergoes training along with the model then I think that might work.
- riemannzeta 2y agoLove this curious and open-minded exploration of how this stuff works. The pyramid strategy loosely tracks with renormalization group theory, which has been formally studied for years as a method of interpreting machine learning models: https://arxiv.org/abs/1410.3831 https://arxiv.org/abs/1410.3831 I love the convergence we're seeing in the use of models from different fields to understand machine learning, fundamental physics, and human consciousness. What a time to be alive.
- yorwba 2y agoIt would be interesting to see what happens when you slightly shift the grid of objects until they're split across multiple tiles, and how that affects accuracy.
- rafaelero 2y agoThey are very likely using VQVAE to create a dictionary of tokens and then just converting images into them with an encoder.
- lisperforlife 2y agoWhy is this not the top comment? FAIR published their C3MLeon paper about decoder-only autoregressive models that work with both text and image tokens. I believe GPT-4o's vocabulary has room for both image and audio tokens. For audio tokens, they probably trained an RVQ-VAE model like Encodec or Soundstream.
- HarHarVeryFunny 2y agoWouldn't that be more applicable to image generation, or at least wanting to encode the image as a whole? If you need to be able to reason about multiple objects in the image and their relative positions, then don't you need to use a tiled approach?
- rafaelero 2y agoVQVAE is trained to reconstruct the image, so in theory it should contain all the information (both content and location) inside its embeddings.
- cs702 2y agoOne possibility is that mapping images to a token embedding consumes ~170x more compute+space than mapping a token id. Another possibility is that OpenAI is mapping each image to ~170 vectors in an embedding space that is shared with token IDs. If that's the case, the architecture of the image-to-fixed-number-of-tokens model has not been disclosed. It could be a standard CNN, a ViT-like model, an autoencoder, a model that routes a variable number of vectors with RGB data to a fixed number of vectors, or something else that has not yet been ublished. The whole thing is likely trained end-to-end.
- cs702 2y ago*that has not yet been published
- CuriouslyC 2y agoAt some point we're going to go from tokens to embeddings for everything. I saw some research on variable length embeddings, I wouldn't be surprised if someone generated a huge embedding space, did some form of PCA on generated embeddings, threw away low eigenvalue vectors, then trained a distilled model that generated variable length embeddings directly from that.
- cs702 2y ago> At some point we're going to go from tokens to embeddings for everything. Yes, I agree. Further down the road, I imagine we will end up finding interesting connections to the symbolic approaches of GOFAI, given that the embedding of a token, object, concept, or other entity in some vector space is basically a kind of symbol that represents that token, object, concept, or entity in that vector space. Interestingly, old terms like "representation" and "capsule," which didn't become as widely adopted as "embedding," tried more explicitly to convey this idea of using vectors/matrices of feature activations to stand in for objects, concepts, and other entities. For example, see Figure 1 in this paper from 2009-2012: http://www.cs.princeton.edu/courses/archive/spring13/cos598C/Representation%20Learning%20-%20A%20Review%20and%20New%20Perspectives.pdf http://www.cs.princeton.edu/courses/archive/spring13/cos598C... -- it's basically what we're talking about!
- sva_ 2y agoGreat article. Perhaps some part of this magic number simply factors in the amount of compute necessary to run the image through the CNN (proportional to compute use per token in the LM).
- jmount 2y agoScanning images is quite the problem in the presence of compression (and now interpolation) https://www.bbc.com/news/technology-23588202 https://www.bbc.com/news/technology-23588202 .
- euclideanfire 2y ago[flagged]
- chpatrick 2y agoYou can download plenty of open source models and try it yourself, OpenAI is just one step ahead.
- geor9e 2y agoNit: the implied premise that this isn't a beautiful and skilled painting https://www.oranlooney.com/post/gpt-cnn_files/malicious_dogs.png https://www.oranlooney.com/post/gpt-cnn_files/malicious_dogs...
- olooney 2y agoThe painting is Charlie and Sheba, from the Museum of Bad Art: https://museumofbadart.org/zoo/ https://museumofbadart.org/zoo/ I found it while Googling for a test image for the malicious prompt test, where it was used as the lead photo for this blog post: https://www.artsy.net/article/artsy-editorial-bad-art-good https://www.artsy.net/article/artsy-editorial-bad-art-good There's definitely something eye-catching about it that really makes it stand out from the crowd.
- deleted 2y ago[deleted]
- HarHarVeryFunny 2y agoI don't think a 13x13 tiling (of N channels/features) can be ruled out just because it can't recognize a grid of 13x13 objects. There is presumably a lot of overlap between the receptive fields of the tiles (due to kernel step sizes). A pyramid of overlapped tiling resolutions is of course possible too.
- ComputerGuru 2y agoWe desperately need a modern open source replacement for tesseract built on current SoTA ML tech. It is insane that we are resorting to using LLMs — which aside from being the wrong tool and far too overpowered for the job also are prone to hallucinations, have insanely expensive training and inference costs, etc — for this purpose because the “best” non-LLM solution is so bad it can’t even correctly ocr monospaced hi-res scans of ascii text with sufficient accuracy.
- asadm 2y agohmmm I haven't tried but does apple's OCR api do better here? ie. is it possible to do it.
- rgovostes 2y agoThe API: https://developer.apple.com/documentation/vision/recognizing_text_in_images https://developer.apple.com/documentation/vision/recognizing... In my experience it works remarkably well for features like scanning documents in Notes and in copying or translating text embedded in images in Safari. It is not open source, but free to use locally. Someone has written a Python wrapper (apple-ocr) around it if you want to use it in other workflows. The model files might be in /System/Library/PrivateFrameworks/TextRecognition.framework if you wanted to port them to other platforms.
- nexuist 2y agoI also wrote a Swift CLI that wraps over the Vision framework: https://github.com/nexuist/seev https://github.com/nexuist/seev Text extraction is included (including the ability to specify custom words not found in the dictionary) but there are also utilities for face detection, classification, etc.
- rvnx 2y agoOne good self-hosted OCR is PaddleOCR, https://github.com/PaddlePaddle/PaddleOCR https://github.com/PaddlePaddle/PaddleOCR Beats everything else, truly international and multi-lingual, including Chinese (as it is made in China)
- imranhou 2y agoNot to be nit-picky but double checking myself, isn't a token just 0.75 words, so 170 token would be 127 words and not 227?
- enjoylife 2y ago> Interestingly enough, it’s actually more efficient to send text as images: A 512x512 image with a small but readable font can easily fit 400-500 tokens worth of text, yet you’re only charged for 170 input tokens plus the 85 for the ‘master thumbnail’ for a grand total of 255 tokens—far less than the number of words on the image. Sounds like an arbitrage opportunity for all those gpt wrappers. Price your cost per token the same, send over the prompt via image, pocket the difference?
- falcor84 2y agoWould it be worthwhile in terms of the added server costs of printing the text to images?
- SubiculumCode 2y agoI'm not sure how chatgpt4o routes information. If a picture is submitted that contains text, does the text then get resubmitted to chatgpt4o as a textual query, or do the model weights themselves essentially transform the textual images to textual tokens. I do wonder if a response to the textual images is similar to a response to text queries...i.e. processed by the the same weights.
- sashank_1509 2y agoI would be disappointed if OpenAI had a separate model for OCR, though I guess that is believable. Much cooler if the LLM just understands language from text
- comboy 2y agoI love how well this is written. Definitely "look how interesting this is" rather than "look how much do I know". And it dives as deep as needs to, while being accessible for almost everyone. One really needs to master some topic to be able to describe it simply. Great job.
- jamesy0ung 2y agoI've always wondered how Text to Image LLMs like stable diffusion work, do they just encode RGB values into a matrix and then have a helper tool convert that data into a jpg?