4 ms·

At the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152

At the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152 tokens. Does anyone understand where the information for the response came from? That is, does Gemini hold onto the original uploaded non-tokenized image, then run an OCR on it to read those titles? Or are all those book titles somehow contained in those 258 tokens? If it's the later, it seems amazing that these tokens contain that much information.

simonw 3y ago
I would LOVE to understand that myself.
jacobr1 3y ago
I'm not sure about Gemini, but OpenAI GTP-V bills at roughly a token per 40x40px square. It isn't clear to me these actually processed as units, but rather it seems like they tried to approximate the cost structure to match text.
zacmps 3y ago
Remember, if it's using a similar tokeniser to GPT-4 (cl100k_base iirc), each token has a dimension of ~100,000. So 258x100,000 is a space of 25,800,000 floats, using f16 (a total guess) that's 51.6kB, probably enough to represent the image at ok quality with JPG.
- simonw 3y ago
  I don't think that's right. A token in GPT-4 is a single integer, not a vector of floats. Input to a model gets embedded into vectors later, but the actual tokens are pretty tiny.
  - zacmps 3y ago
    Ah true, I guess it's still 258 positions by 100,000 possible tokens though.
  - l33tman 3y ago
    But they are not a "single integer" either as in, like a byte... I don't have any good examples but I'm pretty sure the tokens are in the range of thousands of dimensions. It has to encode the properties of the patch of the image it derives from, and even a small 40x40 RGB pixel patch has plenty of information you have to retain.
    - llm_nerd 3y ago
      A token is a single integer from a dictionary of a given model's vocabulary (e.g. GPT-4 has a vocab of ~100k different tokens, Gemma has ~256k). You are discussing embeddings which are a deeper, different element of models. https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer In the given example the video was condensed to a sequence of 258 tokens, and clearly it was a very minimalist, almost-entirely-ocr extraction from the video.
      - l33tman 3y ago
        Yeah but we're not talking about LLMs here but vision transformers, which don't use the same type of token vocabulary to produce embeddings from the input as the LLMs do. The pixel data is much more dense than a few characters is, per token. I looked it up - the original ViT models directly projected for example 16x16 pixel patches into 768-dimensional "tokens". So a 224x224 image ended up as 14*14=196 "tokens" each of which is a 768-dimensional vector. The positional encoding is just added to this vector. This blog-post has the specific numbers, which makes it a bit less abstract than in the original paper: https://amaarora.github.io/posts/2021-01-18-ViT.html https://amaarora.github.io/posts/2021-01-18-ViT.html
        llm_nerd 3y ago
        >Yeah but we're not talking about LLMs here but vision transformers We ultimately are. Gemini is a multimodal model whose core function is an LLM. This doesn't mean that everything flows through the same pathway -- different modalities have different paths -- but eventually there is fusion through which a common representations appears. It's where the worlds combine. That parlance is often tokens, though it obviously depends upon the architecture and we simply don't have those details for Gemini (the paper is extremely superficial). The fact that it will ingest massive videos and then post-facto answer arbitrary queries on it is a good clue, however. >This blog-post has the specific number It's a great link and an enjoyable read, and while the ViT plays a critical role in virtually all image analysis pipelines, including in Gemini where it is a part of OCR, object detection, etc, the numbers you are referring to do not map to tokens. E.g. the 768 dimensions are nothing more than the underlying image data for the tile. e.g. 16x16x3 channels. I'm unaware of any ViT resources that refers to those vectors (vectorized because that's the form GPUs like) as tokens. This system could lazily reuse it, but the way processing happens in ViTs would make that a completely irrational overlap of terms. The role that a token plays in that description is the classifier -- basically the output that classifies each tile. Ultimately the number of tokens that Google or OpenAI assign to processing an image or video is a billing artifact because tokens are the measure by which things are billed. However you can ask these systems for the tokens representing an image and it will be exactly what one would expect. Indeed, the brilliance of image (and thus video) analysis in these multimodal systems is not nearly as deep as first glances might assume, and often it can derive nothing more than the most obvious classifications. e.g. classifications made without knowing anything about what the user specifically wants. It is usually fantastic at things like OCR, which happens to be a very common need. These systems obviously have different usage patterns. I can do simultaneous processing where the image and command work in concert, image analysis deep diving on specifically those elements that are wanted (but that would otherwise be ignored). Or I can do the classic feed a video or an image and then ask questions where the dominant model is to tokenize the video or images using the common flow (OCR, object detection, etc), create a token narrative, and then answer the question from the narrative.
llm_nerd 3y ago
The whole matter of tokens from video is one that has a lot of ambiguity, and is often presented as if these are some unique weird encoding of the contents of the video. But logically the only possible tokenization of videos (or images, or series of images ala video) is basically an image to text model that takes each frame and generates descriptive language -- in English in Gemini -- to describe the contents of the video. e.g. A bookshelf with a number of books. The books seen are "...", "...", etc. A figurine of a squirrel. A stuffed owl. And so on. So the tokenization by design would include the book titles as the primary information, as that's the easiest, most proven extraction from images. From a video such tokenization would include time flow information. But ultimately a lot of the examples people view are far less comprehensive than they think. It isn't surprising that many demonstrations of multimodal models always includes an image with text on it somewhere, utilizing OCR.
- famouswaffles 3y ago
  This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image. https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929
  - llm_nerd 3y ago
    >This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression. This process is obvious and logical if actually thought through. >Each image is about 258 tokens Because Google set that as the "budget" and truncates accordingly. Again, call the API with an image or video and then convert those tokens to text. >https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 This is super weird, and does not remotely prove your point. I literally spend most of my days in ViTs, but thanks for the link.
famouswaffles 3y ago
Image tokens =/ Text tokens. Image tokens are patches of the image. Each image is divided into ~256 parts. Those parts are the tokens. There's no separate run to another OCR.
- llm_nerd 3y ago
  Completely wrong. Well, aside from the edited in bit about OCR. Of course there isn't a separate run to do OCR because that was literally the first step during image analysis. You know, before the conversion to simple tokens.
  - famouswaffles 3y ago
    There's no run to any OCR, first step or not. And you have no idea what you're talking about.
    - deleted 3y ago
      [deleted]
    - llm_nerd 3y ago
      You understand that OCR is the process of extracting text from images, right? You know, such as what Gemini does, and they reference repeatedly in their paper. I have absolutely no idea why you repeatedly make some bizarre distinction about it being a "separate process". Okay, it's been fun talking to you but feel free to have the last word. Good luck.
      - famouswaffles 3y ago
        The transformer (Gemini) predicts text with image and text in the context window. That's it. OCR, Object detection etc all come from the transformer predicting text. Read the Flamingo paper.

simonw 3y ago

I would LOVE to understand that myself.

jacobr1 3y ago

I'm not sure about Gemini, but OpenAI GTP-V bills at roughly a token per 40x40px square. It isn't clear to me these actually processed as units, but rather it seems like they tried to approximate the cost structure to match text.

zacmps 3y ago

Remember, if it's using a similar tokeniser to GPT-4 (cl100k_base iirc), each token has a dimension of ~100,000. So 258x100,000 is a space of 25,800,000 floats, using f16 (a total guess) that's 51.6kB, probably enough to represent the image at ok quality with JPG.

simonw 3y ago
I don't think that's right. A token in GPT-4 is a single integer, not a vector of floats. Input to a model gets embedded into vectors later, but the actual tokens are pretty tiny.
- zacmps 3y ago
  Ah true, I guess it's still 258 positions by 100,000 possible tokens though.
- l33tman 3y ago
  But they are not a "single integer" either as in, like a byte... I don't have any good examples but I'm pretty sure the tokens are in the range of thousands of dimensions. It has to encode the properties of the patch of the image it derives from, and even a small 40x40 RGB pixel patch has plenty of information you have to retain.
  - llm_nerd 3y ago
    A token is a single integer from a dictionary of a given model's vocabulary (e.g. GPT-4 has a vocab of ~100k different tokens, Gemma has ~256k). You are discussing embeddings which are a deeper, different element of models. https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer In the given example the video was condensed to a sequence of 258 tokens, and clearly it was a very minimalist, almost-entirely-ocr extraction from the video.
    - l33tman 3y ago
      Yeah but we're not talking about LLMs here but vision transformers, which don't use the same type of token vocabulary to produce embeddings from the input as the LLMs do. The pixel data is much more dense than a few characters is, per token. I looked it up - the original ViT models directly projected for example 16x16 pixel patches into 768-dimensional "tokens". So a 224x224 image ended up as 14*14=196 "tokens" each of which is a 768-dimensional vector. The positional encoding is just added to this vector. This blog-post has the specific numbers, which makes it a bit less abstract than in the original paper: https://amaarora.github.io/posts/2021-01-18-ViT.html https://amaarora.github.io/posts/2021-01-18-ViT.html
      - llm_nerd 3y ago
        >Yeah but we're not talking about LLMs here but vision transformers We ultimately are. Gemini is a multimodal model whose core function is an LLM. This doesn't mean that everything flows through the same pathway -- different modalities have different paths -- but eventually there is fusion through which a common representations appears. It's where the worlds combine. That parlance is often tokens, though it obviously depends upon the architecture and we simply don't have those details for Gemini (the paper is extremely superficial). The fact that it will ingest massive videos and then post-facto answer arbitrary queries on it is a good clue, however. >This blog-post has the specific number It's a great link and an enjoyable read, and while the ViT plays a critical role in virtually all image analysis pipelines, including in Gemini where it is a part of OCR, object detection, etc, the numbers you are referring to do not map to tokens. E.g. the 768 dimensions are nothing more than the underlying image data for the tile. e.g. 16x16x3 channels. I'm unaware of any ViT resources that refers to those vectors (vectorized because that's the form GPUs like) as tokens. This system could lazily reuse it, but the way processing happens in ViTs would make that a completely irrational overlap of terms. The role that a token plays in that description is the classifier -- basically the output that classifies each tile. Ultimately the number of tokens that Google or OpenAI assign to processing an image or video is a billing artifact because tokens are the measure by which things are billed. However you can ask these systems for the tokens representing an image and it will be exactly what one would expect. Indeed, the brilliance of image (and thus video) analysis in these multimodal systems is not nearly as deep as first glances might assume, and often it can derive nothing more than the most obvious classifications. e.g. classifications made without knowing anything about what the user specifically wants. It is usually fantastic at things like OCR, which happens to be a very common need. These systems obviously have different usage patterns. I can do simultaneous processing where the image and command work in concert, image analysis deep diving on specifically those elements that are wanted (but that would otherwise be ignored). Or I can do the classic feed a video or an image and then ask questions where the dominant model is to tokenize the video or images using the common flow (OCR, object detection, etc), create a token narrative, and then answer the question from the narrative.

llm_nerd 3y ago

The whole matter of tokens from video is one that has a lot of ambiguity, and is often presented as if these are some unique weird encoding of the contents of the video. But logically the only possible tokenization of videos (or images, or series of images ala video) is basically an image to text model that takes each frame and generates descriptive language -- in English in Gemini -- to describe the contents of the video. e.g. A bookshelf with a number of books. The books seen are "...", "...", etc. A figurine of a squirrel. A stuffed owl. And so on. So the tokenization by design would include the book titles as the primary information, as that's the easiest, most proven extraction from images. From a video such tokenization would include time flow information. But ultimately a lot of the examples people view are far less comprehensive than they think. It isn't surprising that many demonstrations of multimodal models always includes an image with text on it somewhere, utilizing OCR.

famouswaffles 3y ago
This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image. https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929
- llm_nerd 3y ago
  >This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression. This process is obvious and logical if actually thought through. >Each image is about 258 tokens Because Google set that as the "budget" and truncates accordingly. Again, call the API with an image or video and then convert those tokens to text. >https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 This is super weird, and does not remotely prove your point. I literally spend most of my days in ViTs, but thanks for the link.

famouswaffles 3y ago

Image tokens =/ Text tokens. Image tokens are patches of the image. Each image is divided into ~256 parts. Those parts are the tokens. There's no separate run to another OCR.

llm_nerd 3y ago
Completely wrong. Well, aside from the edited in bit about OCR. Of course there isn't a separate run to do OCR because that was literally the first step during image analysis. You know, before the conversion to simple tokens.
- famouswaffles 3y ago
  There's no run to any OCR, first step or not. And you have no idea what you're talking about.
  - deleted 3y ago
    [deleted]
  - llm_nerd 3y ago
    You understand that OCR is the process of extracting text from images, right? You know, such as what Gemini does, and they reference repeatedly in their paper. I have absolutely no idea why you repeatedly make some bizarre distinction about it being a "separate process". Okay, it's been fun talking to you but feel free to have the last word. Good luck.
    - famouswaffles 3y ago
      The transformer (Gemini) predicts text with image and text in the context window. That's it. OCR, Object detection etc all come from the transformer predicting text. Read the Flamingo paper.