4 ms·

No it's individual frames https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio https://developers.googleblog.c

No it's individual frames https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio https://developers.googleblog.com/2024/02/gemini-15-availabl... "Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..." But it's very likely individual frames at 1 frame/s https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_... "Figure 5 | When prompted with a 45 minute Buster Keaton movie “Sherlock Jr." (1924) (2,674 frames at 1FPS, 684k tokens), Gemini 1.5 Pro retrieves and extracts textual information from a specific frame in and provides the corresponding timestamp. At bottom right, the model identifies a scene in the movie from a hand-drawn sketch."

simonw 3y ago
Despite that being in their blog post, I'm skeptical. I tried uploading a single frame of the video as an image and it consumed 258 tokens. The 7s video was 1,841 tokens. I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that. UPDATE ... posted that before you edited your post to link to the Gemini 1.5 report. 684,000 (total tokens for the movie) / 2,674 (their frame count for that movie) = 256 tokens - which is about the same as my 258 tokens for a single image. So I think you're right - it really does just split the video into frames and process them as separate images.
- infecto 3y ago
  Edit: Was going to post similar to your update. 1841/258 = ~7
- deleted 3y ago
  [deleted]
- Arelius 3y ago
  I mean, that's just over 7 frames, or one frame/s of video. There are likely fewer then that many I-frames in your video.
- simonw 3y ago
  Added a note about this to my post: https://simonwillison.net/2024/Feb/21/gemini-pro-video/#images-vs-video https://simonwillison.net/2024/Feb/21/gemini-pro-video/#imag...
Zetobal 3y ago
The model is fed individual frames from the movie BUT the movie is segmented into scenes. These scenes, are held in context for 5-10 scenes, depending on their length. If the video exceeds a specific length or better said a threshold of scenes it creates an index and summary. So yes technically the model looks at individual frames but it's a bit more tooling behind it.

simonw 3y ago

Despite that being in their blog post, I'm skeptical. I tried uploading a single frame of the video as an image and it consumed 258 tokens. The 7s video was 1,841 tokens. I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that. UPDATE ... posted that before you edited your post to link to the Gemini 1.5 report. 684,000 (total tokens for the movie) / 2,674 (their frame count for that movie) = 256 tokens - which is about the same as my 258 tokens for a single image. So I think you're right - it really does just split the video into frames and process them as separate images.

infecto 3y ago
Edit: Was going to post similar to your update. 1841/258 = ~7
deleted 3y ago
[deleted]
Arelius 3y ago
I mean, that's just over 7 frames, or one frame/s of video. There are likely fewer then that many I-frames in your video.
simonw 3y ago
Added a note about this to my post: https://simonwillison.net/2024/Feb/21/gemini-pro-video/#images-vs-video https://simonwillison.net/2024/Feb/21/gemini-pro-video/#imag...

Zetobal 3y ago

The model is fed individual frames from the movie BUT the movie is segmented into scenes. These scenes, are held in context for 5-10 scenes, depending on their length. If the video exceeds a specific length or better said a threshold of scenes it creates an index and summary. So yes technically the model looks at individual frames but it's a bit more tooling behind it.