4 ms·

> The test works by inserting a semantically important "needle" frame at random positions in long videos, which the system must then find and analyze. This see

> The test works by inserting a semantically important "needle" frame at random positions in long videos, which the system must then find and analyze. This seems to be somewhat unwise. Such an insertion would qualify as an anomaly. And if it's also trained that way, would you not train the model to find artificial frames where they don't belong? Would it not have been better to find a set of videos where something specific (common, rare, surprising, etc) happens at some time and ask the model about that?

bigmadshoe 11mo ago
Yeah the needle in a haystack tests are so stupid. It seems clear with LLMs that performance degrades massively with context size, yet those tests claim the model performs perfectly.
- patates 11mo ago
  As someone who abuses gemini regularly with a 90% full context, the model performance does degrade for sure but I wouldn't call it massively. I can't show any evidence as I don't have such tests, but it's like coding normally vs coding after a beer or two. For the massive effect, fill it 95% and we're talking vodka shots. 99%? A zombie who can code. But perhaps that's not fair when you have 1M token context size.
- oceansweep 11mo ago
  https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...
IanCal 11mo ago
That rather depends on exactly how this is done, although it's a useful upper bound for many tasks either way. You could say the same for images and yet due to the way some work they straight up cannot see in certain ways. This could describe adding a frame of nonsense into an existing video. It also could describe finding a semantically useful thing in an actual video, where the exact location is randomised by looking at different time crops of the video. For example, finding a book on a desk in a video that's only there in a panning shot, and you then see if it can find it in a 10s cut, 20s cut, 10 minute cut, etc, and near the start/middle/end. Here's the paper: https://arxiv.org/pdf/2511.21631 https://arxiv.org/pdf/2511.21631 > To evaluate the model’s capability in processing long-context inputs, we construct a video “Needle-ina-Haystack” evaluation on Qwen3-VL-235B-A22B-Instruct. In this task, a semantically salient “needle” frame—containing critical visual evidence—is inserted at varying temporal positions within a long video. The model is then tasked with accurately locating the target frame from the long video and answering the corresponding question. During evaluation, videos are uniformly sampled at 1 FPS, and frame resolution is dynamically adjusted to maintain a constant visual token budget. This potentially sounds more like the former, but I can't find more accurate information on how this works. Regardless I'd say again that while not the whole story things like this really are useful to know, and can be very important to test - it's really not a given that models can always find anything in their context window, perhaps even more so for video.

bigmadshoe 11mo ago

Yeah the needle in a haystack tests are so stupid. It seems clear with LLMs that performance degrades massively with context size, yet those tests claim the model performs perfectly.

patates 11mo ago
As someone who abuses gemini regularly with a 90% full context, the model performance does degrade for sure but I wouldn't call it massively. I can't show any evidence as I don't have such tests, but it's like coding normally vs coding after a beer or two. For the massive effect, fill it 95% and we're talking vodka shots. 99%? A zombie who can code. But perhaps that's not fair when you have 1M token context size.
oceansweep 11mo ago
https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/o...

IanCal 11mo ago

That rather depends on exactly how this is done, although it's a useful upper bound for many tasks either way. You could say the same for images and yet due to the way some work they straight up cannot see in certain ways. This could describe adding a frame of nonsense into an existing video. It also could describe finding a semantically useful thing in an actual video, where the exact location is randomised by looking at different time crops of the video. For example, finding a book on a desk in a video that's only there in a panning shot, and you then see if it can find it in a 10s cut, 20s cut, 10 minute cut, etc, and near the start/middle/end. Here's the paper: https://arxiv.org/pdf/2511.21631 https://arxiv.org/pdf/2511.21631 > To evaluate the model’s capability in processing long-context inputs, we construct a video “Needle-ina-Haystack” evaluation on Qwen3-VL-235B-A22B-Instruct. In this task, a semantically salient “needle” frame—containing critical visual evidence—is inserted at varying temporal positions within a long video. The model is then tasked with accurately locating the target frame from the long video and answering the corresponding question. During evaluation, videos are uniformly sampled at 1 FPS, and frame resolution is dynamically adjusted to maintain a constant visual token budget. This potentially sounds more like the former, but I can't find more accurate information on how this works. Regardless I'd say again that while not the whole story things like this really are useful to know, and can be very important to test - it's really not a given that models can always find anything in their context window, perhaps even more so for video.