There is a significant difference. Video has a temporal component (frames tend to be correlated with previous ones), and vision LLMs do not have some sort of hi

There is a significant difference. Video has a temporal component (frames tend to be correlated with previous ones), and vision LLMs do not have some sort of hidden states to keep track of that temporal component. Using captions to bridge this only works to a certain extent (you're giving text descriptions of what happened in the past, not what had _actually_ happened).