This thought just occurred to me: would it make sense to train a model to recognise vector encoded video frames? I completely forgot how video encoders work, b

This thought just occurred to me: would it make sense to train a model to recognise vector encoded video frames? I completely forgot how video encoders work, but I do remember that some encode the differences between one frame and the next with vectores of the "motion" of pixels. The training data would be comprised of frames from a video with labels assigned to time ranges across the length of the video. The we could feed a video stream to a model and it would learn to not only recognise still images, but also motion across time. Make it fast enough and we would have near real time inference of video. If this works, maybe an extension to this model would be to accept its previous inference result as an input to the next frame inference request. Then we'd have results like "a person entered the bright light seen of a sunny day in the country side".