I think video codecs do much more than just store differences between frames. They detect motion and output instructions like "move these pixels 8 to the left

I think video codecs do much more than just store differences between frames. They detect motion and output instructions like "move these pixels 8 to the left and 2 up". Representing that as a difference would be more data.