This sort of problem is really hard, but a little bit of cleverness gets you a long way. Theres some really cool information theoretic bounds that you can cook

This sort of problem is really hard, but a little bit of cleverness gets you a long way. Theres some really cool information theoretic bounds that you can cook up on how good various algorithms can do for video using whatever assumptions you have on the video. (eg does the motion respect physics? do we have sufficient sampling that we don't some ambiguous choice? etc)