3 ms·
My problem is that this behavior can be attained by a shallower and non-generalizable understanding. Instead of realizing that the hand is blocked, perhaps the
by nicklecompte 2y ago
My problem is that this behavior can be attained by a shallower and non-generalizable understanding. Instead of realizing that the hand is blocked, perhaps the system's model is equivalent to the hand disappearing and reappearing with a "swipe." This understanding would not be obtained by a 3D modeling of human anatomy, but rather a hyperfocused study of yoga videos where the camera angle is just like the one shown in the paper (it is mostly the cliched camera angle that raises my suspicions). An understanding like this would not always generalize properly, e.g. instead of a hand being partially occluded in a skew video it visibly pops in and out, or the left hand strangely blends into the right.
There's a general issue with generative AI drawing "a horse riding an astronaut" - art generators still struggle to do this because they just can't generalize to odd scenarios. I strongly suspect this method has a similar issue with "interpolate the frames of this yoga video with a moving handheld camera." AFAIK these systems are not capable of learning how 3D people move when they do yoga: they learn what 2D yoga instructional videos look like, and only incidentally pick up detailed (but non-generalizable) facts about 3D motion.