3 ms·

Great read - I learned quite a bit about this. The only quibble I had is at the end, and it's a very general layperson's view: > But what’s even more impressiv

Great read - I learned quite a bit about this. The only quibble I had is at the end, and it's a very general layperson's view: > But what’s even more impressive - extremely impressive - is that the system decided that the body would go up before going back down between these two poses! (Which is why it’s having trouble with the right arm in the first place! A feature matching system wouldn’t have this problem, because it wouldn’t realize that in the middle position, the body would go up, and the right arm would have to be somewhere. Struggling with things not visible in either input keyframe is a good problem to have - it’s evidence of knowing these things exist, which demonstrates quite the capabilities!).... This system clearly learned a lot about three-dimensional real-world movement behind the 2D images it’s asked to interpolate between. I think that's an awfully strong conclusion to draw from this paper - the authors certainly don't make that claim. The "null hypothesis" should be that most generative video models have a ton of yoga instruction videos shot very similarly to the example shown, and here the AI is simply repeating similar frames from similar videos. Since this most likely wouldn't generalize to yoga videos shot at a skew angle, it's hard to conclude that the system learned anything about 3D real-world movement. Maybe it did! But the study authors didn't come to that conclusion, and since their technique is actually model-independent, they wouldn't be in a good position to check for data contamination / overfitting / etc. The authors seem to think the value of their work is that generative video AI is by default past->future but you can do future->past without changing the underlying model, and use that to smooth out issues in interpolation. I just don't think there's any rational basis for generalizing this to understanding 3D space itself. This isn't a criticism of the paper - the work seems clever but the paper is not very detailed and they haven't released the code yet. And my complaint is only a minor editorial comment on an otherwise excellent writeup. But I am wondering if the author might have been bedazzled by a few impressive results.

yosefk 2y ago
You're technically correct, there's no basis to argue that a "3D representation" was learned as opposed to "enough 2D projections to handle the inputs in question." That said, the hand which did not exist in either of the original 2D frames makes an appearance. I think calling wherever it was pulled out of "the 3rd dimension" is not wrong; it was occluded in both 2D inputs and functionally you had to know about its existence in the 3rd dimension to show it even if you technically did it by learning how pixels look across frames. You can also see much more 3D ish things in the paper, with 2 angles of a room and video created moving the camera between them. Of course in some sense it adds to my point without detracting from yours...
- nicklecompte 2y ago
  My problem is that this behavior can be attained by a shallower and non-generalizable understanding. Instead of realizing that the hand is blocked, perhaps the system's model is equivalent to the hand disappearing and reappearing with a "swipe." This understanding would not be obtained by a 3D modeling of human anatomy, but rather a hyperfocused study of yoga videos where the camera angle is just like the one shown in the paper (it is mostly the cliched camera angle that raises my suspicions). An understanding like this would not always generalize properly, e.g. instead of a hand being partially occluded in a skew video it visibly pops in and out, or the left hand strangely blends into the right. There's a general issue with generative AI drawing "a horse riding an astronaut" - art generators still struggle to do this because they just can't generalize to odd scenarios. I strongly suspect this method has a similar issue with "interpolate the frames of this yoga video with a moving handheld camera." AFAIK these systems are not capable of learning how 3D people move when they do yoga: they learn what 2D yoga instructional videos look like, and only incidentally pick up detailed (but non-generalizable) facts about 3D motion.

yosefk 2y ago

You're technically correct, there's no basis to argue that a "3D representation" was learned as opposed to "enough 2D projections to handle the inputs in question." That said, the hand which did not exist in either of the original 2D frames makes an appearance. I think calling wherever it was pulled out of "the 3rd dimension" is not wrong; it was occluded in both 2D inputs and functionally you had to know about its existence in the 3rd dimension to show it even if you technically did it by learning how pixels look across frames. You can also see much more 3D ish things in the paper, with 2 angles of a room and video created moving the camera between them. Of course in some sense it adds to my point without detracting from yours...

nicklecompte 2y ago
My problem is that this behavior can be attained by a shallower and non-generalizable understanding. Instead of realizing that the hand is blocked, perhaps the system's model is equivalent to the hand disappearing and reappearing with a "swipe." This understanding would not be obtained by a 3D modeling of human anatomy, but rather a hyperfocused study of yoga videos where the camera angle is just like the one shown in the paper (it is mostly the cliched camera angle that raises my suspicions). An understanding like this would not always generalize properly, e.g. instead of a hand being partially occluded in a skew video it visibly pops in and out, or the left hand strangely blends into the right. There's a general issue with generative AI drawing "a horse riding an astronaut" - art generators still struggle to do this because they just can't generalize to odd scenarios. I strongly suspect this method has a similar issue with "interpolate the frames of this yoga video with a moving handheld camera." AFAIK these systems are not capable of learning how 3D people move when they do yoga: they learn what 2D yoga instructional videos look like, and only incidentally pick up detailed (but non-generalizable) facts about 3D motion.