I imagine it would be hard, since these are usually captured with a single camera moving around and taking lots of photos. Here's an edge case I can imagine fo

I imagine it would be hard, since these are usually captured with a single camera moving around and taking lots of photos. Here's an edge case I can imagine for dynamic lighting - Say you capture a scene indoors, and a table casts a dark shadow on the floor. But the NeRFs don't try to understand light sources and shadows yet, so it wouldn't know whether the floor is painted black, or a white surface shadowed by the table, or if there's actually a blue Stanford bunny hiding in the shadows. The 3D scanning rigs that capture small objects like people's faces handle this by manipulating lighting and sampling the BDRF directly. If you can't manipulate the lighting, you can probably guess a BDRF, but there will be limits. Re-animating might be easy. But capturing an animation, I think you'd need multiple cameras or you'd have to settle for guesswork, like a neural network that can hallucinate the hidden side of a person based on the fact that they're a person. If you point the camera at someone who is walking, you'll get a good view of them from one side, but when you wind the video back, the network won't know what their far side looks like at all. A few years ago Intel had a project to capture an animated scene with multiple cameras. The pitch was something like, "Just film everything, and you can position the camera in post-processing." I think they wanted it for football games, but I never heard of it shipping. And again, multiple cameras. Matrix-style.