5 ms·

I totally agree with your assessment, Nerf and Gaussian are much more interesting and I'm sure they will be huge in the future. But I don't think it will ever

by lelag 3y ago

I totally agree with your assessment, Nerf and Gaussian are much more interesting and I'm sure they will be huge in the future. But I don't think it will ever be easy to capture a clean scene randomly, at least not with anything that moves like people in the scene. The issue is that in order for the Nerf/3DGS to be clean, that is to not have too many zones where the data is not sufficient to train the model for that particular spot, you need to have pictures from every angles to train the scene. For a totally static scene, it will work: walk around the scene until you have all the angles, and you are done. The capture won't even need to be perfect because diffusion models will be able to fill in the gap for what you missed. However, if you are talking about recording a memory, presumably with several moving people and objects, it's going to be a challenge as you can't freeze time (or I can't at least). For planned production, it won't be an issue, the trick will be to setup enough cameras around the scene to shoot from all the angles, but I don't think we can expect to ever have a single magical device that you can walk around with and record Nerfs on the fly.

scheeseman486 3y ago
Instead of photographing from every angle you could capture with an array of cameras placed on the outside of a ~1m spheroid, similar to how Google captured their lightfield demos. Generating guassian splats with that data would allow for high quality, perspective correct imagery from any virtual camera placed within that sphere, sort of like a 360 degree photo (and eventually video) that looks totally convincing. Move your head outside the sphere and the effect would break, but I feel 1x1m would be enough to allow for enough freedom of movement to be comfortable. Capture could perhaps be somewhat portable with some clever engineering, think those expanding sphere ball toys.

by lelag 3y ago

scheeseman486 3y ago

Instead of photographing from every angle you could capture with an array of cameras placed on the outside of a ~1m spheroid, similar to how Google captured their lightfield demos. Generating guassian splats with that data would allow for high quality, perspective correct imagery from any virtual camera placed within that sphere, sort of like a 360 degree photo (and eventually video) that looks totally convincing. Move your head outside the sphere and the effect would break, but I feel 1x1m would be enough to allow for enough freedom of movement to be comfortable. Capture could perhaps be somewhat portable with some clever engineering, think those expanding sphere ball toys.