4 ms·

That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mush

That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos. You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again. Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.

jayd16 3y ago
Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?
- sorenjan 3y ago
  They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  - jayd16 3y ago
    I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal along with the color. This should be possible as these values can be represented as a color texture, so you should be able to construct something that renders a normal map and depth map from any angle just like this renders the color currently.

jayd16 3y ago

Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?

sorenjan 3y ago
They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
- jayd16 3y ago
  I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal along with the color. This should be possible as these values can be represented as a color texture, so you should be able to construct something that renders a normal map and depth map from any angle just like this renders the color currently.