5 ms·

I am very uneducated in this so please forgive my ignorance. These videos always look so insanely cool which I really like and from what I understand the scenes

I am very uneducated in this so please forgive my ignorance. These videos always look so insanely cool which I really like and from what I understand the scenes/radiance fields are always static and have lighting baked in. Is there a chance this might be turned into something that can be dynamically lit and support motion?

Lichtso 3y ago
That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos. You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again. Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.
- jayd16 3y ago
  Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?
  - sorenjan 3y ago
    They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
    - jayd16 3y ago
      I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal along with the color. This should be possible as these values can be represented as a color texture, so you should be able to construct something that renders a normal map and depth map from any angle just like this renders the color currently.
ReactiveJelly 3y ago
I imagine it would be hard, since these are usually captured with a single camera moving around and taking lots of photos. Here's an edge case I can imagine for dynamic lighting - Say you capture a scene indoors, and a table casts a dark shadow on the floor. But the NeRFs don't try to understand light sources and shadows yet, so it wouldn't know whether the floor is painted black, or a white surface shadowed by the table, or if there's actually a blue Stanford bunny hiding in the shadows. The 3D scanning rigs that capture small objects like people's faces handle this by manipulating lighting and sampling the BDRF directly. If you can't manipulate the lighting, you can probably guess a BDRF, but there will be limits. Re-animating might be easy. But capturing an animation, I think you'd need multiple cameras or you'd have to settle for guesswork, like a neural network that can hallucinate the hidden side of a person based on the fact that they're a person. If you point the camera at someone who is walking, you'll get a good view of them from one side, but when you wind the video back, the network won't know what their far side looks like at all. A few years ago Intel had a project to capture an animated scene with multiple cameras. The pitch was something like, "Just film everything, and you can position the camera in post-processing." I think they wanted it for football games, but I never heard of it shipping. And again, multiple cameras. Matrix-style.
sillysaurusx 3y ago
Sure. One way is to gather splats under various real world light conditions, then map those to the closest simulated light condition. (I.e. make the data animate over time of day.) The data requirements might become massive, but there are ways to do the interpolation where it isn’t so bad. If a static scene is 2GB, you should be able to get to a rough time of day approximation in less than 16GB, which is renderable on modern GPUs. Then it’s “just” a matter of spending several years optimizing it while waiting for H100s to become consumer grade devices.
- blovescoffee 3y ago
  It’s not actually that difficult. By differentiating on the spherical harmonics of each point/Gaussian we can approximate materials and their response to lighting.
  - sillysaurusx 3y ago
    Sure, if you want to reinvent N dot L. (In other words, yes, you can do that, but then the result will look just as fake as every other “photo realistic” scheme.) The only hope is to measure actual photons hitting actual sensors, which is why Gaussian splatting looks so real to begin with.
    - blovescoffee 3y ago
      > gather splats under various real world light conditions, then map those to the closest simulated light condition is a better alternative from your perspective?
      - sillysaurusx 3y ago
        Oh yes. The key to making realistic-looking video is to sample from the real world. The more closely you do that, the more realistic it looks. The limit case is a phone camera recording a video.
gecko39 3y ago
For the motion part, there is already an extension to gaussian splatting for moving stuff: https://dynamic3dgaussians.github.io/ https://dynamic3dgaussians.github.io/ You can also just string a bunch of them together to create an animation: https://twitter.com/8Infinite8/status/1699460316529090568 https://twitter.com/8Infinite8/status/1699460316529090568 For relighting, there are lots of NeRF variants that do this -- it should be possible to optimize material parameters for the splats.

Lichtso 3y ago

That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos. You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again. Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.

jayd16 3y ago
Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?
- sorenjan 3y ago
  They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  - jayd16 3y ago
    I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal along with the color. This should be possible as these values can be represented as a color texture, so you should be able to construct something that renders a normal map and depth map from any angle just like this renders the color currently.

ReactiveJelly 3y ago

I imagine it would be hard, since these are usually captured with a single camera moving around and taking lots of photos. Here's an edge case I can imagine for dynamic lighting - Say you capture a scene indoors, and a table casts a dark shadow on the floor. But the NeRFs don't try to understand light sources and shadows yet, so it wouldn't know whether the floor is painted black, or a white surface shadowed by the table, or if there's actually a blue Stanford bunny hiding in the shadows. The 3D scanning rigs that capture small objects like people's faces handle this by manipulating lighting and sampling the BDRF directly. If you can't manipulate the lighting, you can probably guess a BDRF, but there will be limits. Re-animating might be easy. But capturing an animation, I think you'd need multiple cameras or you'd have to settle for guesswork, like a neural network that can hallucinate the hidden side of a person based on the fact that they're a person. If you point the camera at someone who is walking, you'll get a good view of them from one side, but when you wind the video back, the network won't know what their far side looks like at all. A few years ago Intel had a project to capture an animated scene with multiple cameras. The pitch was something like, "Just film everything, and you can position the camera in post-processing." I think they wanted it for football games, but I never heard of it shipping. And again, multiple cameras. Matrix-style.

sillysaurusx 3y ago

Sure. One way is to gather splats under various real world light conditions, then map those to the closest simulated light condition. (I.e. make the data animate over time of day.) The data requirements might become massive, but there are ways to do the interpolation where it isn’t so bad. If a static scene is 2GB, you should be able to get to a rough time of day approximation in less than 16GB, which is renderable on modern GPUs. Then it’s “just” a matter of spending several years optimizing it while waiting for H100s to become consumer grade devices.

blovescoffee 3y ago
It’s not actually that difficult. By differentiating on the spherical harmonics of each point/Gaussian we can approximate materials and their response to lighting.
- sillysaurusx 3y ago
  Sure, if you want to reinvent N dot L. (In other words, yes, you can do that, but then the result will look just as fake as every other “photo realistic” scheme.) The only hope is to measure actual photons hitting actual sensors, which is why Gaussian splatting looks so real to begin with.
  - blovescoffee 3y ago
    > gather splats under various real world light conditions, then map those to the closest simulated light condition is a better alternative from your perspective?
    - sillysaurusx 3y ago
      Oh yes. The key to making realistic-looking video is to sample from the real world. The more closely you do that, the more realistic it looks. The limit case is a phone camera recording a video.

gecko39 3y ago

For the motion part, there is already an extension to gaussian splatting for moving stuff: https://dynamic3dgaussians.github.io/ https://dynamic3dgaussians.github.io/ You can also just string a bunch of them together to create an animation: https://twitter.com/8Infinite8/status/1699460316529090568 https://twitter.com/8Infinite8/status/1699460316529090568 For relighting, there are lots of NeRF variants that do this -- it should be possible to optimize material parameters for the splats.