5 ms·

Why use SDFs instead of NeRFs? Those were designed to be differentiable. Then you could turn the NeRF to an SDF later. Related: https://nvlabs.github.io/instan

Why use SDFs instead of NeRFs? Those were designed to be differentiable. Then you could turn the NeRF to an SDF later. Related: https://nvlabs.github.io/instant-ngp/ https://nvlabs.github.io/instant-ngp/

hansworst 4y ago
Nerfs don’t contain a surface representation, but instead contain a occupancy field to represent the shape. This has implications for rendering speed (you generally need more samples along each ray to get an accurate representation). It also has implications for uses outside of rendering, e.g. in physics simulations where you need to know exactly where the surface is. Lastly, you can use SDFs to get arbitrarily detailed meshes by sampling the field at some resolution, who which is trickier with nerfs because the surface is more fuzzy.
- WithinReason 4y ago
  In the linked video presentation [1] they automatically turn the NeRF into a mesh, so you could do the same and get faster rendering speed than SDFs. I wonder if simply enforcing less transparency towards the end of the training in a trained NeRF would help with converging to sharper object boundaries and would get the benefits of fast training without the downsides. The linked paper doesn't even discuss training times which suggests it's really bad. NeRFs a are also direction-dependent so work with specular surfaces too. [1]: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4 https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... (I edited my post so maybe you replied to an earlier version)
yolo69420 4y ago
It's useful to have differentiable versions of all sorts of rendering techniques. They can be used as input to training for whatever you want to do.
dahart 4y ago
Why use NeRFs? An important question is: what is the difference? The instant NGP learns an SDF, so it isn’t an either/or question, right? A couple of people have mentioned to me that the Berkeley paper doesn’t require a neural network, that you can use an optimizer on the density field instead. I don’t fully grasp the exact difference between those things in detail, having not implemented it myself, but using a direct optimizer without a neural network seems like an important conceptual distinction to make and worthy of research, doesn’t it? We should distill the process to its essence, break down the pieces, remove complexity and dependencies until we understand the utility and effectiveness of all the parts, shouldn’t we? Possibly relevant for this paper is also the fact that Nerfs are less than 2 years old, while SDFs in rendering are almost 30 years old, and maybe in practice more than that.
- WithinReason 4y ago
  > Why use NeRFs? Can handle view dependence, can handle transparency, probably faster to train (this paper doesn't mention training speed while the Nvidia one makes a big point about performance), probably higher resolution (comparing final outputs)
  - dahart 4y ago
    It’s important to address the question of what, exactly, is the difference between whatever two things you’re comparing here. There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution, those two processes are really more like different words used to describe the same thing. Training a neural network is an optimization. And NGPs aren’t just a neural network - it also has an explicit field representation. At this point, neural fields aren’t well defined. The Berkeley NeRFs and Nvidia NGPs are two different things in terms of what the NN lears to infer - one is density the other is SDF. And Yes, these two NN papers are learning material properties in addition to the volumetric representation, while the paper here is learning purely an SDF. That’s simply asking a different question, it’s not a matter of better or worse. The advantages depend on your goals. If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?
gfodor 4y ago
A couple of reasons: - SDFs are much more amenable to direct manipulation by humans - SDFs potentially can be decomposed into more human-understandable components - SDFs may provide a better mechanism for constructing a learned latent space for objects (see eg DeepSDF) - Some rendering engine targets may play more nicely with SDFS

hansworst 4y ago

Nerfs don’t contain a surface representation, but instead contain a occupancy field to represent the shape. This has implications for rendering speed (you generally need more samples along each ray to get an accurate representation). It also has implications for uses outside of rendering, e.g. in physics simulations where you need to know exactly where the surface is. Lastly, you can use SDFs to get arbitrarily detailed meshes by sampling the field at some resolution, who which is trickier with nerfs because the surface is more fuzzy.

WithinReason 4y ago
In the linked video presentation [1] they automatically turn the NeRF into a mesh, so you could do the same and get faster rendering speed than SDFs. I wonder if simply enforcing less transparency towards the end of the training in a trained NeRF would help with converging to sharper object boundaries and would get the benefits of fast training without the downsides. The linked paper doesn't even discuss training times which suggests it's really bad. NeRFs a are also direction-dependent so work with specular surfaces too. [1]: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4 https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... (I edited my post so maybe you replied to an earlier version)

yolo69420 4y ago

It's useful to have differentiable versions of all sorts of rendering techniques. They can be used as input to training for whatever you want to do.

dahart 4y ago

Why use NeRFs? An important question is: what is the difference? The instant NGP learns an SDF, so it isn’t an either/or question, right? A couple of people have mentioned to me that the Berkeley paper doesn’t require a neural network, that you can use an optimizer on the density field instead. I don’t fully grasp the exact difference between those things in detail, having not implemented it myself, but using a direct optimizer without a neural network seems like an important conceptual distinction to make and worthy of research, doesn’t it? We should distill the process to its essence, break down the pieces, remove complexity and dependencies until we understand the utility and effectiveness of all the parts, shouldn’t we? Possibly relevant for this paper is also the fact that Nerfs are less than 2 years old, while SDFs in rendering are almost 30 years old, and maybe in practice more than that.

WithinReason 4y ago
> Why use NeRFs? Can handle view dependence, can handle transparency, probably faster to train (this paper doesn't mention training speed while the Nvidia one makes a big point about performance), probably higher resolution (comparing final outputs)
- dahart 4y ago
  It’s important to address the question of what, exactly, is the difference between whatever two things you’re comparing here. There’s little reason to believe that optimizing an SDF and training a NeRF are any different in terms of optimization speed or resolution, those two processes are really more like different words used to describe the same thing. Training a neural network is an optimization. And NGPs aren’t just a neural network - it also has an explicit field representation. At this point, neural fields aren’t well defined. The Berkeley NeRFs and Nvidia NGPs are two different things in terms of what the NN lears to infer - one is density the other is SDF. And Yes, these two NN papers are learning material properties in addition to the volumetric representation, while the paper here is learning purely an SDF. That’s simply asking a different question, it’s not a matter of better or worse. The advantages depend on your goals. If all you want is the geometry, then material properties aren’t an advantage, and could add unnecessary complication, right?

gfodor 4y ago

A couple of reasons: - SDFs are much more amenable to direct manipulation by humans - SDFs potentially can be decomposed into more human-understandable components - SDFs may provide a better mechanism for constructing a learned latent space for objects (see eg DeepSDF) - Some rendering engine targets may play more nicely with SDFS