5 ms·

Neural Geometric Level of Detail: Real-Time Rendering with Implicit 3D Surfaces

171 points by ArtWomb 6y ago

tovacinni 6y ago
Here's a Twitter thread that explains what this paper is about: https://twitter.com/yongyuanxi/status/1354478763065528320 https://twitter.com/yongyuanxi/status/1354478763065528320?
- baxuz 6y ago
  I'm trying to understand this but I'm not sure I'm following. As far as I understand as a complete beginner in 3D rendering: Regular 3d models/meshes are just a collection of points/triangles in a 3d plane, along with some additional data like normal orientations. For rendering, the mesh is constructed and then rendered with some perspective/distance/occlusion settings etc. for each pixel painted on screen. This format isn't a discrete collection of points but some data set where every point queried during 2d painting returns a distance (if it exists) based on some function or ML algorithm, basically returning a point cloud?
  - GreenHeuristics 6y ago
    Think of it like so: float distance_to_surface_of_model(x, y, z); For every point in 3d space you can get the distance to the surface. So, if you have a camera you can shoot a ray into the scene and pick some points on it to ask if it is inside the model. if so you paint that pixel with the surface material of the model. Then it's just a matter of picking a good way of sampling points along the camera ray. usually this distance function is some basic math (distance to sphere or so). But here they take a 3d mesh and create a neural network that can answer the question of distance to it (since meshes are very slow at that).
    - dartmoose 6y ago
      Meshes are actually extremely fast for querying distance to a surface. Even in scenes with lots of geometry, using an acceleration structure like a BVH can be extremely fast. You can even do ray tracing in real time on a GPU now (at least for direct lighting, certainly not for global illumination). By contrast, SDFs require you to do lots of expensive lookups as you slowly march along a ray until you happen to bump into a surface.
      - skye-adaire 6y ago
        This is a bit confused. Current RT implementations are faster at intersecting the mesh along the ray direction than sphere tracing (which require the expensive lookups or evaluations). The distance to the surface is independent of direction, it is only a function of position (as all SDFs are). The depth map is the distance from ray origin to intersection only, not how close that ray came to the surface at any point along it's segment.
        dartmoose 6y ago
        That's fair, my point is that if you only care about rendering something quickly then a mesh is your best bet. Are there applications you have in mind where the distance to the nearest surface is actually useful, regardless of the surface representation? As in, when would knowing the distance to the nearest surface be helpful when you're working with meshes? It seems to me you only need to know the closest distance to a surface when you have an SDF and are trying to compute an intersection along a ray. The nearest sort of application of this kind that I can think of is Walk on Spheres (cool recent paper https://www.cs.cmu.edu/~kmcrane/Projects/MonteCarloGeometryProcessing/paper.pdf https://www.cs.cmu.edu/~kmcrane/Projects/MonteCarloGeometryP...), but in that case the authors do a closest point query, not a distance to the nearest surface. There's a subtle difference here because you actually need the closest point (not just the distance) so that you can look up a boundary condition. It's not clear to me how you could make use of an SDF in this setting, although maybe there's an interesting research direction there.
        skye-adaire 6y ago
        >Are there applications you have in mind where the distance to the nearest surface is actually useful, regardless of the surface representation? Distance fields are a subset of implicit functions (see wikipedia). The use of level curves (which can be interpreted as "distance" for SDFs) is broad. You can dynamically and trivially thicken or shrink objects by adding or subtracting to the distance. Functions defined on the level curves output of an SDF can be used to map a volume to another implicit function (see nTopology). With implicit functions, the non-zero level sets are really just as important, but most focus on the zero level set in rendering. >There's a subtle difference here because you actually need the closest point (not just the distance) so that you can look up a boundary condition. Vector to Closest Point (VCP) is subtle but different in important ways. It is unsigned, and cannot indicate whether a position is inside or outside the object. Implicit modeling is not generally concerned with parameterizing the surface, since we can use volumetric, 3D procedural textures.
xpe 6y ago
After skimming here is my (imperfect) summary. Comments encouraged! Technique: "Neural Geometric LOD (level of detail)" Summary: This technique provides a machine-learning approach to give more efficient representation for 3D surfaces at interactive rates. Related Work: "Our work is most related to prior research on mesh simplification for level of detail (LOD), 3D neural shape representations, and implicit neural rendering." Method: "Our goal is to design a representation which reconstructs detailed geometry and enables continuous level of detail, all whilst being able to render at interactive rates." Experiments: "Across all [tested] datasets [(ShapeNet, Thingi10K, TurboSquid, Shadertoy)] and metrics, we achieve state-of-the-art results" relative to DeepSDF, Fourier Feature Networks, SIREN, and Neural Implicits (NI). Generalization: "surface extraction mechanism can generalize to multiple shapes, even from being trained on a single shape."
ilaksh 6y ago
They site the Occupancy Networks paper as [33]. But I thought that one also used octrees? Which they are saying is supposed to be novel.
- dartmoose 6y ago
  I believe Occupancy Networks is using octrees for surface extraction (i.e. construct a voxel grid given your SDF). See figure 2 in the Occupancy Networks Paper. On the other-hand, octrees are used in this paper to store variable resolution feature volumes (i.e. to improve the representation of the SDF). See figure 3 in Neural Geometric Detail Paper.
gdubs 6y ago
If I understand correctly, this is essentially using neural nets to compress the information representing a 3D mesh [1], similar to how neural nets can do image upscaling on 2D images? Either way, the technology of Star Trek The Next Generation went through a period in the late 90s where it all seemed ridiculous, and now we’ve come full circle and it all seems plausible again. More specifically: I’d love to see a generative adversarial network that can materialize 3D worlds in real time, like the holodeck. More and more the holodeck seems plausible as a system of nets upon nets upon nets. Nets for generating models, nets for generating stories... 1: In this case the meshes are not polygons, but rather signed distance fields. Metaballs are commonly made with this technique. Edit: thanks for catching the spelling mistake. Metaballs misspelled as Meatballs is a plague that continues for over two decades. Usually I try to avoid the trap altogether and say Implicit Surfaces :)
- harperlee 6y ago
  Metaballs also, not only meatballs ;)
- ben_w 6y ago
  For TNG-style holodecks, I think the future is smart-dust drones and beamed power.
- ryandamm 6y ago
  Re: GANNs generating 3D worlds, there are a couple startups that appear to be... well, sniffing around the edges, anyway: https://www.wunderparc.com/ https://www.wunderparc.com/ https://www.prometheanai.com/ https://www.prometheanai.com/ I haven't tried either one out; someone in my industry pointed them out to me recently.
- dartmoose 6y ago
  I don't think the point is compression, SDFs even when represented with a neural net have a much larger memory footprint and are significantly more expensive to render. The main benefit of a SDF are that you don't have topological issues to worry about, which makes them great for reconstruction problems where a surface is evolving over time.
AbrahamParangi 6y ago
I bet you could take advantage of the fact that the function generated with feedforward + relu is piecewise linear to mesh the resulting SDF really efficiently.
whelming_wave 6y ago
Not at all an expert in this, but I'm curious how this compares to non-NN solutions like https://research.nvidia.com/publication/efficient-sparse-voxel-octrees https://research.nvidia.com/publication/efficient-sparse-vox... Presumably it takes less memory, letting a more complex scene be transferred to the GPU more quickly, at the tradeoff of time spent training the model?
- skye-adaire 6y ago
  Memory savings are the main potential gain, since you won't need as deep a tree. This is not the main issue with this tech atm. Also, their distance output will be an estimate, it won't be exact, which will cause render time to underperform vs exact representations. Worse, the NN could return a value greater than actual distance, which will cause artifacts.
sxp 6y ago
If you don't have an understanding of signed distance fields, https://www.youtube.com/watch?v=8--5LwHRhjk https://www.youtube.com/watch?v=8--5LwHRhjk is an amazing demo of their power. There are more videos about the fundamentals of SDFs at https://www.iquilezles.org/live/index.htm https://www.iquilezles.org/live/index.htm The interesting thing about this paper is that they replace the mathematical SDF with a neural net that computes SDF_NN(x,y,z) => -1/1 and use this as a compression system for mesh based inputs. This feels like a graphical version of the https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize for NN-based text compression.
- folli 6y ago
  Inigo truly is a modern-day magician.
  - mortenjorck 6y ago
    Seriously. I've never seen anyone who is simultaneously this fluent in applied mathematics while having this level of visual sensibility as an illustrator. Truly a polymath.
    - timmy-turner 6y ago
      Fascinating, like watching a Bob Ross video but he is drawing with SDF formulas as his palette. Very chilling and pleasant to watch.
  - gnramires 6y ago
    It is a magic you can learn :) Seriously, start experimenting, you'd be surprised. However, overall experience with math and various mathematical experiences help so much, as well as some kind of artistic experience -- even just talking to artists, seeing a few videos on theory of artistic composition, etc. I highly recommend going down this path.
    - jmiskovic 6y ago
      I randomly stumbled into this compositional analysis and wanted to link it here: https://beleefdenachtwacht.nl/en/thema/compositie https://beleefdenachtwacht.nl/en/thema/compositie Any recommended materials? Most I found were made for photography and not for free form scene creation.
zqfm 6y ago
As an aside, this page is great, I was able to watch the videos and see all the images without needing to enable javascript for half a dozen 3rd party URLs!
etaioinshrdlu 6y ago
It combines my 2 favorite things! Deep learning and shadertoy implicit surfaces!
haxiomic 6y ago
Does anyone have an impression of how it compares to Alex Evan's SDF engine[0] used in Dreams? Is it primarily a space compression? I've been looking out for anything outside of media molecule that can come close to it's detail-representation and performance A big part of the secret sauce there seems to be what he called a Compute Shader of Doom used to optimize the brush strokes into the SDF - I'd love to know that shader works! [0] https://www.youtube.com/watch?v=u9KNtnCZDMI&ab_channel=MediaMolecule https://www.youtube.com/watch?v=u9KNtnCZDMI&ab_channel=Media...
- ArtWomb 6y ago
  Alex Evans is also now at NVidia Research. Try pinging him ;) https://twitter.com/mmalex/status/1354455745924911107 https://twitter.com/mmalex/status/1354455745924911107
dartmoose 6y ago
My understanding of this paper (please comment/correct based on your understanding): * Instead of using a single latent feature per each implicit surface, the authors propose using a "volume" of latent features per each surface. This allows for the NNs to better capture the geometric detail while remaining relatively shallow. The result is a more accurate and faster to compute neural SDF. Contrary to the claim of another comment, the neural SDF alone is not the interesting part of this paper--the prior works points to at least three other papers that have explored the idea of representing an SDF with a neural net: Park et al.'s DeepSDF https://arxiv.org/pdf/1901.05103.pdf https://arxiv.org/pdf/1901.05103.pdf, Mescheder et al's Occupancy Networks http://www.cvlibs.net/publications/Mescheder2019CVPR.pdf http://www.cvlibs.net/publications/Mescheder2019CVPR.pdf, and Chen et al's Learning Implicit Fields https://arxiv.org/pdf/1812.02822.pdf https://arxiv.org/pdf/1812.02822.pdf All very interesting papers. * When I say a "volume" of latent features I specifically mean a voxel-grid where the corners of each voxel are latent features and any position X has a corresponding feature Z which is simply the trilinear interpolation of the features on the corners of the voxel. As the authors mention, they try to keep this sparse by leaving any voxel that does not contain the surface "empty". The authors use an octree to create L different feature volumes. As L becomes larger, the resolution of the feature volume increases which means that more fine grained details can be encoded as features. Finally, the authors describe a rendering procedure that makes use of their LOD model (still need to read this part more thoroughly). Some additional thoughts: Why are SDFs useful at all? One comment suggests this is a form of "compression" but meshes have a far smaller memory footprint and are computationally less expensive to render. Ray tracing is extremely fast, largely due to the fact that as a primitive operation in graphics so much time and energy has been invested into understanding how to make it faster with various acceleration structures, like BVHs. So are SDFs actually useful? Yes. Triangle or polygon meshes are great when you have them, but are terribly challenging to work with for reconstruction tasks. For instance, you effectively have to pause occasionally during reconstruction to fix your mesh up so that it isn't complete garbage (triangles with small angles, self-intersections, extremely lopsided side lengths, etc). SDFs support arbitrary topology painlessly, which is why they show up so much in reconstruction/computer vision. So why do we need neural nets to represent them? I think the primary reason you use a neural network to represent a signed distance function is because it's a more efficient representation than storing the SDF in some sort of grid structure (maybe someone else has more thoughts on this?). As a side benefit, it can simplify any sort of differentiable rendering since the surface itself already is represented in a manner that is naturally differentiable via back-propagation.

171 points by ArtWomb 6y ago

Here's a Twitter thread that explains what this paper is about: https://twitter.com/yongyuanxi/status/1354478763065528320 https://twitter.com/yongyuanxi/status/1354478763065528320?

baxuz 6y ago
I'm trying to understand this but I'm not sure I'm following. As far as I understand as a complete beginner in 3D rendering: Regular 3d models/meshes are just a collection of points/triangles in a 3d plane, along with some additional data like normal orientations. For rendering, the mesh is constructed and then rendered with some perspective/distance/occlusion settings etc. for each pixel painted on screen. This format isn't a discrete collection of points but some data set where every point queried during 2d painting returns a distance (if it exists) based on some function or ML algorithm, basically returning a point cloud?
- GreenHeuristics 6y ago
  Think of it like so: float distance_to_surface_of_model(x, y, z); For every point in 3d space you can get the distance to the surface. So, if you have a camera you can shoot a ray into the scene and pick some points on it to ask if it is inside the model. if so you paint that pixel with the surface material of the model. Then it's just a matter of picking a good way of sampling points along the camera ray. usually this distance function is some basic math (distance to sphere or so). But here they take a 3d mesh and create a neural network that can answer the question of distance to it (since meshes are very slow at that).
  - dartmoose 6y ago
    Meshes are actually extremely fast for querying distance to a surface. Even in scenes with lots of geometry, using an acceleration structure like a BVH can be extremely fast. You can even do ray tracing in real time on a GPU now (at least for direct lighting, certainly not for global illumination). By contrast, SDFs require you to do lots of expensive lookups as you slowly march along a ray until you happen to bump into a surface.
    - skye-adaire 6y ago
      This is a bit confused. Current RT implementations are faster at intersecting the mesh along the ray direction than sphere tracing (which require the expensive lookups or evaluations). The distance to the surface is independent of direction, it is only a function of position (as all SDFs are). The depth map is the distance from ray origin to intersection only, not how close that ray came to the surface at any point along it's segment.
      - dartmoose 6y ago
        That's fair, my point is that if you only care about rendering something quickly then a mesh is your best bet. Are there applications you have in mind where the distance to the nearest surface is actually useful, regardless of the surface representation? As in, when would knowing the distance to the nearest surface be helpful when you're working with meshes? It seems to me you only need to know the closest distance to a surface when you have an SDF and are trying to compute an intersection along a ray. The nearest sort of application of this kind that I can think of is Walk on Spheres (cool recent paper https://www.cs.cmu.edu/~kmcrane/Projects/MonteCarloGeometryProcessing/paper.pdf https://www.cs.cmu.edu/~kmcrane/Projects/MonteCarloGeometryP...), but in that case the authors do a closest point query, not a distance to the nearest surface. There's a subtle difference here because you actually need the closest point (not just the distance) so that you can look up a boundary condition. It's not clear to me how you could make use of an SDF in this setting, although maybe there's an interesting research direction there.
        skye-adaire 6y ago
        >Are there applications you have in mind where the distance to the nearest surface is actually useful, regardless of the surface representation? Distance fields are a subset of implicit functions (see wikipedia). The use of level curves (which can be interpreted as "distance" for SDFs) is broad. You can dynamically and trivially thicken or shrink objects by adding or subtracting to the distance. Functions defined on the level curves output of an SDF can be used to map a volume to another implicit function (see nTopology). With implicit functions, the non-zero level sets are really just as important, but most focus on the zero level set in rendering. >There's a subtle difference here because you actually need the closest point (not just the distance) so that you can look up a boundary condition. Vector to Closest Point (VCP) is subtle but different in important ways. It is unsigned, and cannot indicate whether a position is inside or outside the object. Implicit modeling is not generally concerned with parameterizing the surface, since we can use volumetric, 3D procedural textures.

xpe 6y ago

After skimming here is my (imperfect) summary. Comments encouraged! Technique: "Neural Geometric LOD (level of detail)" Summary: This technique provides a machine-learning approach to give more efficient representation for 3D surfaces at interactive rates. Related Work: "Our work is most related to prior research on mesh simplification for level of detail (LOD), 3D neural shape representations, and implicit neural rendering." Method: "Our goal is to design a representation which reconstructs detailed geometry and enables continuous level of detail, all whilst being able to render at interactive rates." Experiments: "Across all [tested] datasets [(ShapeNet, Thingi10K, TurboSquid, Shadertoy)] and metrics, we achieve state-of-the-art results" relative to DeepSDF, Fourier Feature Networks, SIREN, and Neural Implicits (NI). Generalization: "surface extraction mechanism can generalize to multiple shapes, even from being trained on a single shape."

ilaksh 6y ago

They site the Occupancy Networks paper as [33]. But I thought that one also used octrees? Which they are saying is supposed to be novel.

dartmoose 6y ago
I believe Occupancy Networks is using octrees for surface extraction (i.e. construct a voxel grid given your SDF). See figure 2 in the Occupancy Networks Paper. On the other-hand, octrees are used in this paper to store variable resolution feature volumes (i.e. to improve the representation of the SDF). See figure 3 in Neural Geometric Detail Paper.

gdubs 6y ago

If I understand correctly, this is essentially using neural nets to compress the information representing a 3D mesh [1], similar to how neural nets can do image upscaling on 2D images? Either way, the technology of Star Trek The Next Generation went through a period in the late 90s where it all seemed ridiculous, and now we’ve come full circle and it all seems plausible again. More specifically: I’d love to see a generative adversarial network that can materialize 3D worlds in real time, like the holodeck. More and more the holodeck seems plausible as a system of nets upon nets upon nets. Nets for generating models, nets for generating stories... 1: In this case the meshes are not polygons, but rather signed distance fields. Metaballs are commonly made with this technique. Edit: thanks for catching the spelling mistake. Metaballs misspelled as Meatballs is a plague that continues for over two decades. Usually I try to avoid the trap altogether and say Implicit Surfaces :)

harperlee 6y ago
Metaballs also, not only meatballs ;)
ben_w 6y ago
For TNG-style holodecks, I think the future is smart-dust drones and beamed power.
ryandamm 6y ago
Re: GANNs generating 3D worlds, there are a couple startups that appear to be... well, sniffing around the edges, anyway: https://www.wunderparc.com/ https://www.wunderparc.com/ https://www.prometheanai.com/ https://www.prometheanai.com/ I haven't tried either one out; someone in my industry pointed them out to me recently.
dartmoose 6y ago
I don't think the point is compression, SDFs even when represented with a neural net have a much larger memory footprint and are significantly more expensive to render. The main benefit of a SDF are that you don't have topological issues to worry about, which makes them great for reconstruction problems where a surface is evolving over time.

AbrahamParangi 6y ago

I bet you could take advantage of the fact that the function generated with feedforward + relu is piecewise linear to mesh the resulting SDF really efficiently.

whelming_wave 6y ago

Not at all an expert in this, but I'm curious how this compares to non-NN solutions like https://research.nvidia.com/publication/efficient-sparse-voxel-octrees https://research.nvidia.com/publication/efficient-sparse-vox... Presumably it takes less memory, letting a more complex scene be transferred to the GPU more quickly, at the tradeoff of time spent training the model?

skye-adaire 6y ago
Memory savings are the main potential gain, since you won't need as deep a tree. This is not the main issue with this tech atm. Also, their distance output will be an estimate, it won't be exact, which will cause render time to underperform vs exact representations. Worse, the NN could return a value greater than actual distance, which will cause artifacts.

sxp 6y ago

If you don't have an understanding of signed distance fields, https://www.youtube.com/watch?v=8--5LwHRhjk https://www.youtube.com/watch?v=8--5LwHRhjk is an amazing demo of their power. There are more videos about the fundamentals of SDFs at https://www.iquilezles.org/live/index.htm https://www.iquilezles.org/live/index.htm The interesting thing about this paper is that they replace the mathematical SDF with a neural net that computes SDF_NN(x,y,z) => -1/1 and use this as a compression system for mesh based inputs. This feels like a graphical version of the https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize for NN-based text compression.

folli 6y ago
Inigo truly is a modern-day magician.
- mortenjorck 6y ago
  Seriously. I've never seen anyone who is simultaneously this fluent in applied mathematics while having this level of visual sensibility as an illustrator. Truly a polymath.
  - timmy-turner 6y ago
    Fascinating, like watching a Bob Ross video but he is drawing with SDF formulas as his palette. Very chilling and pleasant to watch.
- gnramires 6y ago
  It is a magic you can learn :) Seriously, start experimenting, you'd be surprised. However, overall experience with math and various mathematical experiences help so much, as well as some kind of artistic experience -- even just talking to artists, seeing a few videos on theory of artistic composition, etc. I highly recommend going down this path.
  - jmiskovic 6y ago
    I randomly stumbled into this compositional analysis and wanted to link it here: https://beleefdenachtwacht.nl/en/thema/compositie https://beleefdenachtwacht.nl/en/thema/compositie Any recommended materials? Most I found were made for photography and not for free form scene creation.

zqfm 6y ago

As an aside, this page is great, I was able to watch the videos and see all the images without needing to enable javascript for half a dozen 3rd party URLs!

etaioinshrdlu 6y ago

It combines my 2 favorite things! Deep learning and shadertoy implicit surfaces!

haxiomic 6y ago

Does anyone have an impression of how it compares to Alex Evan's SDF engine[0] used in Dreams? Is it primarily a space compression? I've been looking out for anything outside of media molecule that can come close to it's detail-representation and performance A big part of the secret sauce there seems to be what he called a Compute Shader of Doom used to optimize the brush strokes into the SDF - I'd love to know that shader works! [0] https://www.youtube.com/watch?v=u9KNtnCZDMI&ab_channel=MediaMolecule https://www.youtube.com/watch?v=u9KNtnCZDMI&ab_channel=Media...

ArtWomb 6y ago
Alex Evans is also now at NVidia Research. Try pinging him ;) https://twitter.com/mmalex/status/1354455745924911107 https://twitter.com/mmalex/status/1354455745924911107

dartmoose 6y ago

My understanding of this paper (please comment/correct based on your understanding): * Instead of using a single latent feature per each implicit surface, the authors propose using a "volume" of latent features per each surface. This allows for the NNs to better capture the geometric detail while remaining relatively shallow. The result is a more accurate and faster to compute neural SDF. Contrary to the claim of another comment, the neural SDF alone is not the interesting part of this paper--the prior works points to at least three other papers that have explored the idea of representing an SDF with a neural net: Park et al.'s DeepSDF https://arxiv.org/pdf/1901.05103.pdf https://arxiv.org/pdf/1901.05103.pdf, Mescheder et al's Occupancy Networks http://www.cvlibs.net/publications/Mescheder2019CVPR.pdf http://www.cvlibs.net/publications/Mescheder2019CVPR.pdf, and Chen et al's Learning Implicit Fields https://arxiv.org/pdf/1812.02822.pdf https://arxiv.org/pdf/1812.02822.pdf All very interesting papers. * When I say a "volume" of latent features I specifically mean a voxel-grid where the corners of each voxel are latent features and any position X has a corresponding feature Z which is simply the trilinear interpolation of the features on the corners of the voxel. As the authors mention, they try to keep this sparse by leaving any voxel that does not contain the surface "empty". The authors use an octree to create L different feature volumes. As L becomes larger, the resolution of the feature volume increases which means that more fine grained details can be encoded as features. Finally, the authors describe a rendering procedure that makes use of their LOD model (still need to read this part more thoroughly). Some additional thoughts: Why are SDFs useful at all? One comment suggests this is a form of "compression" but meshes have a far smaller memory footprint and are computationally less expensive to render. Ray tracing is extremely fast, largely due to the fact that as a primitive operation in graphics so much time and energy has been invested into understanding how to make it faster with various acceleration structures, like BVHs. So are SDFs actually useful? Yes. Triangle or polygon meshes are great when you have them, but are terribly challenging to work with for reconstruction tasks. For instance, you effectively have to pause occasionally during reconstruction to fix your mesh up so that it isn't complete garbage (triangles with small angles, self-intersections, extremely lopsided side lengths, etc). SDFs support arbitrary topology painlessly, which is why they show up so much in reconstruction/computer vision. So why do we need neural nets to represent them? I think the primary reason you use a neural network to represent a signed distance function is because it's a more efficient representation than storing the SDF in some sort of grid structure (maybe someone else has more thoughts on this?). As a side benefit, it can simplify any sort of differentiable rendering since the surface itself already is represented in a manner that is naturally differentiable via back-propagation.