7 ms·

Gaussian splatting is pretty cool

324 points by signa11 3y ago

joshvm 3y ago
Does Plenoxels also count? I believe they also use a similar approach https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/
- tayistay 3y ago
  Various comparisons with plenoxels are in the paper.
- mkaic 3y ago
  This paper mentions Plenoxels in the introduction and beats it.
aiddun 3y ago
Could someone please explain to me what the advantage of this is over traditional photogrammetry?
eloop 3y ago
Splatting for volume rendering is quite old - Westover, Lee Alan (July 1991). "SPLATTING: A Parallel, Feed-Forward Volume Rendering Algorithm"
- andybak 3y ago
  Are you saying there's nothing new in this or that it's building on an already established approach? I think the latter is not something anyone is trying to hide - while the former doesn't seem correct.
  - corysama 3y ago
    What’s new in the paper is the technique to generate the gaussians from photos. The output that is used to render uses a technique that is a decade or three old, but not terribly practical until recently. For me, a rendering guy, that’s great! The data used at render time is very simple and flexible. Simpler than triangles even when you get into non-trivial operations.
    - andybak 3y ago
      Is the decades-old type of splatting view-dependent still?
      - corysama 3y ago
        The view-dependent bit reflects the simplicity of the data. The data is similar to an oriented, stretchy box that you interpret as an oriented, stretchy, fuzzy ball (the Gaussian field). What you do from there is up to you. Classically, researchers were interested in just using the splats at all. So, they just assigned a single color to each one. This paper assigns a spherical harmonic-based color sphere instead. That gives it the view angle -> color function. There is a second paper focused on moving splats. It just uses solid colored splats. You could instead associate albedo/specular/roughness from real time materials and do real time lighting. But, you’d have to figure out how to generate/capture those values.
- nsajko 3y ago
  Yeah, it seems like the authors of the new paper may have missed much relevant existing research?
  - tayistay 3y ago
    They haven't. You can't get a paper into SIGGRAPH that misses relevant existing research.
- mikhailfranco 3y ago
  Yes, I used to think of it as throw Gaussian snowballs at the window and see what sticks. P.S. Not sure where the insight came from, but I was living near Boston at the time, and had young children :)
evilhackerdude 3y ago
I'm looking forward to the first native optimized WebGPU implementation of 3DGS rendering. I'm also curious how scene data could be compressed and decompressed efficiently.
- raphlinus 3y ago
  I'm also looking forward to it. One of the big challenges is the sorting, for which I'm unaware of a good WebGPU implementation. I have some more notes on this question in a Zulip thread[1]. [1]: https://xi.zulipchat.com/#narrow/stream/197075-gpu/topic/Gaussian.20splatting https://xi.zulipchat.com/#narrow/stream/197075-gpu/topic/Gau...
  - boywitharupee 3y ago
    Does the sorting need to be done in the renderer, or is that part of the training process?
    - raphlinus 3y ago
      It needs to be done in the renderer. I think it's doable though, the FidelityFX library looks like it can be ported, it'll just run a bit slow because of the lack of subgroups. This particular library isn't based on a fancy scan implementation, as the state-of-the-art CUDA implementations are. There's a bit more followup in the linked Zulip thread.
- blovescoffee 3y ago
  I’m working on this right now!
sneak 3y ago
What's the state of the art for structure from motion? Given video of a space, how does one practically turn it into a 3d scene today?
- naillo 3y ago
  Literally gaussian splatting (and nerfs)
- andybak 3y ago
  Video is just a bunch of still images (except potentially with degraded quality from resolution and/or motion blur) so it's much the same as asking "how does one practically capture a 3d scene" without the video part. Regular photogrammetry - but with interest and practicality gradually increasing for NeRF style approaches. (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?)
  - radarsat1 3y ago
    > (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?) Can I ask why you say this? Camera motion models including Kalman filters of various kinds and constraints on derivatives etc, are absolutely used in SLAM and photogrammetry afaik, and have been for a long time.
    - andybak 3y ago
      OK maybe it's my limited experience but last time i tried it, video was treated as "extract frames and then only use every 50th (or so)" This was a couple of years ago probably. Meshroom and Reality Capture. Maybe I was mistaken or maybe they've improved? Can you point me to software that does use video for photogrammetry in the way you describe?
eurekin 3y ago
Would be cool to see those splats in action from a second, free to move camera
- arijun 3y ago
  What’s the difference between a free-to-move camera and one on a preset track if they’re rendering at 100fps?
  - eurekin 3y ago
    To see the blobs forming, moving, changing
    - arijun 3y ago
      But the 3D blobs don’t move, do they? Only the splats do.
itronitron 3y ago
Wow, the referenced Ecstatica game is wild.
- pjs_ 3y ago
  Ecstatica II was an extraordinary game - funny, weird, violent, amazing sound, and super atmospheric.
- pavlov 3y ago
  For a second I thought it's a really well done fake. The art style looked so obviously beyond 1994 software rendering. But then you start to notice that the backgrounds are all static 3D renderings and there are very few objects in motion after all, so it's certainly feasible on a 486. Maybe you could say it's basically a 3D version of Alone in the Dark using ellipsoids instead of polygons as the primitive. The quality of the animation is unusually good for the time, which is really what creates the coherent illusion. Wikipedia says the game had just one developer paired with a "film animation expert", which makes a lot of sense. https://en.wikipedia.org/wiki/Ecstatica https://en.wikipedia.org/wiki/Ecstatica I'm shocked I've never heard of this game until now.
nwatson 3y ago
Here's a video that may or may not use these techniques, I'd be curious for those who know to judge ... https://youtu.be/dXLCHvRsgRQ?si=wkniCf-7wNPt8DGj https://youtu.be/dXLCHvRsgRQ?si=wkniCf-7wNPt8DGj There's coherence frame-to-frame of the artifacts used to render, but they may be 2D "watercolor" splotches, and some seem very manually seeded, e.g. the shapes used to fill in irises for eyes. In some scenes though the blobs seem to be 3D-represented.
sorenjan 3y ago
I recently came across this video showing how to use Gaussian splatting: Getting Started With 3D Gaussian Splats for Windows (Beginner Guide) - https://www.youtube.com/watch?v=UXtuigy_wYc https://www.youtube.com/watch?v=UXtuigy_wYc
radarsat1 3y ago
> This is using gradient descent and “differentiable rendering” and all the other things that are way over my head. then.. > And finally, they have resisted the temptation to do “neural” anything ;) So, they are doing something similar to NeRF, in a sense, but using a different basis function, and a slightly different target. The whole "neural" or not "neural" is just about what you are optimizing, but it's not that different, conceptually -- they are optimizing a data-driven approximator of a 3d scene-related variable based on images. The big difference of course, based on reading this, is that NeRF models the whole light transmission function ("radiance field") whereas this seems to model only the boundary conditions (what the light "hits"). So given the different modelling target, then yes, a different, and perhaps simpler, representation basis definitely seems warranted, and is shown here to give good results. But the comment, "they avoided neural anything" feels a bit smug, as if "avoiding the hype" is a laudable goal on its own merits. As if people are using neural networks for no reason but because they're cool, and not because they're actually an appropriate solution that are shown to work really well in practice. They surely are cool, don't get me wrong, but the hype is so often justified, because neural networks are really good function approximators. They are also not just one thing -- the choice and breadth of architectures that we happen to call "neural networks" is huge, so explicitly avoiding this huge potential solution space for some reason without good reason feels pretty specious; heck you could even consider having a Gaussian "output layer" to a neural network, which is called an RBF, being something like this "splat" approach. (This is in response to the blog post, not the paper -- I'm sure the paper has plenty of justification for modeling choices.) Having said that, of course it's interesting to also explore simpler, perhaps more efficient approaches, but I don't see "resisting using neural anything" as a good thing, just because they are popular. It ignores that they are popular for a reason. (nb: This assessment is just based on the blog post and giving my impression from the tone of the introduction, I haven't read the paper yet, so might not be accurate with respect to the paper contents. I just found this off-hand comment in the post worth reflecting on.)
- graffix 3y ago
  > I don't see "resisting using neural anything" as a good thing, just because they are popular I read Aras' quip as more narrowly technical. OFC it's ambiguous and you'd have to ask the man himself. The gist of NeRF is to obtain an NN representation of a 5D light field (3D position + 2D direction) from samples (photographs) of the real-world light field. Alarm bells ring already- 5 dimensions isn't that many! Considering NeRF has always used a low-rank spherical harmonic representation of the directional domain, it's even more like 3D-and-change. To reconstruct a function of such low dimensionality, why choose an NN? Then at inference time, for each pixel, you have sample the NN repeatedly over the view ray. This part is exceedingly silly, as compact representations of light fields are a solved bread-and-butter problem in graphics. Later on Plenoxels explicitly took the "Ne" out of "NeRF", giving far higher training and inference performance (also mentioned ITT). To be fair, and later still, Nvidia somewhat redeemed NNs here with Instant NeRF: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... ...where the twist was to interpolate fancy input emeddings, which are run through a tiny NN. That tininess is important, as the need to fetch NN weights from VRAM would kick NNs right off the Pareto frontier. Zooming out, NNs have only seen wide adoption in graphics engineering for reconstruction from sparse data (inc. denoising). Makes sense, as that's a high-dimensional problem. Still, beware that the NN solutions rarely blow handmade algorithms out of the water. I also think using tiny NNs for compression- closely related to reconstruction- has a future too. Beyond that, if NNs were to set the graphics world ablaze, it would've happened by now. Lots of graphics engineering is just approximating functions, so it's natural NNs have some place here. However, our functions tend to be more understandable, tractable, malleable. It's not an application domain where it's virtually impossible to write an algorithmic solution by hand (let alone one that performs well), like natural language understanding.
Escapado 3y ago
I am very uneducated in this so please forgive my ignorance. These videos always look so insanely cool which I really like and from what I understand the scenes/radiance fields are always static and have lighting baked in. Is there a chance this might be turned into something that can be dynamically lit and support motion?
- Lichtso 3y ago
  That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos. You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again. Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.
  - jayd16 3y ago
    Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?
    - sorenjan 3y ago
      They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
9dev 3y ago
It's always fun when science exceeds science fiction's expectations. In this case, I was immediately thinking of the Braindance concept[1] in Cyberpunk 2077, which - more or less - allows to move around in the visual memories of another person, limited by that person's perception during that scene. When moving the camera to a perspective distinct from the original one, the vision disintegrates into kind of three-dimensional pixel blobs, somewhat similar to the notion of blobs here - just a lot less polished, surprisingly, compared to this paper. [1]: https://steelseries.com/blog/how-to-braindance-cyberpunk-2077-428 https://steelseries.com/blog/how-to-braindance-cyberpunk-207...
Macuyiko 3y ago
Very cool work. As a bit of tangent (but wondering whether someone can answer) - the article also makes mention of point based rendering and indeed the fact is has been a staple of particle systems for a long time. However, especially with recent games, I have noticed (purely subjectively) a very subtle shift to a new style of particle systems which are on the one hand fully point oriented (compared to (textured) fragments) but on the other behave more like a physics systems. Examples: - Hogwarts (heavily): https://www.gamespot.com/a/uploads/original/1816/18167535/4097726-hogwartslegacybestspellsallspellsguideunforgivablecurses-1.jpg https://www.gamespot.com/a/uploads/original/1816/18167535/40... - Forspoken (heavily): https://oyster.ignimgs.com/mediawiki/apis.ign.com/project-athia/0/04/FdlsymrXwAgCXE4.jpg https://oyster.ignimgs.com/mediawiki/apis.ign.com/project-at... - Starfield (though more rarely): https://dotesports.com/wp-content/uploads/2023/08/temple-location-Starfield.png https://dotesports.com/wp-content/uploads/2023/08/temple-loc... - AC6 - FF16 (heavily) It's more obvious when you see it 'in motion'. The common denominator seems to be particles as colored transparent points with physics. Especially on console systems it seems that developers are using this for very cheap (CPU-wise, all on GPU) effects. Anyone in gamedev who has some insight in this?
- lelandbatey 3y ago
  I remember that kind of "particle effect" originally being shown off ~11 years ago by the Unreal Engine 4 demo called "Elemental": https://youtu.be/dD9CPqSKjTU https://youtu.be/dD9CPqSKjTU I don't think they're "from" that, but that's the first landmark I can point to.
- deleted 3y ago
  [deleted]
- tbalsam 3y ago
  I've certainly seen game engines trying to advertise much more-involved particle systems over the last few years, that is for sure. ;PPPP Which in some ways I guess makes sense, as the shift towards AI has heavily pushed GPUs almost towards a "sizeable-reduced-instruction-set-GPU-in-GPU" approach with RT cores/tensor cores, etc.... ;P Side note as well, in/from my experience at least, I think these systems may not be as hard to render as it is for the author. A few years ago, someone (at NVIDIA I think?) wrote kernels to move the SE(3) kernels to tensor cores, so I wouldn't be shocked if some of that could be ported to the spherical harmonics portion of the Gaussian splatting during both compression and runtime: https://developer.nvidia.com/blog/accelerating-se3-transformers-training-using-an-nvidia-open-source-model-implementation/ https://developer.nvidia.com/blog/accelerating-se3-transform... Also, side side note, gaussian splatting should be quite efficient...I think? Due to technically always having support in 3D space (and hopefully not too much of a problem with having good support in 3D space). This should mean that even 'sloppy', quick-conversion calculations should work pretty decently in the end. I say all of this knowing very little about most optimizations like billboarding, how things like nanite work, etc, etc. I do like it tho! ;PPPP
jayd16 3y ago
Reminds me of Google's Seurat. https://developers.google.com/vr/discover/seurat https://developers.google.com/vr/discover/seurat
jjcm 3y ago
I was super excited to see this when the Siggraph paper first dropped. For the past decade or so, I've been taking pictures of any room I've lived in from hundreds of angles, with the intention of being able to recreate these in 3d at some point in the future. Gaussian splatting is the first technique I've seen that feels like it can recreate it in a way that feels almost-real. Once the tools around this mature a bit more, I'm super excited to revisit those old rooms and be hit with a wave of nostalgia.
phkahler 3y ago
I rendered voxel iso-surfaces using blobs in the late 90's. I'd scan a full 3d voxel array for surface voxels, use the local density gradient to compute a normal, quantize the normal to one of 240. Then I'd make chains of surface voxels using a table of displacement vectors. Most voxels were 2 bytes - a displacement index (from previous voxel) and a normal vector (index). Lighting was computed for all 240 normal vectors and stuffed in a lookup. Software could paint all these little colored circles in a z-buffer very fast. The only real bummer back then was that it had to be non-perspective. The displacement vectors were coverted to screen space per-frame, so it was paint a blob, offset, lookup color, paint another. One displacement value was used to indicate the chain ended and I needed to store an absolute position next, but most voxels were just 2 bytes.

324 points by signa11 3y ago

Does Plenoxels also count? I believe they also use a similar approach https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/

tayistay 3y ago
Various comparisons with plenoxels are in the paper.
mkaic 3y ago
This paper mentions Plenoxels in the introduction and beats it.

aiddun 3y ago

Could someone please explain to me what the advantage of this is over traditional photogrammetry?

eloop 3y ago

Splatting for volume rendering is quite old - Westover, Lee Alan (July 1991). "SPLATTING: A Parallel, Feed-Forward Volume Rendering Algorithm"

andybak 3y ago
Are you saying there's nothing new in this or that it's building on an already established approach? I think the latter is not something anyone is trying to hide - while the former doesn't seem correct.
- corysama 3y ago
  What’s new in the paper is the technique to generate the gaussians from photos. The output that is used to render uses a technique that is a decade or three old, but not terribly practical until recently. For me, a rendering guy, that’s great! The data used at render time is very simple and flexible. Simpler than triangles even when you get into non-trivial operations.
  - andybak 3y ago
    Is the decades-old type of splatting view-dependent still?
    - corysama 3y ago
      The view-dependent bit reflects the simplicity of the data. The data is similar to an oriented, stretchy box that you interpret as an oriented, stretchy, fuzzy ball (the Gaussian field). What you do from there is up to you. Classically, researchers were interested in just using the splats at all. So, they just assigned a single color to each one. This paper assigns a spherical harmonic-based color sphere instead. That gives it the view angle -> color function. There is a second paper focused on moving splats. It just uses solid colored splats. You could instead associate albedo/specular/roughness from real time materials and do real time lighting. But, you’d have to figure out how to generate/capture those values.
nsajko 3y ago
Yeah, it seems like the authors of the new paper may have missed much relevant existing research?
- tayistay 3y ago
  They haven't. You can't get a paper into SIGGRAPH that misses relevant existing research.
mikhailfranco 3y ago
Yes, I used to think of it as throw Gaussian snowballs at the window and see what sticks. P.S. Not sure where the insight came from, but I was living near Boston at the time, and had young children :)

evilhackerdude 3y ago

I'm looking forward to the first native optimized WebGPU implementation of 3DGS rendering. I'm also curious how scene data could be compressed and decompressed efficiently.

raphlinus 3y ago
I'm also looking forward to it. One of the big challenges is the sorting, for which I'm unaware of a good WebGPU implementation. I have some more notes on this question in a Zulip thread[1]. [1]: https://xi.zulipchat.com/#narrow/stream/197075-gpu/topic/Gaussian.20splatting https://xi.zulipchat.com/#narrow/stream/197075-gpu/topic/Gau...
- boywitharupee 3y ago
  Does the sorting need to be done in the renderer, or is that part of the training process?
  - raphlinus 3y ago
    It needs to be done in the renderer. I think it's doable though, the FidelityFX library looks like it can be ported, it'll just run a bit slow because of the lack of subgroups. This particular library isn't based on a fancy scan implementation, as the state-of-the-art CUDA implementations are. There's a bit more followup in the linked Zulip thread.
blovescoffee 3y ago
I’m working on this right now!

sneak 3y ago

What's the state of the art for structure from motion? Given video of a space, how does one practically turn it into a 3d scene today?

naillo 3y ago
Literally gaussian splatting (and nerfs)
andybak 3y ago
Video is just a bunch of still images (except potentially with degraded quality from resolution and/or motion blur) so it's much the same as asking "how does one practically capture a 3d scene" without the video part. Regular photogrammetry - but with interest and practicality gradually increasing for NeRF style approaches. (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?)
- radarsat1 3y ago
  > (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?) Can I ask why you say this? Camera motion models including Kalman filters of various kinds and constraints on derivatives etc, are absolutely used in SLAM and photogrammetry afaik, and have been for a long time.
  - andybak 3y ago
    OK maybe it's my limited experience but last time i tried it, video was treated as "extract frames and then only use every 50th (or so)" This was a couple of years ago probably. Meshroom and Reality Capture. Maybe I was mistaken or maybe they've improved? Can you point me to software that does use video for photogrammetry in the way you describe?

eurekin 3y ago

Would be cool to see those splats in action from a second, free to move camera

arijun 3y ago
What’s the difference between a free-to-move camera and one on a preset track if they’re rendering at 100fps?
- eurekin 3y ago
  To see the blobs forming, moving, changing
  - arijun 3y ago
    But the 3D blobs don’t move, do they? Only the splats do.

itronitron 3y ago

Wow, the referenced Ecstatica game is wild.

pjs_ 3y ago
Ecstatica II was an extraordinary game - funny, weird, violent, amazing sound, and super atmospheric.
pavlov 3y ago
For a second I thought it's a really well done fake. The art style looked so obviously beyond 1994 software rendering. But then you start to notice that the backgrounds are all static 3D renderings and there are very few objects in motion after all, so it's certainly feasible on a 486. Maybe you could say it's basically a 3D version of Alone in the Dark using ellipsoids instead of polygons as the primitive. The quality of the animation is unusually good for the time, which is really what creates the coherent illusion. Wikipedia says the game had just one developer paired with a "film animation expert", which makes a lot of sense. https://en.wikipedia.org/wiki/Ecstatica https://en.wikipedia.org/wiki/Ecstatica I'm shocked I've never heard of this game until now.

nwatson 3y ago

Here's a video that may or may not use these techniques, I'd be curious for those who know to judge ... https://youtu.be/dXLCHvRsgRQ?si=wkniCf-7wNPt8DGj https://youtu.be/dXLCHvRsgRQ?si=wkniCf-7wNPt8DGj There's coherence frame-to-frame of the artifacts used to render, but they may be 2D "watercolor" splotches, and some seem very manually seeded, e.g. the shapes used to fill in irises for eyes. In some scenes though the blobs seem to be 3D-represented.

sorenjan 3y ago

I recently came across this video showing how to use Gaussian splatting: Getting Started With 3D Gaussian Splats for Windows (Beginner Guide) - https://www.youtube.com/watch?v=UXtuigy_wYc https://www.youtube.com/watch?v=UXtuigy_wYc

radarsat1 3y ago

> This is using gradient descent and “differentiable rendering” and all the other things that are way over my head. then.. > And finally, they have resisted the temptation to do “neural” anything ;) So, they are doing something similar to NeRF, in a sense, but using a different basis function, and a slightly different target. The whole "neural" or not "neural" is just about what you are optimizing, but it's not that different, conceptually -- they are optimizing a data-driven approximator of a 3d scene-related variable based on images. The big difference of course, based on reading this, is that NeRF models the whole light transmission function ("radiance field") whereas this seems to model only the boundary conditions (what the light "hits"). So given the different modelling target, then yes, a different, and perhaps simpler, representation basis definitely seems warranted, and is shown here to give good results. But the comment, "they avoided neural anything" feels a bit smug, as if "avoiding the hype" is a laudable goal on its own merits. As if people are using neural networks for no reason but because they're cool, and not because they're actually an appropriate solution that are shown to work really well in practice. They surely are cool, don't get me wrong, but the hype is so often justified, because neural networks are really good function approximators. They are also not just one thing -- the choice and breadth of architectures that we happen to call "neural networks" is huge, so explicitly avoiding this huge potential solution space for some reason without good reason feels pretty specious; heck you could even consider having a Gaussian "output layer" to a neural network, which is called an RBF, being something like this "splat" approach. (This is in response to the blog post, not the paper -- I'm sure the paper has plenty of justification for modeling choices.) Having said that, of course it's interesting to also explore simpler, perhaps more efficient approaches, but I don't see "resisting using neural anything" as a good thing, just because they are popular. It ignores that they are popular for a reason. (nb: This assessment is just based on the blog post and giving my impression from the tone of the introduction, I haven't read the paper yet, so might not be accurate with respect to the paper contents. I just found this off-hand comment in the post worth reflecting on.)

graffix 3y ago
> I don't see "resisting using neural anything" as a good thing, just because they are popular I read Aras' quip as more narrowly technical. OFC it's ambiguous and you'd have to ask the man himself. The gist of NeRF is to obtain an NN representation of a 5D light field (3D position + 2D direction) from samples (photographs) of the real-world light field. Alarm bells ring already- 5 dimensions isn't that many! Considering NeRF has always used a low-rank spherical harmonic representation of the directional domain, it's even more like 3D-and-change. To reconstruct a function of such low dimensionality, why choose an NN? Then at inference time, for each pixel, you have sample the NN repeatedly over the view ray. This part is exceedingly silly, as compact representations of light fields are a solved bread-and-butter problem in graphics. Later on Plenoxels explicitly took the "Ne" out of "NeRF", giving far higher training and inference performance (also mentioned ITT). To be fair, and later still, Nvidia somewhat redeemed NNs here with Instant NeRF: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... ...where the twist was to interpolate fancy input emeddings, which are run through a tiny NN. That tininess is important, as the need to fetch NN weights from VRAM would kick NNs right off the Pareto frontier. Zooming out, NNs have only seen wide adoption in graphics engineering for reconstruction from sparse data (inc. denoising). Makes sense, as that's a high-dimensional problem. Still, beware that the NN solutions rarely blow handmade algorithms out of the water. I also think using tiny NNs for compression- closely related to reconstruction- has a future too. Beyond that, if NNs were to set the graphics world ablaze, it would've happened by now. Lots of graphics engineering is just approximating functions, so it's natural NNs have some place here. However, our functions tend to be more understandable, tractable, malleable. It's not an application domain where it's virtually impossible to write an algorithmic solution by hand (let alone one that performs well), like natural language understanding.

Escapado 3y ago

I am very uneducated in this so please forgive my ignorance. These videos always look so insanely cool which I really like and from what I understand the scenes/radiance fields are always static and have lighting baked in. Is there a chance this might be turned into something that can be dynamically lit and support motion?

Lichtso 3y ago
That is a good point and the answer in short is: No. Radiance fields have no concept of light emission, reflection, absorption, etc. instead everything is mushed into one value: The light transported. In that sense radiance fields are just 3D photos. You would have to perform reverse-rendering / photo grammetry and estimate where the light sources and the surfaces are, what materials they have and so on. Then you could use traditional path tracing methods on that again. Another thing to think about might be videos (not animation): Continuously capture the radiance field over time and then try to compress away the similarities in between frames to gain temporal coherence.
- jayd16 3y ago
  Would you not be able to store and render normal maps color instead of just albedo? Seems like you should be able to render the scene normals and do a deferred lighting pass. Is depth not properly preserved or something?
  - sorenjan 3y ago
    They're not just storing the albedo, they're optimizing spherical harmonics to represent the color in an anisotropic way, that's why tey're calling it a radiance field. Radiance fields capture both light intensity (including color) and direction. They explain in the paper that it's very difficult to estimate good normals from the sparse point cloud they're starting with (or rather, that's taken as a given and produced as an earlier step using colmap) and that the gaussians doesn't use normals. You could probably make a point cloud from the gaussians and then use one of the existing techniques to estimate their normals, as a first attempt. Remember that it's a bit tricky to talk about depth when the gaussians have both a position (mean value) and a size (covariance). The bicycle spokes are made up of long thin splats, what value do you assign to one of those? That's why I think you would have to sample new points from them as a first step. https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

9dev 3y ago

It's always fun when science exceeds science fiction's expectations. In this case, I was immediately thinking of the Braindance concept[1] in Cyberpunk 2077, which - more or less - allows to move around in the visual memories of another person, limited by that person's perception during that scene. When moving the camera to a perspective distinct from the original one, the vision disintegrates into kind of three-dimensional pixel blobs, somewhat similar to the notion of blobs here - just a lot less polished, surprisingly, compared to this paper. [1]: https://steelseries.com/blog/how-to-braindance-cyberpunk-2077-428 https://steelseries.com/blog/how-to-braindance-cyberpunk-207...

Macuyiko 3y ago

Very cool work. As a bit of tangent (but wondering whether someone can answer) - the article also makes mention of point based rendering and indeed the fact is has been a staple of particle systems for a long time. However, especially with recent games, I have noticed (purely subjectively) a very subtle shift to a new style of particle systems which are on the one hand fully point oriented (compared to (textured) fragments) but on the other behave more like a physics systems. Examples: - Hogwarts (heavily): https://www.gamespot.com/a/uploads/original/1816/18167535/4097726-hogwartslegacybestspellsallspellsguideunforgivablecurses-1.jpg https://www.gamespot.com/a/uploads/original/1816/18167535/40... - Forspoken (heavily): https://oyster.ignimgs.com/mediawiki/apis.ign.com/project-athia/0/04/FdlsymrXwAgCXE4.jpg https://oyster.ignimgs.com/mediawiki/apis.ign.com/project-at... - Starfield (though more rarely): https://dotesports.com/wp-content/uploads/2023/08/temple-location-Starfield.png https://dotesports.com/wp-content/uploads/2023/08/temple-loc... - AC6 - FF16 (heavily) It's more obvious when you see it 'in motion'. The common denominator seems to be particles as colored transparent points with physics. Especially on console systems it seems that developers are using this for very cheap (CPU-wise, all on GPU) effects. Anyone in gamedev who has some insight in this?

lelandbatey 3y ago
I remember that kind of "particle effect" originally being shown off ~11 years ago by the Unreal Engine 4 demo called "Elemental": https://youtu.be/dD9CPqSKjTU https://youtu.be/dD9CPqSKjTU I don't think they're "from" that, but that's the first landmark I can point to.
deleted 3y ago
[deleted]
tbalsam 3y ago
I've certainly seen game engines trying to advertise much more-involved particle systems over the last few years, that is for sure. ;PPPP Which in some ways I guess makes sense, as the shift towards AI has heavily pushed GPUs almost towards a "sizeable-reduced-instruction-set-GPU-in-GPU" approach with RT cores/tensor cores, etc.... ;P Side note as well, in/from my experience at least, I think these systems may not be as hard to render as it is for the author. A few years ago, someone (at NVIDIA I think?) wrote kernels to move the SE(3) kernels to tensor cores, so I wouldn't be shocked if some of that could be ported to the spherical harmonics portion of the Gaussian splatting during both compression and runtime: https://developer.nvidia.com/blog/accelerating-se3-transformers-training-using-an-nvidia-open-source-model-implementation/ https://developer.nvidia.com/blog/accelerating-se3-transform... Also, side side note, gaussian splatting should be quite efficient...I think? Due to technically always having support in 3D space (and hopefully not too much of a problem with having good support in 3D space). This should mean that even 'sloppy', quick-conversion calculations should work pretty decently in the end. I say all of this knowing very little about most optimizations like billboarding, how things like nanite work, etc, etc. I do like it tho! ;PPPP

jayd16 3y ago

Reminds me of Google's Seurat. https://developers.google.com/vr/discover/seurat https://developers.google.com/vr/discover/seurat

jjcm 3y ago

I was super excited to see this when the Siggraph paper first dropped. For the past decade or so, I've been taking pictures of any room I've lived in from hundreds of angles, with the intention of being able to recreate these in 3d at some point in the future. Gaussian splatting is the first technique I've seen that feels like it can recreate it in a way that feels almost-real. Once the tools around this mature a bit more, I'm super excited to revisit those old rooms and be hit with a wave of nostalgia.

phkahler 3y ago

I rendered voxel iso-surfaces using blobs in the late 90's. I'd scan a full 3d voxel array for surface voxels, use the local density gradient to compute a normal, quantize the normal to one of 240. Then I'd make chains of surface voxels using a table of displacement vectors. Most voxels were 2 bytes - a displacement index (from previous voxel) and a normal vector (index). Lighting was computed for all 240 normal vectors and stuffed in a lookup. Software could paint all these little colored circles in a z-buffer very fast. The only real bummer back then was that it had to be non-perspective. The displacement vectors were coverted to screen space per-frame, so it was paint a blob, offset, lookup color, paint another. One displacement value was used to indicate the chain ended and I needed to store an absolute position next, but most voxels were just 2 bytes.