6 ms·

What is the use case for this library, in layman terms? I pass it a blurry pic and it „enhances it” ?

LoganDark 3y ago
Gaussian splatting, at least in 3D, is rasterization technique that, AFAIK, doesn't use polygons, it's intended to allow photorealistic renders based on photogrammetry. It seems like it took off after this SIGGRAPH paper: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ Anyway, this doesn't seem to allow any sort of dynamic lighting whatsoever, but for scans of existing places, such as house tours, street views and so on, this seems super promising.
- slabity 3y ago
  > Anyway, this doesn't seem to allow any sort of dynamic lighting whatsoever, but for scans of existing places, such as house tours, street views and so on, this seems super promising. So this method can only really be used for static scenes made from a point cloud? I remember 10 years ago, a company called Euclideon revealed a demo[1] of a rendering technique (they called it "Unlimited Detail Real-Time Rendering") that put nearly every other rasterization technique of the time to shame. However, the major issue with the demo was that it only showed a static scene with zero dynamic lighting. I don't think they ever revealed the "secret sauce" of how it works, but it seems to have similar limitations to this new process, or at least that's what I expect considering it can't handle something as simple as a dynamic mesh. It was extremely hyped up in certain areas, but over a decade later this technology isn't really used in any mainstream rendering engines because it only works for static scenes. I hate to disparage new rendering techniques, but this really feels like a repeat of what happened back then. What exactly is different in 3D Guassian Splatting? Can it be used in any dynamic scenes in real-time at all? Does it provide any significant advantages from this older system? [1] https://www.youtube.com/watch?v=DrBR_4FohSE https://www.youtube.com/watch?v=DrBR_4FohSE
  - LoganDark 3y ago
    Euclideon's method probably uses fine-grained acceleration structures and LoDs that are too expensive to rebuild in real-time. At least, that's how I took it.
  - andybak 3y ago
    Ruben Fro is doing some cool stuff with animated gaussian splats: https://twitter.com/Ruben_Fro https://twitter.com/Ruben_Fro Think of them as point clouds + lighting. Relighting is viable. Procedural deformation is viable. Animation is just a case of transforming sub-groupings. Collisions might be a bit trickier but you can always derive a low-res collision mesh from the input data and use that as a proxy. It's early days at the moment and people are still exploring the possibilities.
    - slabity 3y ago
      Thank you for sharing that. Those images have dissuaded my pessimism a bit. They prove at the very least the technique can be used in some non-static scenes.
      - corysama 3y ago
        The splats are pretty easy to manipulate. At least as easy as triangles. It’s just that there has not been much attention paid to them historically. So, there are no content pipelines yet.
  - speps 3y ago
    It was probably similar to UE5's Nanite, definitely possible 10 years ago with custom tech and very constrained assets.
    - HellDunkel 3y ago
      Definitly not. It was just a sparse voxel engine with model instancing. Didn‘t go anywhere and for good reason. Nanite does build on some very advanced and creative poly reduction tech and adds streaming to take it to the next level.
- graffix 3y ago
  Right. Note that Gaussian splatting as a rendering primitive dates from the early 90s, but it never saw much use. Splats aren't very good for magnification (important for medical/scientific/engineering visualization), nor do they have easy support for domain repetition (important for video games). The new thing is fast regression of a real light field into Gaussian splats, which can then be rendered at reasonable rates. Dynamic lighting requires full inverse rendering as a preprocess, which is way beyond the scope of the technique. Technically Gaussian splats could form part of an inverse rendering pipeline, and also be the final target representation, but I'm not sure there would be any benefit over alternative rendering primitives.
NotGMan 3y ago
It's for deep learning, not that much "for graphics". The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion". Voxels are bad as they take too much memory and are uniform (each voxel is eg 5x5x5 cm). Gausians are kinda like "variable sized voxels" that also have orientation, color based on angle viewing and can stretch. Imagine 3D blobs basicaly (3D capsules or more like 3D density bubbles with transparency). So the scene can be represented in 3D much more efficiently using gausians splats, which is why they are used for "3d stable diffusion". So now that this has come into use we also need efficient way or rendering them.
- scotty79 3y ago
  Would 2d image generation based on gaussian splats be better than stable diffusion?
  - blovescoffee 3y ago
    Nope. Gaussian splatting is for reconstruction of images or 3D shapes not a method for generating them.
    - scotty79 3y ago
      Stable diffusion is just reconstructing image from random noise. What I mean is could you make a version of stable diffusion that uses splats instead of pixels?
- londons_explore 3y ago
  Really nice explanation here.
- encypruon 3y ago
  Sorry, some objections... > It's for deep learning, not that much "for graphics". No, while it is true that there is some overlap between the techniques and concepts used, gaussian splatting isn't necessarily "for deep learning". The library provides a differentiable rasterizer for gaussian splats. This basically means that you can ask it "if I want my output image to change in this and this direction, in what direction should I change the position / orientation / color / ... of my splats?". This enables users to plug it into other software (that is also commonly used for deep learning) and to optimize the parameters of the splats to represent a particular scene. Since it's primarily a differentiable rasterizer for splats I think it's fair to say that it is "for graphics". > The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion". That it uses gradient descent doesn't mean that it is "deep learning". There are no neural networks or layers here. It's not an "equivalent of stable diffusion". The way it's used now is to learn a representation of a single scene, not unlike photogrammetry. Sure, there may be other use cases for this library, but this is primarily what gaussian splatting is about.

Gaussian splatting, at least in 3D, is rasterization technique that, AFAIK, doesn't use polygons, it's intended to allow photorealistic renders based on photogrammetry. It seems like it took off after this SIGGRAPH paper: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ Anyway, this doesn't seem to allow any sort of dynamic lighting whatsoever, but for scans of existing places, such as house tours, street views and so on, this seems super promising.

slabity 3y ago
> Anyway, this doesn't seem to allow any sort of dynamic lighting whatsoever, but for scans of existing places, such as house tours, street views and so on, this seems super promising. So this method can only really be used for static scenes made from a point cloud? I remember 10 years ago, a company called Euclideon revealed a demo[1] of a rendering technique (they called it "Unlimited Detail Real-Time Rendering") that put nearly every other rasterization technique of the time to shame. However, the major issue with the demo was that it only showed a static scene with zero dynamic lighting. I don't think they ever revealed the "secret sauce" of how it works, but it seems to have similar limitations to this new process, or at least that's what I expect considering it can't handle something as simple as a dynamic mesh. It was extremely hyped up in certain areas, but over a decade later this technology isn't really used in any mainstream rendering engines because it only works for static scenes. I hate to disparage new rendering techniques, but this really feels like a repeat of what happened back then. What exactly is different in 3D Guassian Splatting? Can it be used in any dynamic scenes in real-time at all? Does it provide any significant advantages from this older system? [1] https://www.youtube.com/watch?v=DrBR_4FohSE https://www.youtube.com/watch?v=DrBR_4FohSE
- LoganDark 3y ago
  Euclideon's method probably uses fine-grained acceleration structures and LoDs that are too expensive to rebuild in real-time. At least, that's how I took it.
- andybak 3y ago
  Ruben Fro is doing some cool stuff with animated gaussian splats: https://twitter.com/Ruben_Fro https://twitter.com/Ruben_Fro Think of them as point clouds + lighting. Relighting is viable. Procedural deformation is viable. Animation is just a case of transforming sub-groupings. Collisions might be a bit trickier but you can always derive a low-res collision mesh from the input data and use that as a proxy. It's early days at the moment and people are still exploring the possibilities.
  - slabity 3y ago
    Thank you for sharing that. Those images have dissuaded my pessimism a bit. They prove at the very least the technique can be used in some non-static scenes.
    - corysama 3y ago
      The splats are pretty easy to manipulate. At least as easy as triangles. It’s just that there has not been much attention paid to them historically. So, there are no content pipelines yet.
- speps 3y ago
  It was probably similar to UE5's Nanite, definitely possible 10 years ago with custom tech and very constrained assets.
  - HellDunkel 3y ago
    Definitly not. It was just a sparse voxel engine with model instancing. Didn‘t go anywhere and for good reason. Nanite does build on some very advanced and creative poly reduction tech and adds streaming to take it to the next level.
graffix 3y ago
Right. Note that Gaussian splatting as a rendering primitive dates from the early 90s, but it never saw much use. Splats aren't very good for magnification (important for medical/scientific/engineering visualization), nor do they have easy support for domain repetition (important for video games). The new thing is fast regression of a real light field into Gaussian splats, which can then be rendered at reasonable rates. Dynamic lighting requires full inverse rendering as a preprocess, which is way beyond the scope of the technique. Technically Gaussian splats could form part of an inverse rendering pipeline, and also be the final target representation, but I'm not sure there would be any benefit over alternative rendering primitives.

NotGMan 3y ago

It's for deep learning, not that much "for graphics". The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion". Voxels are bad as they take too much memory and are uniform (each voxel is eg 5x5x5 cm). Gausians are kinda like "variable sized voxels" that also have orientation, color based on angle viewing and can stretch. Imagine 3D blobs basicaly (3D capsules or more like 3D density bubbles with transparency). So the scene can be represented in 3D much more efficiently using gausians splats, which is why they are used for "3d stable diffusion". So now that this has come into use we also need efficient way or rendering them.

scotty79 3y ago
Would 2d image generation based on gaussian splats be better than stable diffusion?
- blovescoffee 3y ago
  Nope. Gaussian splatting is for reconstruction of images or 3D shapes not a method for generating them.
  - scotty79 3y ago
    Stable diffusion is just reconstructing image from random noise. What I mean is could you make a version of stable diffusion that uses splats instead of pixels?
londons_explore 3y ago
Really nice explanation here.
encypruon 3y ago
Sorry, some objections... > It's for deep learning, not that much "for graphics". No, while it is true that there is some overlap between the techniques and concepts used, gaussian splatting isn't necessarily "for deep learning". The library provides a differentiable rasterizer for gaussian splats. This basically means that you can ask it "if I want my output image to change in this and this direction, in what direction should I change the position / orientation / color / ... of my splats?". This enables users to plug it into other software (that is also commonly used for deep learning) and to optimize the parameters of the splats to represent a particular scene. Since it's primarily a differentiable rasterizer for splats I think it's fair to say that it is "for graphics". > The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion". That it uses gradient descent doesn't mean that it is "deep learning". There are no neural networks or layers here. It's not an "equivalent of stable diffusion". The way it's used now is to learn a representation of a single scene, not unlike photogrammetry. Sure, there may be other use cases for this library, but this is primarily what gaussian splatting is about.