10 ms·

Where would you use 3D Gaussian splatting? Static environments for video games?

halfbreed 2y ago
I still wonder this myself, but the most obvious area that comes to mind is real estate virtual tours. Once a splat can render in the browser at high fps, then I see this replacing most all other technologies currently being used.
55555 2y ago
Virtual tours for real estate
- littlestymaar 2y ago
  Are there businesses doing it already or is the tech too immature to be used IRL right now?
  - apinstein 2y ago
    I started and ran a real estate photography platform from 2004-2018. We started r&d on this in ~2016 when consumer VR first came out. At the time we used photogrammetry and it was “dreadful” to try to capture due to mirrors, glass, etc. So I have been following GS tech for a while. I’ve not yet seen anything (open source / papers) that quite gets there yet. I do think it will. In my opinion, there are two useful ways GS can bring to this industry. The first is ability to use photo capture to re-render as a high production quality video similar to what people do with Luma AI today. While this is a really cool capability, it’s also not really that hard to do anymore with drones and gimbals. So, the experience of creating the same thing via GS has to be better and easier, and it’s not clear when that will likely happen due to how painful the capture side is. You really need good real time capture feedback to make sure you have good coverage. Finding out there’s a hole once you’re off location is a deal breaker. The second is to create VR capable experiences. I think the first real useful thing for consumers will be so you can walk around in a small three or 4 foot area and get a stereo sense of what it’s like to be there. This is an amazing consumer experience. But the practicality of scaling this depends on VR hardware and adoption, and that hasn’t yet become commonplace enough to make consumer use “adjacent possible” for broad deployment. I could see it being used on super high end to start out.
noduerme 2y ago
Could be very useful for prototyping camera moves and lighting for film / commercial shoots on location. You might not even need to send a scout, just get a few pictures and be able to thumbnail a whole scene. I could also see a market for people who want to recreate virtual environments from old photos. Also, load the model on a single-lens 360 camera and infer stereoscopic output.
- rationalfaith 2y ago
  [dead]
dagmx 2y ago
No, Gaussian splats are pretty poor for video games. There’s a significant amount of overdraw and they’re not art directable or dynamic. Gaussian splats are much better suited for capturing things where you don’t have artists available and don’t have a ton of performance requirements with regards to frame time. So things like capturing real estate , or historical venues etc.
- vlovich123 2y ago
  Isn’t that a “for now” problem rather than something intractable for the performance anyway? Presumably HW and SW algorithms will continue to improve. Art directable may be a problem but it feels like Gaussian splats + genAI models could be a match made in heaven with the genAI mode generating the starting image and splats generating the 3d scene from it
  - dagmx 2y ago
    Sure, given an unlimited amount of time and resources, it’s possible that Gaussian splats could be performant. But that’s just too vague a discussion point to be meaningful. It’s definitely not in the cards in the near term without a dramatic breakthrough. Splats have been a thing for decades so I’m not holding my breath.
    - vlovich123 2y ago
      I mean here it is running at 22fps. In another 5 years it’s reasonable to conservatively believe hardware and software to be 3x as powerful which gets you to a smooth 60fps. What am I missing on the performance front?
      - dagmx 2y ago
        Well my critique of your comment is just that it’s unbounded. Yes, eventually all compute will get better and we can use once slow technologies. But that’s not a very valuable discussion because nobody is saying it’ll never be useful, just that it isn’t for games today. It also ignores that everything else will be faster too then as well, and ignores needing to target different baselines of hardware. Either way 5 years for a 3x improvement seems unrealistic. 4 years saw a little over a doubling of performance at the highest end with a significant increase in power requirements as well, where we’re now hitting realistic power limits. Taking the 2080 vs 4080 as their respective tiers 153% performance increase 50% more power consumption 50% price increase. So yes performance at the high end will increase, but it’s scaling pretty poorly with cost and power. And the lower end isn’t scaling as linearly. On the lower/mid end (1060 Ti vs 2060 Super) we saw only a 53% increase in that same time period.
deckar01 2y ago
Photography. A small cheap camera array could produce higher resolution, alternate angles, and arbitrary lens parameters that would otherwise require expensive or impossible lenses. Then you can render an array of angles for holographic displays.
kersplody 2y ago
Volumetric live action performance capture. Basically a video you can walk around in. Currently requires a large synchronized camera array. Plays back on most mobile devices. Several major industry efforts in this space ongoing.
- rationalfaith 2y ago
  [dead]
twelvechairs 2y ago
Basically when you don't want to spend time to pre-process e.g. through traditional photogrammetry. So near-real-time events, or where there's huge amounts of pointcloud capture and comparatively little visualisation Edit: others are mentioning real estate I'd think that will prefer some pre processing but ymmv
- tomp 2y ago
  Not really. First if all, most GS take posed images as input, so you need to run a traditional photogrammetry pipeline (COLMAP) anyways. The purpose of GS is that the result is far beyond anything that traditional photogrammetry (dense mesh reconstruction) can manage, especially when it comes to “weird” stuff (semi-transparent objects).
t43562 2y ago
What about virtual tourism? See the pyramids without the expense of going there.
littlestymaar 2y ago
Have you watch the basketball games in the Olympics? Every once in a while, they showed a replay of a key point with some effect of the camera moving between two views in the middle of the shoot. It was not likely to be GS since there was tons of artifacts that didn't look like the ones GS produces, but they could have used it for such stuff. For instance with some kind of 4D GS we could even remap the camera view entirely to have a virtual camera allowing us to see the shoot from the eyes of Steph Curry with Batum and Fournier double teaming him.
jorgemf 2y ago
Gaussian splatting transform images to a cloud points. GPUs can render these points but it is a very slow process. You need to transform the cloud points to meshes. So basically is the initial process to capture environments before converting them to 3D meshes that the GPUs can use for anything you want. It is much cheaper to use pictures to have a 3D representantion of an object or environment than buying professional stuff.
- andybak 2y ago
  > Gaussian splatting transform images to a cloud points. Not exactly. The "splats" are both spread out in space (big ellipsoids), partially transparent (what you end up seeing is the composite of all the splats you can see in a given direction) AND view dependent (they render differently depending on the direction you are looking. Also - there's not a simple spatial relationship between splats and solid objects. The resulting surfaces are a kind of optical illusion based on all the splats you're seeing in a specific direction. (some methods have attempted to lock splats more closely to the surfaces they are meant to represent but I don't know what the tradeoffs are). Generating a mesh from splats is possible but then you've thrown away everything that makes a splat special. You're back to shitty photogrammetry. All the clever stuff (which is a kind of radiance capture) is gone. Splats are a lot faster to render than NeRFs - which is their appeal. But heavier than triangles due to having to sort them every frame (because transparent objects don't composite correctly without depth sorting)
  - vessenes 2y ago
    Minor nit — in what way do splats render differently depending on direction of looking? To my mind these are probabilistic ellipsoids in 3D (or 4D for motion splats) space, and so while any novel view will see a slightly different shape, that’s an artifact of the view changing, not the splat. Do I understand it (or you) correctly?
    - refibrillator 2y ago
      In 3DGS, spherical harmonics are used to model view-dependent changes in color. https://en.m.wikipedia.org/wiki/Spherical_harmonics https://en.m.wikipedia.org/wiki/Spherical_harmonics Basically for each Gaussian there is a set of coefficients and those are used to calculate what color should be rendered depending on the viewing angle of the camera. And the SH coeffs are optimized through gradient descent just like the other parameters including position and shape.
two_handfuls 2y ago
Good question. One thing I know they are good for are 3D photos because they solve a fundamental issue with the current tech: IPD. The current tech (Apple Vision Pro included) uses two photos: one per eye. If the photos were taken from a distance that matches the distance between your eyes, then the effect is convincing. Otherwise, it looks a bit off. The other problem is that a big part of the 3D perception comes from parallax: how the image changes with head motions (even small motions). Techniques that are not limited to two fixed images, but instead allow us to create new views for small motions, are great for much more impressive 3D photos. With more input photos you get a “walkable photo”: a photo that you can take a few steps in, say if you are wearing a VR headset. I’m sure 3D Gaussian splatting is good for other things too, given the excitement around them. Backgrounds in movies maybe?
praveen9920 2y ago
One application I can think of is Google Street View. Gaussian splatting can potentially "smoothen" the transition between the images and make it look more realistic.
lawlessone 2y ago
>Where would you use 3D Gaussian splatting? The primary purpose of Gaussian splatting is to frontpage here every two weeks.

I still wonder this myself, but the most obvious area that comes to mind is real estate virtual tours. Once a splat can render in the browser at high fps, then I see this replacing most all other technologies currently being used.

55555 2y ago

Virtual tours for real estate

littlestymaar 2y ago
Are there businesses doing it already or is the tech too immature to be used IRL right now?
- apinstein 2y ago
  I started and ran a real estate photography platform from 2004-2018. We started r&d on this in ~2016 when consumer VR first came out. At the time we used photogrammetry and it was “dreadful” to try to capture due to mirrors, glass, etc. So I have been following GS tech for a while. I’ve not yet seen anything (open source / papers) that quite gets there yet. I do think it will. In my opinion, there are two useful ways GS can bring to this industry. The first is ability to use photo capture to re-render as a high production quality video similar to what people do with Luma AI today. While this is a really cool capability, it’s also not really that hard to do anymore with drones and gimbals. So, the experience of creating the same thing via GS has to be better and easier, and it’s not clear when that will likely happen due to how painful the capture side is. You really need good real time capture feedback to make sure you have good coverage. Finding out there’s a hole once you’re off location is a deal breaker. The second is to create VR capable experiences. I think the first real useful thing for consumers will be so you can walk around in a small three or 4 foot area and get a stereo sense of what it’s like to be there. This is an amazing consumer experience. But the practicality of scaling this depends on VR hardware and adoption, and that hasn’t yet become commonplace enough to make consumer use “adjacent possible” for broad deployment. I could see it being used on super high end to start out.

noduerme 2y ago

Could be very useful for prototyping camera moves and lighting for film / commercial shoots on location. You might not even need to send a scout, just get a few pictures and be able to thumbnail a whole scene. I could also see a market for people who want to recreate virtual environments from old photos. Also, load the model on a single-lens 360 camera and infer stereoscopic output.

rationalfaith 2y ago
[dead]

dagmx 2y ago

No, Gaussian splats are pretty poor for video games. There’s a significant amount of overdraw and they’re not art directable or dynamic. Gaussian splats are much better suited for capturing things where you don’t have artists available and don’t have a ton of performance requirements with regards to frame time. So things like capturing real estate , or historical venues etc.

vlovich123 2y ago
Isn’t that a “for now” problem rather than something intractable for the performance anyway? Presumably HW and SW algorithms will continue to improve. Art directable may be a problem but it feels like Gaussian splats + genAI models could be a match made in heaven with the genAI mode generating the starting image and splats generating the 3d scene from it
- dagmx 2y ago
  Sure, given an unlimited amount of time and resources, it’s possible that Gaussian splats could be performant. But that’s just too vague a discussion point to be meaningful. It’s definitely not in the cards in the near term without a dramatic breakthrough. Splats have been a thing for decades so I’m not holding my breath.
  - vlovich123 2y ago
    I mean here it is running at 22fps. In another 5 years it’s reasonable to conservatively believe hardware and software to be 3x as powerful which gets you to a smooth 60fps. What am I missing on the performance front?
    - dagmx 2y ago
      Well my critique of your comment is just that it’s unbounded. Yes, eventually all compute will get better and we can use once slow technologies. But that’s not a very valuable discussion because nobody is saying it’ll never be useful, just that it isn’t for games today. It also ignores that everything else will be faster too then as well, and ignores needing to target different baselines of hardware. Either way 5 years for a 3x improvement seems unrealistic. 4 years saw a little over a doubling of performance at the highest end with a significant increase in power requirements as well, where we’re now hitting realistic power limits. Taking the 2080 vs 4080 as their respective tiers 153% performance increase 50% more power consumption 50% price increase. So yes performance at the high end will increase, but it’s scaling pretty poorly with cost and power. And the lower end isn’t scaling as linearly. On the lower/mid end (1060 Ti vs 2060 Super) we saw only a 53% increase in that same time period.

deckar01 2y ago

Photography. A small cheap camera array could produce higher resolution, alternate angles, and arbitrary lens parameters that would otherwise require expensive or impossible lenses. Then you can render an array of angles for holographic displays.

kersplody 2y ago

Volumetric live action performance capture. Basically a video you can walk around in. Currently requires a large synchronized camera array. Plays back on most mobile devices. Several major industry efforts in this space ongoing.

rationalfaith 2y ago
[dead]

twelvechairs 2y ago

Basically when you don't want to spend time to pre-process e.g. through traditional photogrammetry. So near-real-time events, or where there's huge amounts of pointcloud capture and comparatively little visualisation Edit: others are mentioning real estate I'd think that will prefer some pre processing but ymmv

tomp 2y ago
Not really. First if all, most GS take posed images as input, so you need to run a traditional photogrammetry pipeline (COLMAP) anyways. The purpose of GS is that the result is far beyond anything that traditional photogrammetry (dense mesh reconstruction) can manage, especially when it comes to “weird” stuff (semi-transparent objects).

t43562 2y ago

What about virtual tourism? See the pyramids without the expense of going there.

littlestymaar 2y ago

Have you watch the basketball games in the Olympics? Every once in a while, they showed a replay of a key point with some effect of the camera moving between two views in the middle of the shoot. It was not likely to be GS since there was tons of artifacts that didn't look like the ones GS produces, but they could have used it for such stuff. For instance with some kind of 4D GS we could even remap the camera view entirely to have a virtual camera allowing us to see the shoot from the eyes of Steph Curry with Batum and Fournier double teaming him.

jorgemf 2y ago

Gaussian splatting transform images to a cloud points. GPUs can render these points but it is a very slow process. You need to transform the cloud points to meshes. So basically is the initial process to capture environments before converting them to 3D meshes that the GPUs can use for anything you want. It is much cheaper to use pictures to have a 3D representantion of an object or environment than buying professional stuff.

andybak 2y ago
> Gaussian splatting transform images to a cloud points. Not exactly. The "splats" are both spread out in space (big ellipsoids), partially transparent (what you end up seeing is the composite of all the splats you can see in a given direction) AND view dependent (they render differently depending on the direction you are looking. Also - there's not a simple spatial relationship between splats and solid objects. The resulting surfaces are a kind of optical illusion based on all the splats you're seeing in a specific direction. (some methods have attempted to lock splats more closely to the surfaces they are meant to represent but I don't know what the tradeoffs are). Generating a mesh from splats is possible but then you've thrown away everything that makes a splat special. You're back to shitty photogrammetry. All the clever stuff (which is a kind of radiance capture) is gone. Splats are a lot faster to render than NeRFs - which is their appeal. But heavier than triangles due to having to sort them every frame (because transparent objects don't composite correctly without depth sorting)
- vessenes 2y ago
  Minor nit — in what way do splats render differently depending on direction of looking? To my mind these are probabilistic ellipsoids in 3D (or 4D for motion splats) space, and so while any novel view will see a slightly different shape, that’s an artifact of the view changing, not the splat. Do I understand it (or you) correctly?
  - refibrillator 2y ago
    In 3DGS, spherical harmonics are used to model view-dependent changes in color. https://en.m.wikipedia.org/wiki/Spherical_harmonics https://en.m.wikipedia.org/wiki/Spherical_harmonics Basically for each Gaussian there is a set of coefficients and those are used to calculate what color should be rendered depending on the viewing angle of the camera. And the SH coeffs are optimized through gradient descent just like the other parameters including position and shape.

two_handfuls 2y ago

Good question. One thing I know they are good for are 3D photos because they solve a fundamental issue with the current tech: IPD. The current tech (Apple Vision Pro included) uses two photos: one per eye. If the photos were taken from a distance that matches the distance between your eyes, then the effect is convincing. Otherwise, it looks a bit off. The other problem is that a big part of the 3D perception comes from parallax: how the image changes with head motions (even small motions). Techniques that are not limited to two fixed images, but instead allow us to create new views for small motions, are great for much more impressive 3D photos. With more input photos you get a “walkable photo”: a photo that you can take a few steps in, say if you are wearing a VR headset. I’m sure 3D Gaussian splatting is good for other things too, given the excitement around them. Backgrounds in movies maybe?

praveen9920 2y ago

One application I can think of is Google Street View. Gaussian splatting can potentially "smoothen" the transition between the images and make it look more realistic.

lawlessone 2y ago

>Where would you use 3D Gaussian splatting? The primary purpose of Gaussian splatting is to frontpage here every two weeks.