7 ms·

Gaussian splatting is such an impressive technique. I wish it finds real-world applications useful for average person -- right now it's probably the best way to

Gaussian splatting is such an impressive technique. I wish it finds real-world applications useful for average person -- right now it's probably the best way to show photorealistic scenes in VR. There ought to be more usages out there.

hapofuino 2y ago
For an average person, I suspect that some real estate listings will adopt Gaussian splatting soon. Many newer listings include photogrammetry already, and splatting will provide a significant improvement to existing solutions.
- fallat 2y ago
  I'm just starting to use photogrammy in hobbyist game maps to provide "chill maps". Unfortunately splatting will not be possible there.
dwallin 2y ago
There's a lot of use cases, and companies already using photogrammetry techniques have adopted it extremely quickly, but for many of these businesses the usage of gaussian splats is just a technical implementation detail, resulting in quality / feature gains, but doesn't as of now unlock entirely new business models.
- jesol 2y ago
  It's true, Gaussian Splatting is just an alternative to meshing a pointcloud for companies which currently rely on photogrammetry or lidar (Lidar works well as a basis for splatting when there's reference images taken as part of the scan). But I think that misses all the new opportunities that exist with Gaussian Splatting, which really just don't with existing techniques. Gaussian Splats are able to handle more heterogeneous information sources, allowing more sources to help splat an environment. Devices like drones, surveillance cameras, or autonomous systems can be used to create or incrementally update a Gaussian Splat; and there's interesting work to allow them to locate themselves within the splat, not just to show themselves but also to place vision ML outputs into it (such as object detection or segmentation results). Up till now nearly all digital representations of physical environments are either based off the original designs (by things like CAD or BIM files), or are an approximation of the environment (from photogrammetry or Lidar scans). CAD and BIM files suffer from drift, the real environment almost never perfectly matches the design files, small (and large) changes are made; and many times those files aren't even available if the structure isn't new. Photogrammetry and Lidar scans struggle because their output is a pointcloud, and it's very difficult to accurately mesh a pointcloud (Matterport only partially solved this problem and sold for $1.6B). Gaussian Splats overcome these issues; they're comparatively easy to generate for any environment, and allow for very accurate and easy viewing from any angle. I think the Digital Twin space will be turned upside down, and they could potentially even cause huge changes in autonomous and semi-autonomous factories, warehouses, and depots. A single Gaussian Splat could be the source of truth that many autonomous vehicles update through their separate SLAM systems. Operators then would have access to this splat (and it's history) as a source of truth for the environment. Then, using techniques like iComMa[1], it may be possible to directly align XR devices into the Gaussian Splat; allowing operators direct access to location-based information generated by the environment. That's a lot of words to say: Gaussian Splatting is a very neat new technology that could really underpin many future technologies, I'm really excited about it [1]: https://yuansun-xjtu.github.io/iComMa.io/ https://yuansun-xjtu.github.io/iComMa.io/
  - dwallin 2y ago
    I do agree that new use cases are emerging and it will probably enable tons of new businesses. I'm very gung-ho about the technology myself as well. I guess what I'm trying to say is that the new businesses that emerge because of this are not necessarily going to advertise that they use gaussian splats to do it, it's not a buzzy enough term, and many of the industries it serves just care about the results it delivers. Your average tech person is unlikely to hear much about it. Your average graphics engineer will have probably heard about it, but not know about all the use cases that are leveraging it. And your average person in the industry it is changing won't know what is causing the change (they will probably assign it to the nebulous ai bucket). I fully expect gaussian splats to be a quiet revolution.
    - jesol 2y ago
      Yeah, I see your point. I'd be surprised of Gaussian Splatting didn't make it into the advertising for Digital Twin services if/when they add it (like Bently's iTwin or Dassault's Virtual Twin). Whether that translates more broadly into the market, I don't know. On the other hand, I'm playing with the idea of a platform which provides a Gaussian Splat based Digital Twin of an environment so other systems can utilize it to share location-based information. Even though I don't think it'll be possible to build without utilizing Gaussian Splatting; splatting may not end up in any of the pitches or advertising directly.
  - twelvechairs 2y ago
    This is conflating splatting with more general pointcloud data. Splatting is fundamentally about viewing pointcloud data. That's great. But it doesn't deal with all the other functions virtual twins need pointclouds for (e.g. design vs real world conformance). Pointclouds themselves are proving hugely useful in a number of fields but vary considerably in form and application often based on how they are captured (e.g. LiDAR vs SfM photogrammetry) Visualising pointclouds effectively has been a major problem which splatting really solves elegantly so it will be a major practical advance when splatting is added to cad software and javascript map visualisation libraries.
  - kragen 2y ago
golergka 2y ago
I may be mistaken, but it looks like it would only works for completely static scenes with pre-baked lighting.
- crubier 2y ago
  There have been an implementation of GS for video since day one. They have also included spherical harmonics for view dependence since day one.
- dwallin 2y ago
  It's important to disentangle this specific approach of generating a gaussian splatting model from 2d images, with gaussian splats as a general rendering technique. There is nothing that prevents gaussian splatting from being used dynamically. There are a variety of approaches to extend gaussian splats into the time dimension to capture and represent a 3d scene over time. The challenges here about how to capture sufficient scene data (or use ai to fill in insufficient data) and how to compress it. There are also techniques that enable dynamic simulations, or real time animation of collections of splats. Also, adding un-baked lighting to gaussian splats is not particularly hard, you can already throw slats into several game engines / 3d renderers and add new lights to them. The hard part of relighting is taking an existing capture of a scene with baked-in lighting and deriving the resulting material properties and lighting sources. This isn't directly related to gaussian splats themselves though, you would have a similar problem recovering the base materials and lights from a 3d mesh with baked-in lighting textures. This really falls under a separate category of techniques called "inverse rendering". If anything, gaussian splats give us a new tool to help with these sorts of problems. Honestly the biggest remaining roadblocks to more elaborate and widespread uses of gaussians as a rendering method are probably storage and performance related. And I'm optimistic these will be convincingly solved, triangle rasterization has had many orders of magnitude more research, optimization, and custom hardware built around it.
- jtolmar 2y ago
  I don't know if anyone has actually done it yet, but for scans of models in controlled lighting environments, you could use higher dimensional Gaussian Splatting to get more accurate lighting than standard techniques. (Similar to the 4D version with time, use higher dimensions to encode light source coordinates.)
crazygringo 2y ago
Are there any apps that use gaussian splatting in VR? Is there something I can download on my Meta Quest to try it out?
- dwallin 2y ago
  I’ve heard about this one but don’t have an appropriate setup to test it myself: https://www.gracia.ai/ https://www.gracia.ai/
3abiton 2y ago
How does it benefit VR specifically?
- ryandamm 2y ago
  It's more expressive than mesh-and-texture, which is the default: it can represent things with poorly defined edges or transparency. Also, because it's solved via GD, you tend to get better fidelity than traditional photogrammetry that relies on MV correspondence. But also you're just throwing a ton of primitives at the problem; high quality scenes typically have millions of splats. That's a lot of data, so it's no wonder it can be pretty photorealistic. (Still impressive, though.)
skykooler 2y ago
Doesn't VR generally require high framerates at high resolution? Gaussian splatting seems to be one of those borderline-realtime things right now which can't run at 4k120fps yet on consumer hardware.
- ryandamm 2y ago
  It renders pretty fast, actually. The knock on splats is that they create huge file sizes for video data.
eproxus 2y ago
It’s not exactly “average person” usage but Corridor Digital used in one of their video productions to create a looping, infinite corridor with proper reflections and lighting etc.: https://www.youtube.com/watch?v=GaGcLhhhbDs https://www.youtube.com/watch?v=GaGcLhhhbDs
CamperBob2 2y ago
An obvious application might be to make Google Street View not suck. They don't seem to be actively maintaining that anymore, unfortunately.

hapofuino 2y ago

For an average person, I suspect that some real estate listings will adopt Gaussian splatting soon. Many newer listings include photogrammetry already, and splatting will provide a significant improvement to existing solutions.

fallat 2y ago
I'm just starting to use photogrammy in hobbyist game maps to provide "chill maps". Unfortunately splatting will not be possible there.

dwallin 2y ago

There's a lot of use cases, and companies already using photogrammetry techniques have adopted it extremely quickly, but for many of these businesses the usage of gaussian splats is just a technical implementation detail, resulting in quality / feature gains, but doesn't as of now unlock entirely new business models.

jesol 2y ago
It's true, Gaussian Splatting is just an alternative to meshing a pointcloud for companies which currently rely on photogrammetry or lidar (Lidar works well as a basis for splatting when there's reference images taken as part of the scan). But I think that misses all the new opportunities that exist with Gaussian Splatting, which really just don't with existing techniques. Gaussian Splats are able to handle more heterogeneous information sources, allowing more sources to help splat an environment. Devices like drones, surveillance cameras, or autonomous systems can be used to create or incrementally update a Gaussian Splat; and there's interesting work to allow them to locate themselves within the splat, not just to show themselves but also to place vision ML outputs into it (such as object detection or segmentation results). Up till now nearly all digital representations of physical environments are either based off the original designs (by things like CAD or BIM files), or are an approximation of the environment (from photogrammetry or Lidar scans). CAD and BIM files suffer from drift, the real environment almost never perfectly matches the design files, small (and large) changes are made; and many times those files aren't even available if the structure isn't new. Photogrammetry and Lidar scans struggle because their output is a pointcloud, and it's very difficult to accurately mesh a pointcloud (Matterport only partially solved this problem and sold for $1.6B). Gaussian Splats overcome these issues; they're comparatively easy to generate for any environment, and allow for very accurate and easy viewing from any angle. I think the Digital Twin space will be turned upside down, and they could potentially even cause huge changes in autonomous and semi-autonomous factories, warehouses, and depots. A single Gaussian Splat could be the source of truth that many autonomous vehicles update through their separate SLAM systems. Operators then would have access to this splat (and it's history) as a source of truth for the environment. Then, using techniques like iComMa[1], it may be possible to directly align XR devices into the Gaussian Splat; allowing operators direct access to location-based information generated by the environment. That's a lot of words to say: Gaussian Splatting is a very neat new technology that could really underpin many future technologies, I'm really excited about it [1]: https://yuansun-xjtu.github.io/iComMa.io/ https://yuansun-xjtu.github.io/iComMa.io/
- dwallin 2y ago
  I do agree that new use cases are emerging and it will probably enable tons of new businesses. I'm very gung-ho about the technology myself as well. I guess what I'm trying to say is that the new businesses that emerge because of this are not necessarily going to advertise that they use gaussian splats to do it, it's not a buzzy enough term, and many of the industries it serves just care about the results it delivers. Your average tech person is unlikely to hear much about it. Your average graphics engineer will have probably heard about it, but not know about all the use cases that are leveraging it. And your average person in the industry it is changing won't know what is causing the change (they will probably assign it to the nebulous ai bucket). I fully expect gaussian splats to be a quiet revolution.
  - jesol 2y ago
    Yeah, I see your point. I'd be surprised of Gaussian Splatting didn't make it into the advertising for Digital Twin services if/when they add it (like Bently's iTwin or Dassault's Virtual Twin). Whether that translates more broadly into the market, I don't know. On the other hand, I'm playing with the idea of a platform which provides a Gaussian Splat based Digital Twin of an environment so other systems can utilize it to share location-based information. Even though I don't think it'll be possible to build without utilizing Gaussian Splatting; splatting may not end up in any of the pitches or advertising directly.
- twelvechairs 2y ago
  This is conflating splatting with more general pointcloud data. Splatting is fundamentally about viewing pointcloud data. That's great. But it doesn't deal with all the other functions virtual twins need pointclouds for (e.g. design vs real world conformance). Pointclouds themselves are proving hugely useful in a number of fields but vary considerably in form and application often based on how they are captured (e.g. LiDAR vs SfM photogrammetry) Visualising pointclouds effectively has been a major problem which splatting really solves elegantly so it will be a major practical advance when splatting is added to cad software and javascript map visualisation libraries.
- kragen 2y ago

golergka 2y ago

I may be mistaken, but it looks like it would only works for completely static scenes with pre-baked lighting.

crubier 2y ago
There have been an implementation of GS for video since day one. They have also included spherical harmonics for view dependence since day one.
dwallin 2y ago
It's important to disentangle this specific approach of generating a gaussian splatting model from 2d images, with gaussian splats as a general rendering technique. There is nothing that prevents gaussian splatting from being used dynamically. There are a variety of approaches to extend gaussian splats into the time dimension to capture and represent a 3d scene over time. The challenges here about how to capture sufficient scene data (or use ai to fill in insufficient data) and how to compress it. There are also techniques that enable dynamic simulations, or real time animation of collections of splats. Also, adding un-baked lighting to gaussian splats is not particularly hard, you can already throw slats into several game engines / 3d renderers and add new lights to them. The hard part of relighting is taking an existing capture of a scene with baked-in lighting and deriving the resulting material properties and lighting sources. This isn't directly related to gaussian splats themselves though, you would have a similar problem recovering the base materials and lights from a 3d mesh with baked-in lighting textures. This really falls under a separate category of techniques called "inverse rendering". If anything, gaussian splats give us a new tool to help with these sorts of problems. Honestly the biggest remaining roadblocks to more elaborate and widespread uses of gaussians as a rendering method are probably storage and performance related. And I'm optimistic these will be convincingly solved, triangle rasterization has had many orders of magnitude more research, optimization, and custom hardware built around it.
jtolmar 2y ago
I don't know if anyone has actually done it yet, but for scans of models in controlled lighting environments, you could use higher dimensional Gaussian Splatting to get more accurate lighting than standard techniques. (Similar to the 4D version with time, use higher dimensions to encode light source coordinates.)

crazygringo 2y ago

Are there any apps that use gaussian splatting in VR? Is there something I can download on my Meta Quest to try it out?

dwallin 2y ago
I’ve heard about this one but don’t have an appropriate setup to test it myself: https://www.gracia.ai/ https://www.gracia.ai/

3abiton 2y ago

How does it benefit VR specifically?

ryandamm 2y ago
It's more expressive than mesh-and-texture, which is the default: it can represent things with poorly defined edges or transparency. Also, because it's solved via GD, you tend to get better fidelity than traditional photogrammetry that relies on MV correspondence. But also you're just throwing a ton of primitives at the problem; high quality scenes typically have millions of splats. That's a lot of data, so it's no wonder it can be pretty photorealistic. (Still impressive, though.)

skykooler 2y ago

Doesn't VR generally require high framerates at high resolution? Gaussian splatting seems to be one of those borderline-realtime things right now which can't run at 4k120fps yet on consumer hardware.

ryandamm 2y ago
It renders pretty fast, actually. The knock on splats is that they create huge file sizes for video data.

eproxus 2y ago

It’s not exactly “average person” usage but Corridor Digital used in one of their video productions to create a looping, infinite corridor with proper reflections and lighting etc.: https://www.youtube.com/watch?v=GaGcLhhhbDs https://www.youtube.com/watch?v=GaGcLhhhbDs

CamperBob2 2y ago

An obvious application might be to make Google Street View not suck. They don't seem to be actively maintaining that anymore, unfortunately.