10 ms·

Level of Gaussians: Real-Time View Synthesis for Millions of Square Meters

183 points by corysama 2y ago

corysama 2y ago
I got excited because the code was just released. But, apparently the paper is still not available? Sorry...
blovescoffee 2y ago
Crazy good results but without the paper (which the link at the time just goes back to the site) it's a bit difficult to check how good. What data is required, how long are training runs/how many steps?
- logtempo 2y ago
  Using 200 photos taken with a conventional camera, at a refresh rate of 105 frames per second - the quality of video game images - the result gives the illusion of walking through the video. Better still, if you zoom in, you can see finer details, such as the spokes of a bicycle wheel, in excellent detail. It use neural network techniques, but it's not strictly using NN. it do the same result as Nerf from google in 30min, nvidia result in 7minute. It can achieve more than 100fps if you let it train longer. https://www.inria.fr/fr/3d-gaussian-splatting-vision-ordinateur-informatique-graphique https://www.inria.fr/fr/3d-gaussian-splatting-vision-ordinat...
- was_a_dev 2y ago
  The fact we have released code before the paper is wild. Typically the promise of open sourced code never comes to fruition
speps 2y ago
Actual title is: Real-Time View Synthesis for Large Scenes with Millions of Square Meters Which makes more sense than: Real-Time View Synthesis for Square Meters
- corysama 2y ago
  Title edited. Thanks. I couldn't fit the whole title. But, didn't think I cut out "Millions of"...
Retr0id 2y ago
I hope the next-gen google earth looks something like this.
londons_explore 2y ago
Please Google, implement this in google maps (especially on mobile). It's been over a decade and we're still stuck with 2D maps and boxy untextured buildings.
- cubefox 2y ago
  Google uses texture mapped polygons instead of 3D Gaussians, so this wouldn't work for Google Maps. But there actually is a collection of libraries which does the same thing for polygonal data: https://vcg.isti.cnr.it/nexus/ https://vcg.isti.cnr.it/nexus/ One of the guys working on this is Federico Ponchio. His 2008 PhD thesis, which provided the core insight for Unreal Engine's Nanite, is referenced at bottom.
  - londons_explore 2y ago
    > Google uses texture mapped polygons instead of 3D Gaussians, Time to switch I'd say... Polygons are a poor fit, especially for trees and windows and stuff that needs to be semitransparent/fluffy. I suspect the gaussians will compress better, and give better visual quality for a given amount of data downloaded and GPU VRAM. (the current polygon model uses absolutely loads of both, leading to a poor experience for those without the highest end computers and fast internet connections).
- leodriesch 2y ago
  I am really impressed by the Apple Maps implementation. I think it also uses textured polygons, but does so in a very good looking way and at 120 fps on an iPhone, showing even a whole city in textured 3d.
  - martinkallstrom 2y ago
    Apple bought a Swedish startup called C3 and their became 3D part of Apple Maps. That startup was a spin-off from Saab Aerospace, who had developed a vision system for terrain-following missiles. Saab ran a project with the municipal innovation agency in Linköping and the result was that they decided this tech should be possible to find civilian use cases for. C3 decided to fly small Cessnas in grids across a few major cities and also Hoover Dam, and built a ton of code on top of the already extremely solid foundation from Saab. The timing was impeccable (now many years ago) and they managed to get Microsoft, Apple and Samsung into a bidding war which drove up the price. But it was worth it for Apple to have solid 3D in Apple Maps and the tech has stood the test of time.
lend000 2y ago
Looks even better than Microsoft flight simulator. Awesome!
jiggawatts 2y ago
So this is just Level-of-Detail (LoD) implemented for Gaussian splats? Impressive results, but I would have figured this is an obvious next-step... Also, is it bad that the first thing I thought of was that commanders in the Ukraine war could use this? E.g.: stitch together the video streams from thousands of drones to build up an up-to-date view of the battlefield?
angusturner 2y ago
Can anyone familiar with 3d graphics speculate what would be required to implement this into a game engine? I'm guessing that adding physics, collision-detection etc. on top of this is non-trivial compared to using a mesh? But I feel like for stuff like tree foliage (where maybe you don't care about collisions?), this would be really awesome, given the limitations of polygons. + also just any like background scenery, stuff out of the player's reach.
- modeless 2y ago
  It's easy to render these in a game engine. I'm sure physics and collision detection are possible. The big, huge, gigantic issue is actually lighting. These scenes come with real world lighting baked in. This is great because it looks amazing, it's 100% correct, far better than the lighting computed by any game engine or even movie-quality offline ray tracer. This is a big part of why they look so good! But it's also a curse. Games need to be interactive. When things move, lighting changes. Even something as simple as opening a door can have a profound effect on lighting. Anything that moves changes the lighting on itself and everything around it. Let alone moving actual lights around, changing the time of day, etc. There's absolutely no way to change the baked-in lighting in one of these captures in a high quality way. I've seen several papers that attempt it and the results all suck. It's not the fault of the researchers, it's a very hard problem. There are two main issues: One, in order to perfectly re-light a scene you first have to de-light it, that is, compute the lighting-independent BRDF of every surface. The capture itself doesn't even contain enough information to do this in an unambiguous way. You can't know for sure how a surface would react under different lighting conditions than were present in the pictures that made up the original scan. Maybe in theory you can guess well enough in most cases and extrapolate, and AI can likely help a lot here, but in practice we are far away from good quality so far. Two, given the BRDF of all surfaces and a set of new lights, you have to apply the new lighting to the scene. Real-time solutions for lighting are very approximate and won't be anywhere near the quality of the lighting in the original scan. So you'll lose some of that photorealistic quality when you do this, even if your BRDFs are perfect (they won't be). It will end up looking like regular game graphics instead of the picture-perfect scans you want. If you try to blend the new lighting with the original lighting, the boundaries will probably be obvious. You're competing with perfection! Even offline rendering would struggle to match the quality of the baked-in lighting in these captures. To me the ultimate solution needs to involve AI. Analytically relighting everything perfectly is infeasible, but AI can likely do approximate lighting that looks more plausible in most cases, especially when trying to match captured natural lighting. I'm not sure exactly how it will work, but AI is already being used in rendering and its use will only increase.
mrwyz 2y ago
Cool, but not touching this; no license and requires Inria's proprietary rasterizer. People should stop basing all of this new research on proprietary software, when we have open source implementations [1][2]. [1] gsplat: https://github.com/nerfstudio-project/gsplat https://github.com/nerfstudio-project/gsplat [2] opensplat: https://github.com/pierotofy/opensplat https://github.com/pierotofy/opensplat
- cubefox 2y ago
  I'm surprised anything in 3D Gaussian splatting uses a rasterizer. I thought those were only used for polygonal data.
  - VelesDude 2y ago
    I mean technically rasterization means taking any vector data and plotting it in a 2D space... so I guess it is correct. But yes, I know what you are getting at. This would normally be done via a software/shader pipeline rather than a GPU's polygonal process.
  - reasonableklout 2y ago
    Rasterization is actually why 3D Gaussian Splats have been so successful. Being able to render 3DGS scenes by iterating over the objects and drawing the pixels each one covers is much faster than ray-marching every pixel, which is how neural radiance fields (the last hot 3D reconstruction technology) are rendered.
- CuriousCosmic 2y ago
  It has a license now fwiw. It's a pretty basic "free for non-commercial use, contact us for commercial use" license. And the Inria rasterizer is not proprietary either. It's non-commercial open source with the option to purchase a commercial license. These are perfectly reasonable tech stacks for research projects to build off of. If you have an issue with the license, implement it yourself based on the papers (which all outline the necessary details to do so).
  - deleted 2y ago
littlestymaar 2y ago
Gaussian splatting feel magical, and with 4D Gaussian splatting now being a thing, 3D movies that are actually 3D, and in which you can navigate could be a reality in the coming years. (And I suspect the first use-case will be porn, as usual).
- datascienced 2y ago
  Movies can become games as well.
KaiserPro 2y ago
Sorry to be naive, but isn't this basically applying pointcloud decimation to achieve dynamic level of detail? Am I missing something or is there a new concept that doesn't exist in standard point cloud renderers?
- i_love_cookies 2y ago
  [dead]
cchance 2y ago
Every time I see these I wonder what if you use those massive 50 gigapixel images and a few of them and throw it into gaussians

183 points by corysama 2y ago

I got excited because the code was just released. But, apparently the paper is still not available? Sorry...

Crazy good results but without the paper (which the link at the time just goes back to the site) it's a bit difficult to check how good. What data is required, how long are training runs/how many steps?

logtempo 2y ago
Using 200 photos taken with a conventional camera, at a refresh rate of 105 frames per second - the quality of video game images - the result gives the illusion of walking through the video. Better still, if you zoom in, you can see finer details, such as the spokes of a bicycle wheel, in excellent detail. It use neural network techniques, but it's not strictly using NN. it do the same result as Nerf from google in 30min, nvidia result in 7minute. It can achieve more than 100fps if you let it train longer. https://www.inria.fr/fr/3d-gaussian-splatting-vision-ordinateur-informatique-graphique https://www.inria.fr/fr/3d-gaussian-splatting-vision-ordinat...
was_a_dev 2y ago
The fact we have released code before the paper is wild. Typically the promise of open sourced code never comes to fruition

speps 2y ago

Actual title is: Real-Time View Synthesis for Large Scenes with Millions of Square Meters Which makes more sense than: Real-Time View Synthesis for Square Meters

corysama 2y ago
Title edited. Thanks. I couldn't fit the whole title. But, didn't think I cut out "Millions of"...

Retr0id 2y ago

I hope the next-gen google earth looks something like this.

londons_explore 2y ago

Please Google, implement this in google maps (especially on mobile). It's been over a decade and we're still stuck with 2D maps and boxy untextured buildings.

cubefox 2y ago
Google uses texture mapped polygons instead of 3D Gaussians, so this wouldn't work for Google Maps. But there actually is a collection of libraries which does the same thing for polygonal data: https://vcg.isti.cnr.it/nexus/ https://vcg.isti.cnr.it/nexus/ One of the guys working on this is Federico Ponchio. His 2008 PhD thesis, which provided the core insight for Unreal Engine's Nanite, is referenced at bottom.
- londons_explore 2y ago
  > Google uses texture mapped polygons instead of 3D Gaussians, Time to switch I'd say... Polygons are a poor fit, especially for trees and windows and stuff that needs to be semitransparent/fluffy. I suspect the gaussians will compress better, and give better visual quality for a given amount of data downloaded and GPU VRAM. (the current polygon model uses absolutely loads of both, leading to a poor experience for those without the highest end computers and fast internet connections).
leodriesch 2y ago
I am really impressed by the Apple Maps implementation. I think it also uses textured polygons, but does so in a very good looking way and at 120 fps on an iPhone, showing even a whole city in textured 3d.
- martinkallstrom 2y ago
  Apple bought a Swedish startup called C3 and their became 3D part of Apple Maps. That startup was a spin-off from Saab Aerospace, who had developed a vision system for terrain-following missiles. Saab ran a project with the municipal innovation agency in Linköping and the result was that they decided this tech should be possible to find civilian use cases for. C3 decided to fly small Cessnas in grids across a few major cities and also Hoover Dam, and built a ton of code on top of the already extremely solid foundation from Saab. The timing was impeccable (now many years ago) and they managed to get Microsoft, Apple and Samsung into a bidding war which drove up the price. But it was worth it for Apple to have solid 3D in Apple Maps and the tech has stood the test of time.

lend000 2y ago

Looks even better than Microsoft flight simulator. Awesome!

jiggawatts 2y ago

So this is just Level-of-Detail (LoD) implemented for Gaussian splats? Impressive results, but I would have figured this is an obvious next-step... Also, is it bad that the first thing I thought of was that commanders in the Ukraine war could use this? E.g.: stitch together the video streams from thousands of drones to build up an up-to-date view of the battlefield?

angusturner 2y ago

Can anyone familiar with 3d graphics speculate what would be required to implement this into a game engine? I'm guessing that adding physics, collision-detection etc. on top of this is non-trivial compared to using a mesh? But I feel like for stuff like tree foliage (where maybe you don't care about collisions?), this would be really awesome, given the limitations of polygons. + also just any like background scenery, stuff out of the player's reach.

modeless 2y ago
It's easy to render these in a game engine. I'm sure physics and collision detection are possible. The big, huge, gigantic issue is actually lighting. These scenes come with real world lighting baked in. This is great because it looks amazing, it's 100% correct, far better than the lighting computed by any game engine or even movie-quality offline ray tracer. This is a big part of why they look so good! But it's also a curse. Games need to be interactive. When things move, lighting changes. Even something as simple as opening a door can have a profound effect on lighting. Anything that moves changes the lighting on itself and everything around it. Let alone moving actual lights around, changing the time of day, etc. There's absolutely no way to change the baked-in lighting in one of these captures in a high quality way. I've seen several papers that attempt it and the results all suck. It's not the fault of the researchers, it's a very hard problem. There are two main issues: One, in order to perfectly re-light a scene you first have to de-light it, that is, compute the lighting-independent BRDF of every surface. The capture itself doesn't even contain enough information to do this in an unambiguous way. You can't know for sure how a surface would react under different lighting conditions than were present in the pictures that made up the original scan. Maybe in theory you can guess well enough in most cases and extrapolate, and AI can likely help a lot here, but in practice we are far away from good quality so far. Two, given the BRDF of all surfaces and a set of new lights, you have to apply the new lighting to the scene. Real-time solutions for lighting are very approximate and won't be anywhere near the quality of the lighting in the original scan. So you'll lose some of that photorealistic quality when you do this, even if your BRDFs are perfect (they won't be). It will end up looking like regular game graphics instead of the picture-perfect scans you want. If you try to blend the new lighting with the original lighting, the boundaries will probably be obvious. You're competing with perfection! Even offline rendering would struggle to match the quality of the baked-in lighting in these captures. To me the ultimate solution needs to involve AI. Analytically relighting everything perfectly is infeasible, but AI can likely do approximate lighting that looks more plausible in most cases, especially when trying to match captured natural lighting. I'm not sure exactly how it will work, but AI is already being used in rendering and its use will only increase.

mrwyz 2y ago

Cool, but not touching this; no license and requires Inria's proprietary rasterizer. People should stop basing all of this new research on proprietary software, when we have open source implementations [1][2]. [1] gsplat: https://github.com/nerfstudio-project/gsplat https://github.com/nerfstudio-project/gsplat [2] opensplat: https://github.com/pierotofy/opensplat https://github.com/pierotofy/opensplat

cubefox 2y ago
I'm surprised anything in 3D Gaussian splatting uses a rasterizer. I thought those were only used for polygonal data.
- VelesDude 2y ago
  I mean technically rasterization means taking any vector data and plotting it in a 2D space... so I guess it is correct. But yes, I know what you are getting at. This would normally be done via a software/shader pipeline rather than a GPU's polygonal process.
- reasonableklout 2y ago
  Rasterization is actually why 3D Gaussian Splats have been so successful. Being able to render 3DGS scenes by iterating over the objects and drawing the pixels each one covers is much faster than ray-marching every pixel, which is how neural radiance fields (the last hot 3D reconstruction technology) are rendered.
CuriousCosmic 2y ago
It has a license now fwiw. It's a pretty basic "free for non-commercial use, contact us for commercial use" license. And the Inria rasterizer is not proprietary either. It's non-commercial open source with the option to purchase a commercial license. These are perfectly reasonable tech stacks for research projects to build off of. If you have an issue with the license, implement it yourself based on the papers (which all outline the necessary details to do so).
- deleted 2y ago

littlestymaar 2y ago

Gaussian splatting feel magical, and with 4D Gaussian splatting now being a thing, 3D movies that are actually 3D, and in which you can navigate could be a reality in the coming years. (And I suspect the first use-case will be porn, as usual).

datascienced 2y ago
Movies can become games as well.

KaiserPro 2y ago

Sorry to be naive, but isn't this basically applying pointcloud decimation to achieve dynamic level of detail? Am I missing something or is there a new concept that doesn't exist in standard point cloud renderers?

i_love_cookies 2y ago
[dead]

cchance 2y ago

Every time I see these I wonder what if you use those massive 50 gigapixel images and a few of them and throw it into gaussians