7 ms·
CityGaussian: Real-time high-quality large-scale scene rendering with Gaussians
- chpatrick 3y ago"The average speed is 36 FPS (tested on A100)." Real-Time if you have $8k I guess.
- RicoElectrico 3y ago"Two more papers down the line..." ;)
- Fauntleroy 3y agoIndeed, this very much looks like what we'll likely see from Google Earth within a decade—or perhaps half that.
- mortenjorck 3y agoI’ve seen very impressive Gaussian splatting demos of more limited urban geographies (a few city blocks) running on consumer hardware, so the reason this requires research-tier Nvidia hardware right now is probably down to LOD streaming. More optimization on that front, and this could plausibly come to Google Earth on current devices. “What a time to be alive” indeed!
- xyproto 3y ago2 years tops, since the technology is there, it would be a considerable improvement to Google Maps, and Google has the required resources.
- rallyforthesun 3y agoAs it seems the first 3DGS which uses Lods and blocks, there might be place for optimization. This might become useful for use cases in Virtual Production, probably not for mobiles.
- jsheard 3y agoGood ol' "SIGGRAPH realtime", when a graphics paper describes itself as achieving realtime speeds you always have to double check that they mean actually realtime and not "640x480 at 20fps on the most expensive hardware money can buy". Anything can be realtime if you set the bar low enough.
- phkahler 3y ago>> Anything can be realtime if you set the bar low enough. I was doing "realtime ray tracing" on Pentium class computers in the 1990s. I took my toy ray tracer and made an OLE control and put it inside a small Visual Basic app which handled keypress-navigation. It could run in a tiny little window (size of a large icon) at reasonable frame rates. Might even say it was using Visual Basic! So yeah "realtime" needs some qualifiers ;-)
- TeMPOraL 3y agoFair, but today it could probably run 30FPS full-screen at 2K resolution, without any special effort, on an average consumer-grade machine; better if ported to take advantage of the GPU. Moore's law may be dead in general, but computing power still increases (notwithstanding the software bloat that makes it seem otherwise), and it's still something to count on wrt. bleeding edge research demos.
- cchance 3y agoI mean A100's were cutting edge a year or so ago now we're at what H200 and B200 or is it 300's like it may be a year or 2 more but the A100 speed will trickle down to the average consumer as well.
- TeMPOraL 3y agoAnd, from the other end, research demonstrations tend to have a lot of low-hanging fruits wrt. optimization, which will get picked if the result is interesting enough.
- oivey 3y ago
- mywittyname 3y agoPresumably, this is can be used as the first stage in a pipeline. Take the models and textures generated from source data using this, cached it, and stream that data to clients for local rendering. Consumer GPUs are probably 2-3 generations out from being as capable as an A100.
- Legend2440 3y agoThere are no models or textures, it's just a point cloud of color blobs. You can convert it to a mesh, but in the process you'd lose the quality and realism that makes it interesting.
- littlestymaar 3y agoI chuckled a bit too when I saw it. By the way, what's the compute power difference between an A100 and a 4090?
- entropicdrifter 3y ago4090 is faster in terms of compute, but the A100 has 40GB of VRAM.
- enlyth 3y agoI believe the main advantage of the A100 is the memory bandwidth. Computationally the 4090 has a higher clock speed and more CUDA cores, so in that way it is faster. So for this specific application it really depends on where the bottleneck is
- pierotofy 3y agoA lot of 3DGS/Nerf research is like this unfortunately (ugh). Check https://github.com/pierotofy/OpenSplat https://github.com/pierotofy/OpenSplat for something you can run on your 10 year old laptop, even without a GPU! (I'm the author)
- somethingsome 3y agoI know, I don't get the fuzz either, I've coded real-time gaussian splat renderers >7 years ago with LOD and they were able to show any kind of point cloud. They worked with a basic 970 GTX on a big 3d screen and also on oculus dk2.
- kookamamie 3y agoIt's the old story of a an outsider group (AI researchers, in this case) re-inventing the wheel discovered ages ago by experts of the domain.
- m463 3y agootoh I remember those old GPU benchmarks that ran at 10 fps when they came out, then over time... https://www.techpowerup.com/forums/attachments/all-cards-png.325351/ https://www.techpowerup.com/forums/attachments/all-cards-png...
- datascienced 3y agoJust wait 2 years it’ll be on your phone.
- anigbrowl 3y agoYou gotta start somewhere
- aurareturn 3y agoI'm going to guess that the next-gen consumer GPU (5090) will be twice as fast as A100 and will not cost $8k. So I don't know see an insurmountable problem.
- diggan 3y agoNo, not unless Nvidia is thinking about financial suicide. The current split between "pro" and "consumer" isn't because it was impossible to avoid, it's because Nvidia is doing market segmentation in order to extract more money from the pro segment.
- rallyforthesun 3y agoReally advanced approach to render larger scenes with 3DGaussians, cant wait to test the code :-)
- 999900000999 3y agoExcited to see what license this is released under. Would love to see some open source games using this.
- jsheard 3y agoPerformance aside, someone needs to figure out a generalizable way to make the scenes dynamic before it will really be usable for games. History is littered with alternatives to triangles meshes that looked promising until we realised there's no efficient way to animate them.
- 999900000999 3y agoCan you explain what a dynamic is ? I was more thinking you'd run this tool, and then have an algorithm convert it( bake the mesh).
- lawlessone 3y agoThey probably mean animated, changeable etc. Like movement, or changes in lighting.
- CuriouslyC 3y agoEven if this doesn't replace triangles everywhere, I'm guessing it's still going to be the easiest way to generate a large volume of static art assets, which means we will see hybrid rendering pipelines.
- jsheard 3y agoAIUI these algorithms currently bake all of the lighting into the surface colors statically, which mostly works if the entire scene is constructed as one giant blob where nothing moves but if you wanted to render an individual NeRF asset inside an otherwise standard triangle-based pipeline then it would need to be more adaptable than that. Even if the asset itself isn't animated it would need to adapt to the local lighting at the bare minimum, which I haven't seen anyone tackle yet, the focus has been on the rendering-one-giant-static-blob problem. For hybrid pipelines to work the splatting algorithm would probably need to output the standard G-Buffer channels (unlit surface color, normal, roughness, etc) which can then go through the same lighting pass as the triangle-based assets, rather than the splatting algorithm trying to infer lighting by itself and inevitably getting a result that's inconsistent with how the triangle-based assets are lit. Think of those old cartoons where you could always tell when part of the scenery was going to move because the animation cel would stick out like a sore thumb against the painted background, that's the kind of illusion break you would get if the lighting isn't consistent.
- forrestthewoods 3y agoCan someone convince me that 3D gaussian splatting isn't a dead end? It's an order of magnitude too slow to render and order of magnitude too much data. It's like raster vs raytrace all over again. Raster will always be faster than raytracing. So even if raytracing gets 10x faster so too will raster. I think generating traditional geometry and materials from gaussian point clouds is maybe interesting. But photogrammetry has already been a thing for quite awhile. Trying to render a giant city in real time via splats doesn't feel like "the right thing". It's definitely cool and fun and exciting. I'm just not sure that it will ever be useful in practice? Maybe! I'm definitely not an expert so my question is genuine.
- kfarr 3y agoYes this has tons of potential. It's analogous but different to patented techniques used by Unreal engine. Performance is not the focus in most research at the moment. There isn't even alignment on unified format with compression yet. The potential for optimization is very clear and straightforward to adapt to many devices, it's similar to point cloud LOD, mesh culling, etc. Splat performance could be temporary competitive advantage for viewers, but similar to video decompression and other 3d standards that are made available via open source, it will likely become commonplace in a few years to have high quality high fps splat viewing on most devices as tablestakes. The next question is what are the applications thereof.
- Legend2440 3y agoNothing comes close to this for realism, it's like looking at a photo. Traditional photogrammetry really struggles with complicated scenes, and reflective or transparent surfaces.
- mschuetz 3y agoIt's currently unparalleled when it comes to realism as in realistic 3D reconstruction from the real world. Photogrammetry only really works for nice surfacic data, whereas gaussian splats work for semi-volumetric data such as fur, vegetation, particles, rough surfaces, and also for glossy/specular surfaces and volumes with strong subdivision surface properties, or generally stuff with materials that are strongly view-dependent.
- kfarr 3y agoNot quite the same thing, but over the weekend I hacked google maps 3d tiles (mesh) together with a gaussian splat and the effect is pretty similar and effective: Example 1 with code linked: https://twitter.com/kfarr/status/1773934700878561396 https://twitter.com/kfarr/status/1773934700878561396 Example 2 https://twitter.com/3dstreetapp/status/1775203540442697782 https://twitter.com/3dstreetapp/status/1775203540442697782
- sbarre 3y agoThis is super cool! Congrats on the PoC ...
- cchance 3y agoThats really cool is there a github with the code... getting errors on that first link in devtools Uncaught (in promise) Error: Failed to fetch resource https://tile.googleapis.com/v1/3dti https://tile.googleapis.com/v1/3dti...
- kfarr 3y agoProbably rate limited api calls given the hug of Twitter and HN. Capped at 1k per day see https://github.com/3DStreet/aframe-loader-3dtiles-component https://github.com/3DStreet/aframe-loader-3dtiles-component Code is available via glitch url
- aantix 3y agoWow, amazing work!
- aaroninsf 3y agoAre you on Bluesky? Would love to follow. But not, you know, over there.
- kfarr 3y agoClosest I can offer is https://sfba.social/@kfarr/112204337465256518 https://sfba.social/@kfarr/112204337465256518
- syrusakbary 3y agoGaussian splatting is truly amazing for 3d reconstruction. I can't wait to see once it's applied to the world of driverless vehicles and AI!
- jnsjjdk 3y agoThis does not look significantly better then e.g. cities skylines, especially since they neither zoomed in or out, always showing only a very limited frame Am I missing something?
- dartos 3y agoThis was rendered from photographs, I believe
- neuronexmachina 3y agoThis is a 3D reconstruction, rather than a game rendering.
- cchance 3y agoLOL this isn't a game engine, its real life photos being converted into gausian 3d views.
- jayd16 3y agoActually it sounds like its from renders of an Unreal demo. So synthetic photos from a game engine. LOL
- chankstein38 3y agoAll 3 of the other commenters are replying without having done any actual thought or research. The paper repeatedly references MatrixCity and another commenter above found this https://city-super.github.io/matrixcity/ https://city-super.github.io/matrixcity/ which also, I'd like to add, calls out that it's fully Synthetic. And, from what I understand, is extracted from Unreal Engine.
- boywitharupee 3y agowhat's the memory and compute requirements for this?
- speps 3y agoNote that the dataset from the video is called Matrix city. It's highly likely extracted from the Unreal Engine 5 Matrix demo released a few years ago. The views look very similar, so it's photorealistic but not from photos. EDIT: here it is, and I was right! https://city-super.github.io/matrixcity/ https://city-super.github.io/matrixcity/
- speps 3y agoReplying to myself with a question, as someone could have the answer: Would it be possible to create the splats without the training phase? If we have a fully modelled scene in Unreal Engine for example (like Matrix city), you shouldn't need to spend all the time training to recreate the data...
- kfarr 3y agoYes, and then it gets interesting to think about procedurally generated splats, such as spawning a randomized distribution of grass splats on a field for example
- dkjaudyeqooe 3y agoTo me the big issue is image quality versus generative efficiency. If splats make rending complicated scenes efficient without requiring a lot of data/calculation "scaffolding" then you could do almost everything procedurally, maybe using AI models to fill in definitional detail.
- sorenjan 3y agoYes, it's possible to create gaussian splats from a mesh. See for example step 3 in SuGaR: https://imagine.enpc.fr/~guedona/sugar/ https://imagine.enpc.fr/~guedona/sugar/
- fudged71 3y agoAre you referring to the gaussian splat rasterizer?
- mhuffman 3y agoQuick question for anyone that may have more technical insight, is Gaussian Splatting the technology that Unreal Engine has been using to have such jaw dropping demos with their new releases?
- andybak 3y agoNo. Mostly unrelated.
- rmccue 3y agoUnreal Engine 5 is a combination of a few technologies: * Virtualised geometry (Nanite) allowing very detailed models * Very high quality models and textures from photogrammetry (Megascans) * Real-time global illumination (Lumen) Combining these is what allows the very high fidelity demos, as they’re each step changes from the previous techniques in Unreal. Megascans (and the Quixel library) are a big part of the “photorealness” of these demos, because they’re basically literally photos.
- deleted 3y ago[deleted]
- notachatbot1234 3y agoNo but here are some nice talks on the Nanite geometry technology of UE: https://www.youtube.com/watch?v=NRnj_lnpORU https://www.youtube.com/watch?v=NRnj_lnpORU https://www.youtube.com/watch?v=eviSykqSUUw https://www.youtube.com/watch?v=eviSykqSUUw
- satvikpendem 3y agoFunny to see just how prolific Gauss was since so many things are named after him and continue to be newly named after him, such as this example of Gaussian splatting, which, while he obviously didn't directly invent it, contributed to the mathematics of it significantly.
- Sohcahtoa82 3y ago"Monte Carlo" is in a similar space. https://en.wikipedia.org/wiki/Monte_Carlo_(disambiguation) https://en.wikipedia.org/wiki/Monte_Carlo_(disambiguation) 8 entries in "Science and Technology" alone.
- dukeofdoom 3y agoDoes anyone know how to add motion blur to a game. I'm learning pygame. Say I'm making Mario in pygame, and when Mario jumps, I want him to look blurry. I mean I can take an average of 9 pixels, and create a blurry version of Mario. But is that how it's that usually done in other games. Since like a lot of games are supper sharp, with no motion blur. I'm wondering if its even done. It's kind of big deal in film, and the need to shoot at 25fps to achieve cinematic motion blur.
- jayd16 3y agoRender the motion vector of objects to another render texture. (ie calculate the velocity of each object and render that as a color) Use that to define the amplitude and direction of your blur effect in a post pass. And you might want it to be the motion relevant to the camera. For Mario, probably not, but for an FPS you want to edges of the screen to blur as the camera moves forward.
- dukeofdoom 3y agoThank you.
- gmerc 3y agoRelated https://city-super.github.io/octree-gs/ https://city-super.github.io/octree-gs/