8 ms·
Compressing Gaussian Splats
- scheeseman486 3y agoThe moment I saw the use of motion vectors with gaussian splats I started wondering if it was a viable way to achieve high performance, high quality spatial video. Instead of capturing from all angles like most current approaches (which seems more geared to create content that is used in place of a 3D model) why not capture from a fixed perspective, using an array of cameras covering about 1m square to allow for slop in head position, providing parallax and perspective correct rendering. I'd presume that'd make it even more compressible too, since you could get rid of any data that isn't visible from outside that frustum. It would be a true hologram, completely supplanting 180/360 stereo video. Imagine Avatar on such a format. Or, lets be real, porn.
- cousin_it 3y agoI wonder what style of non-realistic graphics would be best suited to gaussian splats. Like how pixel art and low-poly came from limitations of yesterday's technologies, but became distinctive styles in their own right.
- deltaburnt 3y agoSounds kind of similar to https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
- scheeseman486 3y agoThat was definitely on my mind, that uses lightfields though. The file sizes are ludicrous.
- deltaburnt 3y agoYep, though the layered mesh post processing is nice because most consumer hardware at this point is pretty good at rendering plain ol meshes. The big challenge for mass adoption of nerfs/gsplats techniques will be getting them running and viewable on a phone web browser.
- jayd16 3y agoSo basically what it is now but with view and occlusion culling?
- scheeseman486 3y agoMore compression with considerations made to allow for streaming and some kind of equivalent to I/P/B-frames, spatial audio, metadata (chapters, aspect ratios etc) in a standardized file container. At the moment it's all rather basic and bespoke.
- krenzo 3y agoHave you seen this? https://shenhanqian.github.io/gaussian-avatars https://shenhanqian.github.io/gaussian-avatars
- ladberg 3y agoKinda disappointing that the spherical harmonic data is just totally thrown out. It's a pretty big factor in making gaussian splats look so good, and I'm sure there are interesting ways of compressing the data!
- slimbuck 3y agoIt's definitely possible to compress/palletise the SH bands (see https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/ https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-m... for one approach). We just found that splats encode some view-dependent shininess anyway (for example the guitar body https://playcanvas.com/model-viewer?load=https://code.playcanvas.com/viewer/guitar-cleaned.ply https://playcanvas.com/model-viewer?load=https://code.playca...) and so considered the extra data and rendering cost of SH not worth it for now. It would be quite straightforward to add SH to the PLY file and we'll definitely consider it in future!
- ladberg 3y agoOh nice! I didn't realize those kinds of effects were possible without SH
- jayd16 3y agoAre the baked in reflections technically bug or is it fundamental in some way?
- taneq 3y agoThey’re fundamental to the scene if you recite light transport and not just geometry. Gaussian splats are just a clever way to sample a lightfield and reflections are part of the lightfield.
- jayd16 3y agoWell it's more that I'm asking if lighting data should live entirely in the spherical harmonics. It's certainly feasible and common in the game world. ...But maybe that doesn't make sense in the GS context.
- stjo 3y agoThis blog goes a lot more in depth in different techniques for compressing them (although for the purpose of using less VRAM): https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/ https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-s... https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/ https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-m... Including quantising spherical harmonics, grouping splats into chunks and compressing their position and rotation in groups, etc.
- jobigoud 3y agoI see various people online calling the primitive "splats" (ex: the scene has millions of splats, 248 bytes for each splat, etc). But the primitive are the 3D gaussians, "splat" just refers to how they are rendered. I wonder why people call them "splats", because it's catchier?
- treyd 3y agoThey're not rendered like simply converting a point to screen space or doing polygonal texture filling, "splatting" to the viewport is actually a pretty accurate description of how they're rendered.
- jobigoud 3y agoYes that's exactly what I'm saying. I'm wondering why people call the geometric primitive "splats" instead of "gaussians". I see people refer to the .ply file as containing millions of "splats". It's just strange to me. You could take these primitives (3D gaussians + other attributes like spherical harmonics coeffs) and render them differently, even if it would be slower. They only become "splats" in the renderer.
- folli 3y agoCan anyone point me to some good high level introduction to Gaussian splats? E.g. what's the benefit of using splats vs traditional polygons? is it somehow easier for the neural network to create these from the 2D photos? or what's the magic behind this?
- reactordev 3y agohttps://huggingface.co/blog/gaussian-splatting https://huggingface.co/blog/gaussian-splatting Think of this as a LiDAR visualization on steroids... A point cloud of information that can be used to render "The Blob".
- corysama 3y agoPolygons are great because you can make a solid, closed surface. But, they are a pain because you have to maintain the connectivity between the vertices. Adding or removing polygons in a mesh is serious work. Especially doing so on a GPU. Most LOD schemes do heavy precomputation at data compilation time and have the GPU just view a sliding window into static results. Splats however are loose points. It’s harder to get a solid appearance from them. But, you can add/remove/move each point independently. Splats don’t really use neural techniques. But, they are well set up for neural techniques to work with them. Generating the splats from photos uses some sort of linear algebra optimization/back-propagation. The Gaussian aspect is required to enable that to work because gaussians are differentiable. Hopefully that means GS should fit in well with recent advancements in differentiable rendering.
- afjeafaj848 3y agoThat rendering style looks really similar to Dreams on playstation Anyone know if that game uses the same technique?
- slimbuck 3y agoYes Dreams is actually a huge inspiration with the realtime rendering side of gaussian splatting. Specifically this presentation by Alex Evans https://www.mediamolecule.com/blog/article/siggraph_2015 https://www.mediamolecule.com/blog/article/siggraph_2015. Dreams doesn't use gaussian splats as such, but we still learn a lot about how to compress and render a huge number of particles efficiently. (We're not doing half of this on PlayCanvas... yet).
- bmikaili 3y agohttps://lightgaussian.github.io/ https://lightgaussian.github.io/ in similar vein
- dwallin 3y agoMore so than just compressing the data, I feel like the biggest gains for splats will come from a chunked LOD system, and streaming data into memory. Regardless of how efficient you make the representation, you are still going to run into fundamental limitations without this. On the low hanging fruit side of things, tools should really start integrating a Spherical Harmonics skybox into the training steps to better handle large scale distant details.