7 ms·
3D Gaussian Splatting as Markov Chain Monte Carlo
- corysama 2y agoThis is nice because the original 3DGS technique initialized itself with a point cloud generated using a traditional COLMAP process https://en.wikipedia.org/wiki/Structure_from_motion https://en.wikipedia.org/wiki/Structure_from_motion
- Maken 2y agoThe original 3DGS paper relied on some really hand-wavy heuristics to optimice the gaussians' location and size (and yet it worked remarkably well).
- Etheryte 2y agoThe value of the parameter was chosen by I picked it.
- deleted 2y ago[deleted]
- Terr_ 2y agoHey guys, I think I found the NSA crypto-saboteur! :p
- eigenman 2y agoAs someone who works in numerical optimization, this is a dirty little secret of our profession. The optimization algorithms in the literature are great at finding local minima, but often are very sensitive to the initialization as to how small the objective is. Good heuristics for initialization are thus critical for finding a good (small objective) minimizer. Sometimes this gets to the point where the local optimization algorithm does a trivial refinement of the heuristic’s solution.
- jonas21 2y agoThe point cloud is a by-product of of aligning the input images. You essentially get it for free.
- sorenjan 2y agoSo do this paper, although they get basically the same result from random initialization. But as mentioned, the point cloud is free since they need the camera poses anyway. > We initialize our samples either randomly or from point clouds, typically from Structure-from-Motion (SfM) as in 3DGS
- krasin 2y ago> But as mentioned, the point cloud is free since they need the camera poses anyway. If you have a rigid multi-camera rig, the camera poses might be known from calibration, but then the particular scene shot on such a rig, could be reconstructed without COLMAP or other structure-from-motion tools, if I understand it correctly.
- programjames 2y agoSo, just to make it clear, the main difference in this paper is adding a small amount of noise to each update? I'm a little frustrated that I read through the whole paper and still am not sure about this.
- deleted 2y ago[deleted]
- mafuyu 2y agoI'm not an expert at all, but my understanding is that Gaussian splatting is essentially a rendering technique. Normally, you'd take actual data, like a set of photos, and optimize some Gaussians against it to arrive at a volumetric representation. In the case of AI-generated splats, it's kinda flipped. Instead of optimizing against a known ground truth, the AI is generating Gaussians to be rendered. The insight of this paper is that we already have great statistical tools for numerically estimating ground truth based on a bunch of Gaussians, so why not just apply those?
- dwallin 2y agoNo the main difference is looking at the problem from a new perspective, relating it to a large body of existing work in statistics (see https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo and https://en.wikipedia.org/wiki/Stochastic_gradient_Langevin_dynamics https://en.wikipedia.org/wiki/Stochastic_gradient_Langevin_d...). Then they are able to use this new perspective to add several improvements that lead to significant quality improvements, clearly establishing the validity and utility of the new theoretical underpinnings. It's likely this will be massively influential in the direction of future research in the space. The actual changes this led to are: 1) As you mentioned, they added noise. But notably, they state it was "designed carefully" to conform to the requirements of SGLD and they detail how they designed the noise. 2) They simplified the original operations of "move, split, clone, prune, and add" and their related heuristics, into a single operation type. They do so guided by existing knowledge about MCMC frameworks, leading to a simpler model with stronger theoretical underpinnings (huge win!). 3) Adjustments to how gaussians are added and pruned to better fit with the new model. This seems more like housekeeping rather than something novel in and of itself.
- dguest 2y agoI'm sad that the PDF doesn't use hyperref. When you get used to clicking a few links to to the a cited reference the whole scroll-to-reference, copy-paste to google, scroll to paper song and dance seems a lot less fun. Probably a cool paper though.
- hapofuino 2y agoThe paper is also on arXiv, which includes the TeX source. The experimental HTML view for this paper has automatic (internal) hyperlinks like hyperref: https://arxiv.org/abs/2404.09591 https://arxiv.org/abs/2404.09591 Edit: The TeX source has the `draft` option for hyperref that disabled hyperlinks in the produced pdf. External links to references aren't recoverable from the included main.bbl (probably because it was built with the `draft` option).
- gessha 2y agoYeah, scientific papers are a pain to read. Some have a Latex extension which links you to the paper down in the references but it's not bidirectional - you have to scroll back to your previous position.
- flor1s 2y agoSome PDF readers have a Back button (like the built-in Zotero PDF reader).
- TheRealPomax 2y ago"built-in" to what?
- flor1s 2y agoI mean the PDF reader that is contained in Zotero. Am I using the wrong terminology?
- kombine 2y agoTo add to another comment, Okular PDF reader has a back button, which is super convenient to come back to the main text after clicking on a reference.
- StrLght 2y agoGaussian splatting is such an impressive technique. I wish it finds real-world applications useful for average person -- right now it's probably the best way to show photorealistic scenes in VR. There ought to be more usages out there.
- hapofuino 2y agoFor an average person, I suspect that some real estate listings will adopt Gaussian splatting soon. Many newer listings include photogrammetry already, and splatting will provide a significant improvement to existing solutions.
- fallat 2y agoI'm just starting to use photogrammy in hobbyist game maps to provide "chill maps". Unfortunately splatting will not be possible there.
- dwallin 2y agoThere's a lot of use cases, and companies already using photogrammetry techniques have adopted it extremely quickly, but for many of these businesses the usage of gaussian splats is just a technical implementation detail, resulting in quality / feature gains, but doesn't as of now unlock entirely new business models.
- jesol 2y agoIt's true, Gaussian Splatting is just an alternative to meshing a pointcloud for companies which currently rely on photogrammetry or lidar (Lidar works well as a basis for splatting when there's reference images taken as part of the scan). But I think that misses all the new opportunities that exist with Gaussian Splatting, which really just don't with existing techniques. Gaussian Splats are able to handle more heterogeneous information sources, allowing more sources to help splat an environment. Devices like drones, surveillance cameras, or autonomous systems can be used to create or incrementally update a Gaussian Splat; and there's interesting work to allow them to locate themselves within the splat, not just to show themselves but also to place vision ML outputs into it (such as object detection or segmentation results). Up till now nearly all digital representations of physical environments are either based off the original designs (by things like CAD or BIM files), or are an approximation of the environment (from photogrammetry or Lidar scans). CAD and BIM files suffer from drift, the real environment almost never perfectly matches the design files, small (and large) changes are made; and many times those files aren't even available if the structure isn't new. Photogrammetry and Lidar scans struggle because their output is a pointcloud, and it's very difficult to accurately mesh a pointcloud (Matterport only partially solved this problem and sold for $1.6B). Gaussian Splats overcome these issues; they're comparatively easy to generate for any environment, and allow for very accurate and easy viewing from any angle. I think the Digital Twin space will be turned upside down, and they could potentially even cause huge changes in autonomous and semi-autonomous factories, warehouses, and depots. A single Gaussian Splat could be the source of truth that many autonomous vehicles update through their separate SLAM systems. Operators then would have access to this splat (and it's history) as a source of truth for the environment. Then, using techniques like iComMa[1], it may be possible to directly align XR devices into the Gaussian Splat; allowing operators direct access to location-based information generated by the environment. That's a lot of words to say: Gaussian Splatting is a very neat new technology that could really underpin many future technologies, I'm really excited about it [1]: https://yuansun-xjtu.github.io/iComMa.io/ https://yuansun-xjtu.github.io/iComMa.io/
- vessenes 2y agoOK, I read the paper, agree the results look good, like the idea of better formal grounding for how to choose where your splats are, and ... I still have no idea what that top image is of. Is it the distribution of where they put initial splats for any given 2D image?? Why does the caption mention buildings? I'm really lost.
- Vt71fcAqt7 2y ago>Unlike existing approaches to 3D Gaussian Splatting, we propose to interpret the training process of placing and optimizing Gaussians as a sampling process. Rather than defining a loss function and simply taking steps towards a local minimum, we define a distribution G which assigns high probability to collections of Gaussians which faithfully reconstruct the training images. What is the practical difference here? MCMC itself samples more from higher probilities than lower ones (ie. towards a local minimum). Is it just that we sample more from lower ends of the distribution? Or is it more about formalizing the previous algorithm so that it is easier to play with the different parameters? (eg. the acceptance threshold)
- TheRealPomax 2y agoBeing able to now apply many tools from the field of statistics to the problem, rather than only being able to use gradient related tools.
- ziofill 2y agoWhat are the “consumer” applications of 3D splatting? It looks super cool, and I can see applications for the next gen of maps, but I don’t understand if it’s going to be an end user technology or not.
- chefandy 2y agoProbably about as consumer-related as any other part of a 3d rendering pipeline. Most 3D rendered images you see-- VFX, motion graphics, animation-- were rendered and composited into video clips. Some things like video games render 3D in real time.
- jtolmar 2y agoIt might end up being used in video games, either because the performance continues pushing ahead, or because someone just wants its distinctive look. Speaking of uses for gaussian splatting - would anyone have any unique use cases for being able to generate splat models wildly faster?
- dwallin 2y agoYeah, there are some applications in the area of SLAM that benefit from quickly generating plausible gaussian representations.
- cubefox 2y agoIt's very unlikely it will be used in video games. Video games pretty much exclusively use texture mapped polygon meshes as data structure. They easy to render, they plays well with object animations, they allow for dynamic lights, and there exists a lot of mature software to create and edit this data. 3D Gaussian splats also render quite fast, but they are bad for animations, they don't allow for dynamic lights (as far as I can tell), and there are basically no tools to create and edit models, except from photogrammetry.
- kragen 2y agodo you mean something that end users are aware of using? probably not. but probably everything end-users do that involves digital models of three-dimensional objects will use gaussian splatting fairly soon: cad, cam, photography, maps, video games, surgery, georectification of satellite images, diagnostic radiography, ultrasound, radar, sonar, animation, missile guidance, other robotics applications, fashion design, 3-d printing, etc.
- falloon 2y agoAnother paper building off the Inria (non commercial license) method, there are several open source alternatives out there.
- anon115 2y agocan't comprehend the text :(