5 ms·
Show HN: Real-Time 3D Gaussian Splatting in WebGL
- jansan 3y agoSo far I have only seen gaussian splatting used on photographic data. Would it make sens to use it for other graphics data, too. Or in other words, does it have potential to be used in games?
- Solvency 3y agoSure, why not? It's just a fancy point cloud. I can easily imagine an open world Minecraft-esque game that uses this for its base engine instead of voxels.
- Lichtso 3y agoDepends, radiance field approaches (like gaussian splatting) are basically 3D photos. They do only capture color at geometry (position and direction), but have no concept of surfaces, materials and light transport in general (emission, absorption, transmission, reflection, scattering, etc.). In other words, they can only do static scenes (no animations) with pre-baked lighting. The industry seems to be trying to move away from this with things like PBR (physical based rendering) and ray / path tracing which enables far better dynamic lighting. Also, they are extremely space inefficient at the moment. A scene that would take a good traditional rendering engine a few dozen GB would take TB instead. Though, that might improve in the future with more optimization. One exception to the above, where gaussian splatting might be interesting to see is procedural / generated content (possibly even animated). Especially for volumetric effects which currently use particle systems, like smoke, fire, clouds, flowing water, etc.
- poslathian 3y agoI thought I understood that the speculator highlights and view dependent color problems you mention are massively improved via adding spherical harmonics into each ellipse?
- andrewstuart 3y agoWhat am I looking at?
- bestest 3y agobasically this: https://github.com/graphdeco-inria/gaussian-splatting https://github.com/graphdeco-inria/gaussian-splatting — a somewhat different approach at rendering 3d scenes.
- underlipton 3y agoA thought occurred to me: is this similar to how Media Molecule's Dreams on the PS4/5 renders its scenes?
- FinalDestiny 3y ago> Media Molecule “Dreams” has a splat-based renderer (I think the shipped version is not purely splat-based but a combination of several techniques). From: https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pretty-cool/ https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr... Good eye
- bluescrn 3y agoInteresting video here (from 2015) on the development of the Dreams rendering tech: https://www.youtube.com/watch?v=u9KNtnCZDMI https://www.youtube.com/watch?v=u9KNtnCZDMI
- pixelpoet 3y agoAlso maybe worth mentioning that the 2nd author of the InstantNGP paper is also a cofounder of and lead tech guy at Media Molecule, Alex Evans. I've been a huge fan of his work since 90s demoscene and briefly got to work with him at Lionhead Studios, guy's a legit genius.
- KaiserPro 3y agoGaussian splatting is a fancy word for pointcloud but with coloured shapes instead of points. Its been around for ages, but It was never used because if you have a million points in a point cloud, you'd need to artistically manipulate a million points. Its like 3d hair, its pretty simple, just render a billion hairs, but in practice its hard to make it look good. Here we tell a machine learning model to adjust the angle, colour, shape and size of a million primitives (ie a square, circle, triangle etc.) so that it looks like a the photos we provide.
- crtasm 3y agoClick through to the github for a list of the controls (I didn't think to try spacebar!) and links to other example scenes.
- bluescrn 3y agoWhen you zoom out there's lots of visible polygon edges that don't look like they should really be there, as if it's trying to draw soft 'blobs' but the texture coords aren't quite right? Is that a bug or an intentional part of the technique?
- KaiserPro 3y agoIntentional. Basically its a semidense point cloud [1], but instead of a point, there is a blob which has been coloured, angled and scaled to match the input picture. This means they are optimised to be viewed from a certain distance. Think of it like a 3d vector drawing, if you zoom in too much, or pull one part away, it all starts to look a bit funky. [1]https://www.researchgate.net/publication/326621750/figure/fig3/AS:652485718986753@1532576126869/Semi-dense-point-cloud-generated-by-17-overlaid-with-the-loop-closure-trajectory-of-the.png https://www.researchgate.net/publication/326621750/figure/fi...
- lwansbrough 3y agoDoes this use the method proposed by Kerbl and Kopanas at SIGGRAPH 2023? https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
- porphyra 3y agoYes, but this is just the splatting/rendering part and not the optimization part that generates the reconstruction in the first place.
- Lichtso 3y agoReally cool, I am also working on a port of gaussian-splatting [0] but to WebGPU. Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. That is because perspective projections have three additional effects: - Parallax movements (that is the view plane moves parallel to the ellipsoids) change the shape of the projected ellipse. E.g. a sphere only appears circular when in center of the view, once it moves to the edges it becomes stretched into an ellipse. This effect is manually counter balanced by this matrix I believe [2]. - Rotating an ellipse can change the position it appears at, or in other words creates additional translation. This effect is zero if the ellipse has one of its three axes pointing straight at the view (parallel to the normal of the view plane). But, if it is rotated 45°, then the tip of the ellipse that is closer to the view plane becomes larger through the perspective while the other end becomes smaller. Put together, this slightly shifts the center of the appearance away from the projected center of the ellipsoid. - Conic sections can not only result in ellipses but also parabola and hyperbola. This however is an edge case that only happens when the ellipsoid intersects with the view plane and can probably be ignored as one would clip away such ellipsoids anyway. The last two effects are not accounted for in these calculations in any of the implementations I have seen so far. What would be correct to do instead? Do not calculate the 3D covariance. Instead calculate the bounding cone around the ellipsoid which has its vertex at the camera position (perspective origin). Then intersect that with the view plane and the resulting conic section is guaranteed to be the correct contour of the perspective projection of the ellipsoid. [0]: https://github.com/graphdeco-inria/gaussian-splatting https://github.com/graphdeco-inria/gaussian-splatting [1]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2360800da2e572526632ea35/main.js#L558C1-L587 https://github.com/antimatter15/splat/blob/3695c57e8828fedc2... [2]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2360800da2e572526632ea35/main.js#L578-L582 https://github.com/antimatter15/splat/blob/3695c57e8828fedc2...
- porphyra 3y agoIn general, a Gaussian is no longer a true Gaussian after camera projection since the pinhole camera projection function is nonlinear (due to dividing by z). However, if the Gaussian is small relative to the size of the image, you can apporximate it by linearizing the projection function. Therefore the Gaussian splatting paper uses the Jacobian of the projection function as described in equation 5 of the paper [0]. In practice, this approximation is extremely good. This Jacobian is the matrix you mentioned in the third link and it is mathematically sound and not "manually counter balanced". For a derivation, see [1]. [0] https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_gaussian_splatting_low.pdf https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_... [1] https://math.stackexchange.com/a/4716514/43771 https://math.stackexchange.com/a/4716514/43771
- gabereiser 3y agoThis is beyond cool. Point clouds are one thing but this… this is amazing. Kudos and great job. It even runs on my work Lenovo at 60fps.
- esperent 3y agoIt runs on my mid range phone at 36fps. Did not expect that. Lots of artefacts though, especially if I move the camera.
- gabereiser 3y agothose artifacts are part of the algorithm.
- naavis 3y agoThis is really cool! The control scheme is confusing though. Instead of the typical WASD for moving and using the mouse to look around, dragging the mouse moves forwards and backwards and orbits around some point, A and D strafe, while W and S look up and down. EDIT: Looks like a full list of controls is in the readme: https://github.com/antimatter15/splat#controls https://github.com/antimatter15/splat#controls
- alanbernstein 3y agoIt's very similar to the N64 FPS controls (i.e. Goldeneye): arrow keys (joystick) for the "primary movements" of forward/backward and yaw, with which you can move and look anywhere in a 2D space. Then, WASD (C buttons) for the "secondary movements" of strafe and pitch.
- gorkish 3y agoIt's pretty telling that you had to reach back 26 years to find a control scheme that could be used as an analogy. I don't even know where to start with the mouse controls. Up/down translation lock after right click; reverse yaw. Thing is a test of patience!
- moffkalast 3y agoIt actually seems like the FreeCAD control scheme almost verbatim, I always hated that thing and its insistence to not provide any way to orbit around the up vector. Like, are there people whose head does not rotate around their neck on axis but ends up sideways and rolled when they turn or something, to whom this makes perfect sense? I can't see any other explanation.
- gorkish 3y agoBeing brutally honest here, but I just cant get over the control scheme enough to even appreciate the rendering demo. It is unusably unintuitive and awful.
- antimatter15 3y ago
- adfm 3y agoVery impressive! Curious what the frame rate would be like for stereoscopic rendering of the same scene on the same hardware. Are there optimizations to be had past the halfway mark?
- Yenrabbit 3y agoDefinitely. One of the time-consuming parts of rendering is sorting the gaussians by distance to the camera, which for two nearby cameras could be optimized. This also goes for adjacent frames - assuming smooth motion im pretty sure there is some speedup to be had by assuming the previous sort will be close to correct rather than starting from scratch each frame.
- smusamashah 3y agoIs it possible to increase the number of points (resolution) with some setting? I want to see more refined view on a higher end machine.
- msk-lywenn 3y agoRuns fine on my 2016 iPhone SE. kudos
- agys 3y agoLast sentence of the readme…!
- gsuuon 3y agoWould this technique work for video? The readme of the inria work[1] seems to imply a model is trained per static scene, does that rule out video? [1] https://github.com/graphdeco-inria/gaussian-splatting https://github.com/graphdeco-inria/gaussian-splatting
- stefanbaumann 3y agoIt's already a thing [1]. They also have a project website [2] with some nice videos, although the code hasn't yet been released. [1] https://arxiv.org/abs/2308.09713 https://arxiv.org/abs/2308.09713 [2] https://dynamic3dgaussians.github.io/ https://dynamic3dgaussians.github.io/
- evrimoztamur 3y agoThe recording from the point of view of the football they were tossing at each other made me feel things. My friend mentioned that it's like the 'braindance' from Cyberpunk 2077.
- gsuuon 3y agoWow - it didn't occur to me but it feels exactly like a braindance.
- teucris 3y agoWow. I was literally just working on my own implementation. You beat me to it! Great work!
- tmilard 3y agoGaussian Spatting is the new sensation of the sumer in the 3D Scanning Field. Will it live to its expectation ?
- matt3210 3y agoFancy! I like that on mobile I can drag to move around!
- crubier 3y agoWow this is insanely cool. If you make it work within ThreeJS, you're going to leave a trace in the history of 3D on the web with that stuff!
- q_andrew 3y agoCan't wait to pull this up on my desktop tomorrow.
- cchance 3y agoGotta try on a pc for some reason on iOS the cloudiness feels more like nerf than Gaussian to me for some reason, gotta try it on pc later
- jheriko 3y agowhy the hell is it that anyone who makes these "clever" demos provides the world's shittiest camera that adds unwelcome rolls. late 90s bedroom me is shaking his head.
- klysm 3y agoI've never experienced this set of mouse controls for a 3D view ever before and was highly confused for a bit.