8 ms·
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- jjcm 3y agoI'd love to see a machine learning model trained on the resulting data of this. It'd be crazy to see if it can effectively learn and generate realistic looking video as an output.
- arijun 3y agoThis looks great! The main potential use for this must be for VR video with 6 degrees of freedom. What they have now does an incredible job of conveying space, but feels a bit limiting when your view doesn’t translate with you.
- Cieric 3y agoI've been slowly building my own rendering and training on a non cuda library (trying with vulkan/spirv) I'm curious how many cameras they used here though.
- mortenjorck 3y agoHard to believe the original Gaussian Splatting paper is still less than three months old, given the explosion of new techniques in recent weeks. It's wild to see the state of the art in environment and object capture suddenly advancing this quickly – beyond the obvious applications like real estate, I wonder what else GS will end up transforming.
- bane 3y agoTo be at risk of an "AcKchYuALly". Gaussian Splatting has been around since at least the early 90s. There's even a few old games made with the technique. The paper I think your referring to made the interesting leap that a 3d radiance field could be rerendered out as a field of Gaussian splats, and that this would probably run faster in modern GPU pipelines for real-time performance. It looks like they also have the nice property of being able to be shifted around in memory quickly hence the animation property seen here.
- krasin 3y agoInteresting! Can you please name some of these old games made with Gaussian Splatting? I would be interested to play, to get a sense why polygons won in that round (and likely to lose in this one).
- otteromkram 3y agoFrom another user: https://www.youtube.com/watch?v=dnOXk3QJWN8 https://www.youtube.com/watch?v=dnOXk3QJWN8
- chaboud 3y agoI used additive gaussian fields (restricted by bounding regions) for this back in the late 90's for audio visualizations in a ripper/player called "Siren" (back when we actually thought we could charge money for something like that). The technique worked well on non-accelerated (CPU only) hardware of the era, with the additive approach saving the pain of needing to keep a z buffer or fragment list. Gaussian voxel reconstruction is useful in medical and GIS settings, which, if memory serves, is where Kyle Freeman from Novalogic drew on for his work on Comanche. As far as I know, that was the first commercial game with voxel rendering... It's been a bit since I played it, but the swimming jaggies make me think that it was Manhattan distance height map offset by planar traversal (kinda like Doom raycasting) or some similar trick. I don't recall any intersections or overhangs, but, to be fair, I was a middle schooler when Comanche came out. It also ran fine on my weak sauce PC. Once acceleration hit, transformation of triangles with fixed-function pipelines took over. The ability to push textured triangles with minimal per-pixel value adjustment took over. Slowly but surely we've swing back to high ALU balance (albeit via massive stream parallelism). We've shifted from heavy list/vertex transformers to giant array multiply/add processors. It's a pretty great time to be a processing nerd.
- constantlm 3y agoCould you point us to some examples of old games using this technique? Would be awesome to see.
- 3y ago
- deleted 3y ago[deleted]
- atonalfreerider 3y agoThis one was released today as well. Works out of the box: https://github.com/JonathonLuiten/Dynamic3DGaussians https://github.com/JonathonLuiten/Dynamic3DGaussians
- jacobgorm 3y agoWe used to call this technique "vector balls" on the Amiga, here is one famous example: https://www.youtube.com/watch?v=gjKkUTlhIek https://www.youtube.com/watch?v=gjKkUTlhIek . I remember implementing it myself for an unreleased demo. I realize that a lot has happened since, but this is likely where it all started :)
- reactordev 3y agoThis gives me hope that one day we'll have a holodeck. Holy crap! The applications for this are pretty broad. From safety (scene reconstruction from video sources) to real-estate, to hollywood and video games. I'm just blown away. Will we eventually see 4D GS AR/XR scenes we can walk about? I feel like that would make the perfect VR sherlock holmes game.
- waynecochran 3y agoOne more step towards the next simulation level.
- jayd16 3y agoWhy wouldn't you be able to walk about now? They already have examples with free camera movement. To make it an XR scene, you just need to render two cameras and pipe it into a headset.
- croes 3y agoAren't the scenes static?
- syntaxing 3y agoDoes anyone have a video or post that explains the optimization part for the original paper? I understand most of it but that part and can’t seem to wrap my head around it.
- blovescoffee 3y agoWhat parts confuse you? There are a few steps in optimization. There are lots of papers on differentiable rendering, but the pruining of gaussians and the actual treatment of gaussians, I don't think there's a blog post.
- kroo 3y ago[dead]
- magicalhippo 3y agoJust glossed over the paper but it seems, in principle, simple enough (though rather brilliant IMHO). Essentially they're doing what you do when you train a neural network, only that instead of adjusting weights connecting "neurons", you adjust the shape and position of gaussians, and the coefficients of spherical harmonics for the colors. This requires the rendering step to be differentiable, so that you can back-propagate the error between the rendering and the ground-truth image. The next key step is to every N iterations adjust the number of gaussians. Either fill in details by cloning a gaussian in an area which is undercovered, or split a gaussian in an area which is overcovered. They use the gradient of the view-space position to determine if more detail is needed, ie those gaussians which the optimizer wants to move significantly over the screen seems to be in a region with not enough detail. They then use the covariance of the gaussians to determine to split or to clone. Gaussians with large variance get split, the others cloned. They also remove gaussians which are almost entirely transparent, no point in keeping those around. That's my understanding at least, after a first time gloss-through.
- webdood90 3y agoYou: > Essentially they're doing what you do when you train a neural network, only that instead of adjusting weights connecting "neurons", you adjust the shape and position of gaussians, and the coefficients of spherical harmonics for the colors. My brain: > They're providing inverse reactive current to generate unilateral phase detractors, automatically synchronizing cardinal gram meters.
- nialv7 3y agoHoly heck this is going to fundamentally change media production
- ge96 3y agoWondering when this technique will be used for meal calorie counters
- MattRix 3y agoNot seeing how that is related?
- MrTrvp 3y agoReminds me the Deja Vu movie and how they maneuver angles.
- VikingCoder 3y agoWell, Rule 34 is about to happen. And "splatting" is already a decent name...
- deleted 3y ago[deleted]
- crtified 3y agoHa! Or Rule 34a, "every sufficiently observed phenomenon, has just become somebody's new fetish". Although actually, and on a slightly more innocent (but just as edgy!) note, the thing that immediately popped into my head upon reading "4D Gaussian Splatting", was the music from the 1992 Future Crew demo Unreal, and the image of it's inter-scene title screens. ["IYKYK", but basically, that famous old PC demo consists of several short sections, each showcasing a particular coding/graphical technique - each section prefaced by a title screen which named the effect being showcased.] YT of Unreal demo, as citation for this highly-important observation : https://www.youtube.com/watch?v=InrGJ7C9B3s https://www.youtube.com/watch?v=InrGJ7C9B3s
- qm9 3y agoGerman demo group Farbrausch pioneered at this.
- heurist 3y agoAfter reconstruction, is there any way to scan for a particular condition in the model, and map it onto the 3D structure? For instance, find the broken cookie, or find a surface matching some input image.
- andyferris 3y agoI suspect typical point-cloud feature extraction techniques would work. Things like identify planar regions, from that join connecting planar regions into clusters, etc. The time component is super interesting here though!
- dwallin 3y agoSeem fairly tractable to use Segment Anything or a similar method to derive plausible semantic clusters of splats.
- jayd16 3y agoDoes anyone know if the pixel overdraw of the GS scene is consistent from every view angle? I'm asking because I would assume there is inconsistent GS density but the paper doesn't give a range of FPS measurements or 99th percentile or anything like that.
- kookamamie 3y agoI'm pretty certain it is not - consider surfaces seen in steep angles, vs. ones seen perpendicularly. If we assume no culling or pruning occurs for the splats, steep angles result in way more overdraw.
- jayd16 3y agoThis doesn't necessarily follow. If the splats are coplanar to the surface they are representing then viewing them at an angle wouldn't change how they overlap. But that said, I assume you're right.
- vavooom 3y agoThis is just incredible technology
- KaoruAoiShiho 3y agoFeel like this changes everything, trying it out right now...
- raytopia 3y agoWith tech like this I'm starting to wonder if realistic games are going to become normalized and what will happen as a result. Also has anyone been working on solving the "blurry" look these splats have up close?
- andyferris 3y agoEach Gaussian "splat" is literally a little blurry blob. The way to make it sharper is to increase the resolution - i.e. increase the number of splats, decrease the size of each one. This increases both training time and render time though.
- andyferris 3y ago> realistic games That said, games don't have to be super realistic to be fun. E.g. I could imagine a game based on GS at "Minecraft resolution".
- TheRoque 3y agoBut if I'm not mistaken, this technique still requires to get a ton of pictures from many angles ? It's fine for visiting an apartment or watching a cooking video in 3D, but how possibly can you apply this to a videogame that has much more degrees of freedom ? Are you gonna scan an entire city with a drone to create a GTA-like ?
- Nevermark 3y ago> We introduce the 4D Gaussian Splatting (4D-GS) to achieve real-time dynamic scene rendering while also enjoying high training and storage efficiency. This seems to be a rendering efficiency innovation, not particular to scanning. That means it applies to artificially generated environments, whether photo realistic or stylized, and whether based on a real environment or a completely fictional one. But of course, any photorealistic, extremely faithful to the smallest detail, rendering of a real place is going to involve a lot of scanning. That is true for any kind of rendering.
- teaearlgraycold 3y agoCombine this with state of the art VR tech (something with good eye tracking and 4k per eye) and we're living in the future.
- xrguy 3y agoi like how galaxies look like ellipsoids if you zoom out
- petargyurov 3y agoAnyone know how well this technique deals with mesh/lattice type structures? For example, fences, ladders, climbing frames, etc.
- andybak 3y agoGaussian Splatting in general or this specific approach to animation. Can't comment on the latter but fine detail renders very nicely on still scenes. https://lumalabs.ai/capture/ed9d985b-9cc1-49e0-a39c-88afa2038fcd https://lumalabs.ai/capture/ed9d985b-9cc1-49e0-a39c-88afa203... https://lumalabs.ai/capture/83e9aae8-7023-448e-83a6-53ccb377ec86 https://lumalabs.ai/capture/83e9aae8-7023-448e-83a6-53ccb377... https://lumalabs.ai/capture/7f8df9c9-c548-4a47-9892-e945637c7558 https://lumalabs.ai/capture/7f8df9c9-c548-4a47-9892-e945637c... https://lumalabs.ai/capture/076fcfdc-ea80-4fdc-8159-c9fed831dc61 https://lumalabs.ai/capture/076fcfdc-ea80-4fdc-8159-c9fed831...
- petargyurov 3y agoWow, that's impressive. Thank you for these.
- lnyan 3y agoThis is bad news for me. I am working on a simliar project (gaussian splatting + dynamic scene). Our method is different with the mentioned 4D gaussian splatting, but I am unsure shall I continue or not.
- andybak 3y agoWhy is that bad news? If you're interested in the outcome - someone has saved you work. If you're interested in the process - or exploring your specific approach then why stop?
- ShamelessC 3y agoBecause of the sunk cost fallacy which is a fallacy only if you ignore the emotions and trauma of having to abandon work you identify with or is responsible for your self esteem.
- SiempreViernes 3y agoDon't see how the sunk cost has anything to do with it, OP clearly hoped to be first, as being first would almost guarantee a payoff of some form. Coming out with a method second is much less likely to be rewarded by the community: not being rewarded for work completed is not sunk cost, its just a straight up loss.
- wildpeaks 3y agoPlease continue working on it: being first doesn't imply being the best, all research is iterative.
- sheepscreek 3y agoIf it’s a different method, it is definitely worth sharing. The additional interest might actually be helpful.
- JHonaker 3y agoThere's nothing wrong with concurrently developing something similar. I can almost guarantee there will be something different enough about what you've developed to be considered novel (if you care about publication). If you don't care about publication, then definitely keep going! Hell people are still argue about which implementation of complete specifications are the best (for good reason).
- cubefox 3y agoInteresting that the original publication this is based on (that won the SIGGRAPH 2023 best paper award) didn't get a lot of attention on HN at the time: https://news.ycombinator.com/item?id=36285374 https://news.ycombinator.com/item?id=36285374
- shultays 3y agoCan someone help me understand what this is actually doing?
- sheepscreek 3y agoCan someone also explain the implications of this on gaming?
- willis936 3y agoYoutube recommended this video to me, which concisely explains splatting. Information necessary for trade is there but the trade is left to the viewer. The key drawback that isn't highlighted is that you need a physical space to be a close approximation of what you want to render. So if you want to make a few counter strike maps based off of your workplace (not recommended) then this would be a good technology, but if you want to make an open world on an alien planet you're likely better off with traditional rendering. https://youtu.be/HVv_IQKlafQ https://youtu.be/HVv_IQKlafQ
- omneity 3y agoCan someone explain to me how is it possible using gaussians to have different reflections based on the angle of view like on the demos? I'm finding it hard to grasp.
- deleted 3y ago[deleted]
- jacobgorm 3y agoI believe that is due to the use of Spherical Harmonics.
- Aardwolf 3y agoThat seems more complex to store and render than everything else about a gaussian splat, how are these used efficiently?
- jacobgorm 3y agoThere is a bit of explanation here https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pretty-cool/ https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr... (found via Google).
- astlouis44 3y agoMind blowing stuff.
- fennecfoxy 3y agoGreat video I saw a while ago on this: https://www.youtube.com/watch?v=HVv_IQKlafQ https://www.youtube.com/watch?v=HVv_IQKlafQ (albeit for 3d, not 4d). His editing is hilarious too.