4 ms·
The moment I saw the use of motion vectors with gaussian splats I started wondering if it was a viable way to achieve high performance, high quality spatial vid
by scheeseman486 3y ago
The moment I saw the use of motion vectors with gaussian splats I started wondering if it was a viable way to achieve high performance, high quality spatial video. Instead of capturing from all angles like most current approaches (which seems more geared to create content that is used in place of a 3D model) why not capture from a fixed perspective, using an array of cameras covering about 1m square to allow for slop in head position, providing parallax and perspective correct rendering. I'd presume that'd make it even more compressible too, since you could get rid of any data that isn't visible from outside that frustum.
It would be a true hologram, completely supplanting 180/360 stereo video. Imagine Avatar on such a format. Or, lets be real, porn.
- cousin_it 3y agoI wonder what style of non-realistic graphics would be best suited to gaussian splats. Like how pixel art and low-poly came from limitations of yesterday's technologies, but became distinctive styles in their own right.
- deltaburnt 3y agoSounds kind of similar to https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
- scheeseman486 3y agoThat was definitely on my mind, that uses lightfields though. The file sizes are ludicrous.
- deltaburnt 3y agoYep, though the layered mesh post processing is nice because most consumer hardware at this point is pretty good at rendering plain ol meshes. The big challenge for mass adoption of nerfs/gsplats techniques will be getting them running and viewable on a phone web browser.
- jayd16 3y agoSo basically what it is now but with view and occlusion culling?
- scheeseman486 3y agoMore compression with considerations made to allow for streaming and some kind of equivalent to I/P/B-frames, spatial audio, metadata (chapters, aspect ratios etc) in a standardized file container. At the moment it's all rather basic and bespoke.
- krenzo 3y agoHave you seen this? https://shenhanqian.github.io/gaussian-avatars https://shenhanqian.github.io/gaussian-avatars