5 ms·

For truly immersive VR video it seems we’ll need to support moving your head in 6 degrees of freedom, so you can peak around objects etc. We’re a long ways off

by rlt 2y ago

For truly immersive VR video it seems we’ll need to support moving your head in 6 degrees of freedom, so you can peak around objects etc. We’re a long ways off from it being realtime, but I can imagine generative AI being used to interpolate frames from an array of cameras spaced fairly widely apart.

jayd16 2y ago
To some degree it's already being done on the headset in realtime to produce a passthrough image with proper depth
- j5155 2y ago
  Not really, that image is stereo but it isn’t 360 degree and it can’t support arbitrary viewing angles and positions after the fact. It works perfectly for passthrough because the headset cameras are physically locked to your head.
  - jayd16 2y ago
    It does support arbitrary viewing angles in the sense that IPD is accounted for. It generates a view for your specific eye position. It handles in painting around the user's hands as well. Of course it's not magic so the views it can generate are limited.
jdietrich 2y ago
Lytro had this more-or-less figured out, before they ran out of money. https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0-light-field-camera-improved-quality-faster-captures/ https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0...
choppaface 2y ago
In the Vision Pro demo appointment reel (i.e. anybody can demo the headset for free), everything is 6DOF, most is just stereo. Immersive VR is interactivity, which is more about good UX / gamification and less about using a dual-8k stereo capture device.
acka 2y ago
There's a cool VR demo experience showcasing this on Steam: Welcome to Light Fields[0]. [0] https://store.steampowered.com/app/771310/Welcome_to_Light_Fields/ https://store.steampowered.com/app/771310/Welcome_to_Light_F...
- PetitPrince 2y ago
  Yes, this is the closest to what I would consider a true VR photo. Stereoscopic images are gimmicky and feels utterly restrictive compared to this. From the same lab 2 years later, there's "Immersive Light Field Video with a Layered Mesh Representation" which is the same, but instead of photo it's video.
  - acka 2y ago
    Interesting! Thanks for the pointer.
xnx 2y ago
Google had a complete system for capturing volumetric light fields in 2020: https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/
lhl 2y ago
There was a lot of work on big light-field capturing rigs in the past, but honestly, with the current advancements in neural radiance fields, gaussian splatting, meshing and other scene reconstruction techniques even monocular capture can generate pretty amazing results. I'm sure that having ultra-high res binocular video could be converted into some pretty high quality free-look output (that can at least be played back in real-time) on high-powered mobile devices could be possible today (w/ just some additional optimization/engineering work). Recent research examples: * https://vidu4d-dgs.github.io/ https://vidu4d-dgs.github.io/ * https://fudan-zvg.github.io/4d-gaussian-splatting/ https://fudan-zvg.github.io/4d-gaussian-splatting/ * https://guanjunwu.github.io/4dgs/ https://guanjunwu.github.io/4dgs/ * https://aoliao12138.github.io/VideoRF/ https://aoliao12138.github.io/VideoRF/
spyder 2y ago
4D Gaussian splatting will probably get us there, there are many research papers about it for example: https://fudan-zvg.github.io/4d-gaussian-splatting/ https://fudan-zvg.github.io/4d-gaussian-splatting/ But even with that, it would allow only limited movement because if you move to much in the reconstructed space then it would have to show you occluded areas that has been never recorded. But for that case generative AI could be used to fill in the the gaps, which means it wouldn't be true to life reconstruction anymore, but it could be still useful where that isn't that important.

by rlt 2y ago

jayd16 2y ago

To some degree it's already being done on the headset in realtime to produce a passthrough image with proper depth

j5155 2y ago
Not really, that image is stereo but it isn’t 360 degree and it can’t support arbitrary viewing angles and positions after the fact. It works perfectly for passthrough because the headset cameras are physically locked to your head.
- jayd16 2y ago
  It does support arbitrary viewing angles in the sense that IPD is accounted for. It generates a view for your specific eye position. It handles in painting around the user's hands as well. Of course it's not magic so the views it can generate are limited.

jdietrich 2y ago

Lytro had this more-or-less figured out, before they ran out of money. https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0-light-field-camera-improved-quality-faster-captures/ https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0...

choppaface 2y ago

In the Vision Pro demo appointment reel (i.e. anybody can demo the headset for free), everything is 6DOF, most is just stereo. Immersive VR is interactivity, which is more about good UX / gamification and less about using a dual-8k stereo capture device.

acka 2y ago

There's a cool VR demo experience showcasing this on Steam: Welcome to Light Fields[0]. [0] https://store.steampowered.com/app/771310/Welcome_to_Light_Fields/ https://store.steampowered.com/app/771310/Welcome_to_Light_F...

PetitPrince 2y ago
Yes, this is the closest to what I would consider a true VR photo. Stereoscopic images are gimmicky and feels utterly restrictive compared to this. From the same lab 2 years later, there's "Immersive Light Field Video with a Layered Mesh Representation" which is the same, but instead of photo it's video.
- acka 2y ago
  Interesting! Thanks for the pointer.

xnx 2y ago

Google had a complete system for capturing volumetric light fields in 2020: https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/

lhl 2y ago

There was a lot of work on big light-field capturing rigs in the past, but honestly, with the current advancements in neural radiance fields, gaussian splatting, meshing and other scene reconstruction techniques even monocular capture can generate pretty amazing results. I'm sure that having ultra-high res binocular video could be converted into some pretty high quality free-look output (that can at least be played back in real-time) on high-powered mobile devices could be possible today (w/ just some additional optimization/engineering work). Recent research examples: * https://vidu4d-dgs.github.io/ https://vidu4d-dgs.github.io/ * https://fudan-zvg.github.io/4d-gaussian-splatting/ https://fudan-zvg.github.io/4d-gaussian-splatting/ * https://guanjunwu.github.io/4dgs/ https://guanjunwu.github.io/4dgs/ * https://aoliao12138.github.io/VideoRF/ https://aoliao12138.github.io/VideoRF/

spyder 2y ago

4D Gaussian splatting will probably get us there, there are many research papers about it for example: https://fudan-zvg.github.io/4d-gaussian-splatting/ https://fudan-zvg.github.io/4d-gaussian-splatting/ But even with that, it would allow only limited movement because if you move to much in the reconstructed space then it would have to show you occluded areas that has been never recorded. But for that case generative AI could be used to fill in the the gaps, which means it wouldn't be true to life reconstruction anymore, but it could be still useful where that isn't that important.