4 ms·

Looks impressive in the video on your website. Is the video passthrough offset to the right/left due to the camera location or are you able to correct that in

Looks impressive in the video on your website. Is the video passthrough offset to the right/left due to the camera location or are you able to correct that in app (asking due to the open peripheral vision).

macguillicuddy 3y ago
Thank you! We do correct for this in the app - it's quite a complicated process because we only have a single 'monocular' camera image which we have to compose with 3D content. We use some tricks to render the camera in a way that feels comfortable with the content. We consider the camera passthrough to compete very favourably with Quest 3 / Pro and have also received independent feedback to say so, primarily because it's high resolution and high framerate (60 fps while we believe the Quest passthrough is only 30).
- locallyidle36 3y ago
  To clarify, are users able to perceive depth from the video passthrough? I'd imagine you'd need to adapt that monocular video into a stereoscopic 3d scene using lidar/etc. Or is it more of other tricks that make the 2D passthrough comfortable when viewed with the 3D AR content?
  - tb100 3y ago
    Hi - I'm Simon, and I've been leading the development on the low-level runtime bits for Zapbox. We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon"). Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience. The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement. The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do. What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :) [1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.

macguillicuddy 3y ago

Thank you! We do correct for this in the app - it's quite a complicated process because we only have a single 'monocular' camera image which we have to compose with 3D content. We use some tricks to render the camera in a way that feels comfortable with the content. We consider the camera passthrough to compete very favourably with Quest 3 / Pro and have also received independent feedback to say so, primarily because it's high resolution and high framerate (60 fps while we believe the Quest passthrough is only 30).

locallyidle36 3y ago
To clarify, are users able to perceive depth from the video passthrough? I'd imagine you'd need to adapt that monocular video into a stereoscopic 3d scene using lidar/etc. Or is it more of other tricks that make the 2D passthrough comfortable when viewed with the 3D AR content?
- tb100 3y ago
  Hi - I'm Simon, and I've been leading the development on the low-level runtime bits for Zapbox. We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon"). Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience. The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement. The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do. What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :) [1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.