5 ms·
As someone close to this in the industry, I definitely agree. I also really don't know why he decided to deemphasize the perspective and depth correctness so m
by Zee2 4y ago
As someone close to this in the industry, I definitely agree.
I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here:
> In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory.
I don't think this is even close to capturing how important depth and perspective correct passthrough is.
Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time.
Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background.
The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.
- nomel 4y agoI’ll admit, I have completely cooked breakfast (eggs and toast, rummaging, flipping and all) with the Quest Pro passthrough, while watching a video on a giant virtual screen floating in my kitchen. The depth accuracy is very impressive.
- zmmmmm 4y agoyes I think the success of the reprojection is a much under appreciated achievement. The author repeatedly questions their claim of "seamless" but its one of the most stunning aspects that you can, for example, see your real arm outside the headset exactly map onto your virtual arm and hand inside it, as do other objects that bridge from external reality to the virtual reality view. This "seamless" nature of it is really impressive when you consider that the whole view is actually being completely reprojected and not just displayed natively from the camera feeds.
- Zee2 4y agoYeah, I really don't think people understand how critical the reprojection is. A lot of folks, apparently Karl Guttag included, think that the resolution of the passthrough is the most important factor for usefulness. And/or that the resolution of the cameras is even directly related to the quality of the passthrough to begin with! The pixel count is just one of the dozens of quality metrics, and is arguably actually one of the less important ones. I also can't tell if KGuttag believes that you can simply pass the video feed through to the user and not have an absolute usability disaster. His paragraph about the "pyrrhic victory" sort of implies that not preferring depth-correct scene reprojection was an option. In my experience, that is absolutely non-negotiable in terms of shipping a device that's usable to the majority of consumers. Another reason why folks also don't understand why passthrough is so energy intensive. You're also running an entire additional pass of steresocopic spacewarp on what is essentially four different camera feeds, on an incredibly high resolution scene reconstruction/SLAM mesh.
- ncr100 4y agoPost to his blog if you want an answer - he responds
- wvenable 4y agoIn a regular Quest 2 (not the pro as discussed here) you can use pass-through and actually reach out and grab a glass of water and drink it. It's super impressive and actually makes the pass-through useful for the things that you use pass-through for. Would it be nice to be able to actually read text and not have distortion when objects get too close? Yes, of course. But it's not necessary.
- Zee2 4y agoExactly. Reading text or having 20/20 vision is so, so, so not the most important part of passthrough! Comfort, accuracy when reaching out and grabbing things, walking/running, avoiding obstacles on the ground, pets/humans, etc. Non-depth-correct passthrough risks all of these things.
- blensor 4y agoWe are actively using passthrough in VRWorkout (to be aware of your surroundings while jumping and knee high running) and I have to say that even the low quality Quest 2 passthrough feels magical for that [1]. If you do look at 0:20 you can see that it really shines when the virtual hands interact with real objects [1] https://youtu.be/DDwmXsvD-NI https://youtu.be/DDwmXsvD-NI
- cokeandpepsi 4y agoYeah I've been really enjoying the passthough as the homescreen background feature they added, it also works on WebXR pages too now which has been fun for the small experiments/toys I like to make on my downtime
- geokon 4y agoJust so I understand correctly - the fact that the cameras are places a couple of inches in front of where you're actual eyeballs are causes a lot of discomfort? And so you need to warp the camera feed to look like it should from your eyeball's perspective?
- andybak 4y agoI'm not sure if it's entirely due to the camera position or if there are other factors (e.g. differences between the camera optics and your eye's) but I can confirm that using a device without reprojection results in quite extreme distortion for objects closer than an arm's length. Which unfortunately includes your hands.
- ghusbands 4y agoThere's also the distance between the eyes not matching that of the cameras. And the fact that the 3d scene that isn't from the cameras does not have your eyes in the same position.