5 ms·

The author persistently conflates AR and MR throughout the posting, evaluating the Quest Pro as an AR device but the concluding at the end it's a terrible "MR"

by zmmmmm 4y ago

The author persistently conflates AR and MR throughout the posting, evaluating the Quest Pro as an AR device but the concluding at the end it's a terrible "MR" device because it failed at their AR tests. It seems to suggest they really think these terms are synonymous but they aren't. While AR aims to enable display and interaction with the real world but with augmentations, MR aims to bring virtual objects into the real world and allow you to interact with those. You really don't need such a high fidelity view of the real world to do that. The virtual objects are rendered with very high fidelity. I am fairly confused by the low res cameras used in the QPro. I have to assume Meta were just heavily constrained by the need for them to do double duty as tracking cameras and pass through cameras, so higher quality cameras just weren't available. But why only a single color pass through camera? This really doesn't make much sense other than if you allow that the processor on the Quest Pro is so limited that even if it had both feeds it probably couldn't process them. I think it's actually great news that there is so much optimization left to do. Seems like they should be able to improve it a lot over time, and in the meantime it's pretty adequate to enable exploration of some unique experiences. edit: VR => MR

charcircuit 4y ago
This blog post is written by Karl Guttag who is someone who is interested in AR, but not very interested in VR. The reason he is coming at it from an AR angle is that is what kind of device he is looking for. If you look at the rest of his blog you can see many articles related to AR.
- kakali 4y ago
  That’s fine, but it’s okay to call out the author’s bias. They contributed wonderful tests of the limitations of the headset. But the headset is only a failure for the authors use case. (AR)
  - charcircuit 4y ago
    I agree. If you know the author the bias is clear, but it's okay to point it out for people who are not familiar with him.
  - andybak 4y ago
    Having used Hololens vs several passthrough-AR devices (Quest 2 etc) I'd take "poor image quality" over "limited FOV and the need to use it in a darkened room" any day. Until AR display tech gets significantly better I think AR based on passthrough cameras is the only sensible approach.
Zee2 4y ago
As someone close to this in the industry, I definitely agree. I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here: > In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory. I don't think this is even close to capturing how important depth and perspective correct passthrough is. Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time. Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background. The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.
- nomel 4y ago
  I’ll admit, I have completely cooked breakfast (eggs and toast, rummaging, flipping and all) with the Quest Pro passthrough, while watching a video on a giant virtual screen floating in my kitchen. The depth accuracy is very impressive.
- zmmmmm 4y ago
  yes I think the success of the reprojection is a much under appreciated achievement. The author repeatedly questions their claim of "seamless" but its one of the most stunning aspects that you can, for example, see your real arm outside the headset exactly map onto your virtual arm and hand inside it, as do other objects that bridge from external reality to the virtual reality view. This "seamless" nature of it is really impressive when you consider that the whole view is actually being completely reprojected and not just displayed natively from the camera feeds.
georgeecollins 4y ago
I also think there is a lot to be said for a VR headset with even mediocre pass through. I find VR very engaging and I think it can be very useful -- you can have the screen space of many monitors if the headset has good enough resolution (not good enough on a quest), plus there is a ton you can do with modeling and visualization. But it is a big turn off to have no visual idea of what is going on around you. Just to be able to press a button and be able to see that there is someone standing around you, or you are too close to a couch, is a big win for a VR headset in my opinion. Obviously the visual fidelity will improve in the display and the pass through. But just starting to get some pass through is very nice.
mb7733 4y ago
> While AR aims to enable display and interaction with the real world but with augmentations, MR aims to bring virtual objects into the real world and allow you to interact with those. Would you mind giving user-facing examples to illustrate the difference? As in something you consider AR but not MR and vice-versa. I've googled a bit but I don't understand the distinction between "enable display and interaction with the real world but with [virtual] augmentations" and "bring virtual objects into the real world and allow you to interact with them". As someone with little AR/VR/MR experience these sound like two ways of phrasing the same thing. Thanks!
- deleted 4y ago
  [deleted]
- zmmmmm 4y ago
  Consider something like Gravity Sketch [0] as MR. You are designing a virtual object and it is useful to bring it into the real world to (a) collaborate on it with others in a shared workspace and (b) envision it within its real context. For example, you can design a chair and actually place it in a living room while you are working on it to see how it looks. An AR example would be the classic overlay of text on a sign you are looking at with a version that is translated into your language. [0] https://www.oculus.com/deeplink/?action=view&path=app/1587090851394426&ref=oculus_desktop https://www.oculus.com/deeplink/?action=view&path=app/158709...
  - makeitdouble 4y ago
    Not working in the field, I’m also having a hard time to grasp the difference. Is one MR because it’s 3D and the other AR because of 2D ? And is the collaboration part a defining feature ? Where would you set Pokemon Go’s “AR view” [0] kind of implementations for instance ? (First link I found was for a headset, but I’m also actually curious how you qualify the phone only version of it) Or what would it change if in you Gravity Sketch example each user would be seeing colors or rendering adapted to their vision/preferences making it a separate experience for each of them ? [0] https://youtu.be/48fVUHnR5I8 https://youtu.be/48fVUHnR5I8
Baeocystin 4y ago
I can't help but view the (myriad) odd choices in the Quest Pro in the light of Carmack's resignation letter. It really does have the camel feeling of a horse designed by committee, instead of anything filled with 'give a damn'. I hope the folks there can figure out the operational difficulties, and find a way past whatever org chart drama is the source of the dysfunction. The Quest 2 has been a delight, but enough time has passed that the limitations are starting to chafe. I would love to see a true Quest 3 made my people who actually like the things.
- xwolfi 4y ago
  The problem is that we dont exactly need ar/vr: we can put screens around for a fraction of the cost, and video games are nice to sit down: we can run after each other while giggling in the forest if we want some movement fun and dont exactly need anything overlayed. I can understand they try to see far to anticipate a future where they'd make a lot of money being useful, but there's such low latency and richness of perception with normal atomic real world interactions that I'm not sure we can or want to replace it with something that simply adds a paid intermediary. It s like adding a blockchain to transact cash for instance.
- zmmmmm 4y ago
  It's definitely a compromise device. My gut feeling is various factors delayed its delivery and they ultimately were left with the choice of, deliver it now as it is or potentially lose the whole business market due to being overrun by Apple, HTC or others in that arena. Nearly all the interesting tech in the Quest Pro will be outdated within a year - it would be impossible for them to ship it if they waited even another 6 months. So I think they said "ship it" and honestly I think they were right - as people say, ultimately "shipping is everything" [0]. Witness Apple struggling year after year to get their AR/VR product out. It's actually really hard, and if you wait for all your problems to be solved you will never ship anything. The product may be full of compromises but the process of shipping it will put Meta miles ahead of those that still haven't got anything out the door yet. [0] https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e94a8626 https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e...
sdiupIGPWEfh 4y ago
Personally, I actively reject this distinction between AR & MR. For as long as I've been aware of the concept, the term "Augmented Reality" covered both use cases, including virtual objects embedded into the real world. The distinction never struck me as interesting or useful; virtual objects embedded into the real world being the most useful way to interact with augmentations to the real world. Moreover, "Mixed Reality" strikes me as marketing-speak. Compare the search trends for AR [1] vs MR [2]. While both terms have been in use for a long time, AR was always more prevalent, with a notable rise in interest around 2009. My first encounter with the term MR was in marketing materials for Microsoft's HoloLens circa 2015. Now compare the search trends for MR [2] with HoloLens [3]. Searches for MR don't begin to trend upward until after the introduction of HoloLens, where Microsoft's marketing materials tried to make a big deal of said distinction and the rest of the world started to run with it. As a supporting argument for the original broader usage of the term AR, I'll cite discussions of a 2007 anime series, Dennō Coil [4], which explored various examples of fantastical AR usage. Probably just a coincidence, but the search trends for Dennō Coil [4] do precede the search trends for AR [1] rather closely. Naturally, all the usual caveats about word meanings changing over time apply, but this one is a pet peeve of mine. [1] https://trends.google.com/trends/explore?date=all&geo=US&q=%22augmented%20reality%22 https://trends.google.com/trends/explore?date=all&geo=US&q=%... [2] https://trends.google.com/trends/explore?date=all&geo=US&q=%22mixed%20reality%22 https://trends.google.com/trends/explore?date=all&geo=US&q=%... [3] https://trends.google.com/trends/explore?date=all&geo=US&q=HoloLens https://trends.google.com/trends/explore?date=all&geo=US&q=H... [4] https://trends.google.com/trends/explore?date=all&geo=US&q=%2Fm%2F02q9d95 https://trends.google.com/trends/explore?date=all&geo=US&q=%...
- mensetmanusman 4y ago
  AR will never be able to mix depth to objects with the real world due to the physics of image planes. MR could do that, so it might be one key difference.
  - Zee2 4y ago
    Full lightfield AR displays would have no issue with this. Even some hypothetical giga-sandwich of waveguide planes like a Magic Leap 1 on steroids could do an effective approximation of this. (Assuming by "mix depth to objects with the real world" you mean "having virtual objects alongside real objects with convincing depth alignment")
kguttag 4y ago
You are correct that MR today implies that the Virtual image is locked to the real world with some form of SLAM. I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean. My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality. There are probably decades of "optimizations" left to do.
grumbel 4y ago
> While AR aims to enable display and interaction with the real world but with augmentations, MR aims to bring virtual objects into the real world and allow you to interact with those. That's literally the same thing in terms of hardware. If you want to bring virtual objects in the real world, and not have weird z-sorting issues all over the place, you need to have an understanding of the real world, which in turn gives you the possibility to do augmentation. Beside, MR is just a confusingly overloaded term these days. It can refer to: * Microsoft's Mixed Reality brand of VR headsets (that completely lack any AR/MR features) * filming people playing VR games in third person and mixing the game footage into it with LIV or similar software * doing AR with VR headsets and pass-through cameras, instead of see-through optics (e.g. Lynx R1) > But why only a single color pass through camera? Pico4 makes the same mistake. I really don't get it. Lenovo Mirage Solo back in 2018 used two front facing cameras at a proper IPD, and while the image is still quite low resolution, having actual 3D with zero distortion made the thing feel like wearing actual glasses, not like looking at a weird camera feed. It's a ginormous quality improvement for very little extra effort. Using pass-through on that thing is still the only time I could literally forget I was wearing a headset.
gumby 4y ago
Is the AR/MR distinction you describe relevant to non-experts (99.999% of the prospective customer base) or just to those who have to implement it?

by zmmmmm 4y ago

charcircuit 4y ago

This blog post is written by Karl Guttag who is someone who is interested in AR, but not very interested in VR. The reason he is coming at it from an AR angle is that is what kind of device he is looking for. If you look at the rest of his blog you can see many articles related to AR.

kakali 4y ago
That’s fine, but it’s okay to call out the author’s bias. They contributed wonderful tests of the limitations of the headset. But the headset is only a failure for the authors use case. (AR)
- charcircuit 4y ago
  I agree. If you know the author the bias is clear, but it's okay to point it out for people who are not familiar with him.
- andybak 4y ago
  Having used Hololens vs several passthrough-AR devices (Quest 2 etc) I'd take "poor image quality" over "limited FOV and the need to use it in a darkened room" any day. Until AR display tech gets significantly better I think AR based on passthrough cameras is the only sensible approach.

Zee2 4y ago

As someone close to this in the industry, I definitely agree. I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here: > In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory. I don't think this is even close to capturing how important depth and perspective correct passthrough is. Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time. Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background. The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.

nomel 4y ago
I’ll admit, I have completely cooked breakfast (eggs and toast, rummaging, flipping and all) with the Quest Pro passthrough, while watching a video on a giant virtual screen floating in my kitchen. The depth accuracy is very impressive.
zmmmmm 4y ago
yes I think the success of the reprojection is a much under appreciated achievement. The author repeatedly questions their claim of "seamless" but its one of the most stunning aspects that you can, for example, see your real arm outside the headset exactly map onto your virtual arm and hand inside it, as do other objects that bridge from external reality to the virtual reality view. This "seamless" nature of it is really impressive when you consider that the whole view is actually being completely reprojected and not just displayed natively from the camera feeds.

georgeecollins 4y ago

I also think there is a lot to be said for a VR headset with even mediocre pass through. I find VR very engaging and I think it can be very useful -- you can have the screen space of many monitors if the headset has good enough resolution (not good enough on a quest), plus there is a ton you can do with modeling and visualization. But it is a big turn off to have no visual idea of what is going on around you. Just to be able to press a button and be able to see that there is someone standing around you, or you are too close to a couch, is a big win for a VR headset in my opinion. Obviously the visual fidelity will improve in the display and the pass through. But just starting to get some pass through is very nice.

mb7733 4y ago

> While AR aims to enable display and interaction with the real world but with augmentations, MR aims to bring virtual objects into the real world and allow you to interact with those. Would you mind giving user-facing examples to illustrate the difference? As in something you consider AR but not MR and vice-versa. I've googled a bit but I don't understand the distinction between "enable display and interaction with the real world but with [virtual] augmentations" and "bring virtual objects into the real world and allow you to interact with them". As someone with little AR/VR/MR experience these sound like two ways of phrasing the same thing. Thanks!

deleted 4y ago
[deleted]
zmmmmm 4y ago
Consider something like Gravity Sketch [0] as MR. You are designing a virtual object and it is useful to bring it into the real world to (a) collaborate on it with others in a shared workspace and (b) envision it within its real context. For example, you can design a chair and actually place it in a living room while you are working on it to see how it looks. An AR example would be the classic overlay of text on a sign you are looking at with a version that is translated into your language. [0] https://www.oculus.com/deeplink/?action=view&path=app/1587090851394426&ref=oculus_desktop https://www.oculus.com/deeplink/?action=view&path=app/158709...
- makeitdouble 4y ago
  Not working in the field, I’m also having a hard time to grasp the difference. Is one MR because it’s 3D and the other AR because of 2D ? And is the collaboration part a defining feature ? Where would you set Pokemon Go’s “AR view” [0] kind of implementations for instance ? (First link I found was for a headset, but I’m also actually curious how you qualify the phone only version of it) Or what would it change if in you Gravity Sketch example each user would be seeing colors or rendering adapted to their vision/preferences making it a separate experience for each of them ? [0] https://youtu.be/48fVUHnR5I8 https://youtu.be/48fVUHnR5I8

Baeocystin 4y ago

I can't help but view the (myriad) odd choices in the Quest Pro in the light of Carmack's resignation letter. It really does have the camel feeling of a horse designed by committee, instead of anything filled with 'give a damn'. I hope the folks there can figure out the operational difficulties, and find a way past whatever org chart drama is the source of the dysfunction. The Quest 2 has been a delight, but enough time has passed that the limitations are starting to chafe. I would love to see a true Quest 3 made my people who actually like the things.

xwolfi 4y ago
The problem is that we dont exactly need ar/vr: we can put screens around for a fraction of the cost, and video games are nice to sit down: we can run after each other while giggling in the forest if we want some movement fun and dont exactly need anything overlayed. I can understand they try to see far to anticipate a future where they'd make a lot of money being useful, but there's such low latency and richness of perception with normal atomic real world interactions that I'm not sure we can or want to replace it with something that simply adds a paid intermediary. It s like adding a blockchain to transact cash for instance.
zmmmmm 4y ago
It's definitely a compromise device. My gut feeling is various factors delayed its delivery and they ultimately were left with the choice of, deliver it now as it is or potentially lose the whole business market due to being overrun by Apple, HTC or others in that arena. Nearly all the interesting tech in the Quest Pro will be outdated within a year - it would be impossible for them to ship it if they waited even another 6 months. So I think they said "ship it" and honestly I think they were right - as people say, ultimately "shipping is everything" [0]. Witness Apple struggling year after year to get their AR/VR product out. It's actually really hard, and if you wait for all your problems to be solved you will never ship anything. The product may be full of compromises but the process of shipping it will put Meta miles ahead of those that still haven't got anything out the door yet. [0] https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e94a8626 https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e...

sdiupIGPWEfh 4y ago

Personally, I actively reject this distinction between AR & MR. For as long as I've been aware of the concept, the term "Augmented Reality" covered both use cases, including virtual objects embedded into the real world. The distinction never struck me as interesting or useful; virtual objects embedded into the real world being the most useful way to interact with augmentations to the real world. Moreover, "Mixed Reality" strikes me as marketing-speak. Compare the search trends for AR [1] vs MR [2]. While both terms have been in use for a long time, AR was always more prevalent, with a notable rise in interest around 2009. My first encounter with the term MR was in marketing materials for Microsoft's HoloLens circa 2015. Now compare the search trends for MR [2] with HoloLens [3]. Searches for MR don't begin to trend upward until after the introduction of HoloLens, where Microsoft's marketing materials tried to make a big deal of said distinction and the rest of the world started to run with it. As a supporting argument for the original broader usage of the term AR, I'll cite discussions of a 2007 anime series, Dennō Coil [4], which explored various examples of fantastical AR usage. Probably just a coincidence, but the search trends for Dennō Coil [4] do precede the search trends for AR [1] rather closely. Naturally, all the usual caveats about word meanings changing over time apply, but this one is a pet peeve of mine. [1] https://trends.google.com/trends/explore?date=all&geo=US&q=%22augmented%20reality%22 https://trends.google.com/trends/explore?date=all&geo=US&q=%... [2] https://trends.google.com/trends/explore?date=all&geo=US&q=%22mixed%20reality%22 https://trends.google.com/trends/explore?date=all&geo=US&q=%... [3] https://trends.google.com/trends/explore?date=all&geo=US&q=HoloLens https://trends.google.com/trends/explore?date=all&geo=US&q=H... [4] https://trends.google.com/trends/explore?date=all&geo=US&q=%2Fm%2F02q9d95 https://trends.google.com/trends/explore?date=all&geo=US&q=%...

mensetmanusman 4y ago
AR will never be able to mix depth to objects with the real world due to the physics of image planes. MR could do that, so it might be one key difference.
- Zee2 4y ago
  Full lightfield AR displays would have no issue with this. Even some hypothetical giga-sandwich of waveguide planes like a Magic Leap 1 on steroids could do an effective approximation of this. (Assuming by "mix depth to objects with the real world" you mean "having virtual objects alongside real objects with convincing depth alignment")

kguttag 4y ago

You are correct that MR today implies that the Virtual image is locked to the real world with some form of SLAM. I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean. My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality. There are probably decades of "optimizations" left to do.

grumbel 4y ago

> While AR aims to enable display and interaction with the real world but with augmentations, MR aims to bring virtual objects into the real world and allow you to interact with those. That's literally the same thing in terms of hardware. If you want to bring virtual objects in the real world, and not have weird z-sorting issues all over the place, you need to have an understanding of the real world, which in turn gives you the possibility to do augmentation. Beside, MR is just a confusingly overloaded term these days. It can refer to: * Microsoft's Mixed Reality brand of VR headsets (that completely lack any AR/MR features) * filming people playing VR games in third person and mixing the game footage into it with LIV or similar software * doing AR with VR headsets and pass-through cameras, instead of see-through optics (e.g. Lynx R1) > But why only a single color pass through camera? Pico4 makes the same mistake. I really don't get it. Lenovo Mirage Solo back in 2018 used two front facing cameras at a proper IPD, and while the image is still quite low resolution, having actual 3D with zero distortion made the thing feel like wearing actual glasses, not like looking at a weird camera feed. It's a ginormous quality improvement for very little extra effort. Using pass-through on that thing is still the only time I could literally forget I was wearing a headset.

gumby 4y ago

Is the AR/MR distinction you describe relevant to non-experts (99.999% of the prospective customer base) or just to those who have to implement it?