6 ms·

Can someone explain to me why sufficiently good VR can't function as AR? Just to lay out my point: AR has to have extremely good detail, framerate, and even de

Can someone explain to me why sufficiently good VR can't function as AR? Just to lay out my point: AR has to have extremely good detail, framerate, and even depth of field to work well. It has to track everything perfectly and blend the two lightfields seemlessly or the immersion is broken and you risk nausea or at least annoyance. But if you have high framerate, ability to simulate depth of field, high knowledge of the environment (enough to overlay seemlessly), etc, then couldn't you just... operate purely in VR with reality streamed in via lightfield cameras? That would allow you to do things that are impossible in AR, like display darker overlays on lighter surfaces, and everything would be at the same framerate so it'd be easier to seemlessly integrate everything. So what does really, really good AR get you that really, really good VR with reality overlay does not? Seems like you can avoid a lot of super hard to effectively impossible hardware optics problems by just doing things with an opaque visor... I'm not an expert in this field, so I'm probably using terminology incorrectly.

blululu 7y ago
A number of challenges exist for pass through AR. I’m short the camera system that you need does not exist. Cameras add latency. This causes nausea. Camera resolution is low. When the screen is less than 10 cm from your eyes the grain is very noticeable. Cameras only have limited focal range. Camera dynamic range is low. I’m many settings it is hard to capture images that your eye could handle.
- Robotbeat 7y ago
  Thank you. That makes sense. I know some crazy high resolution lightfield cameras exist, but the hardware is so heavy there's no chance of getting it on your head. (In principle you could address this with a huge array of tiny cameras on a headset, I suppose? Like this: https://www.deccanchronicle.com/150421/technology-mobiles-and-tabs/article/52-megapixel-cameras-smartphones https://www.deccanchronicle.com/150421/technology-mobiles-an... or this: https://newsbeezer.com/austriaeng/lg-patent-shows-smartphone-with-16-cameras/ https://newsbeezer.com/austriaeng/lg-patent-shows-smartphone... ) Apparently display technology is also a limitation. Here's one attempt to pass-through AR (thank you for that vocabulary, BTW): https://www.youtube.com/watch?v=hNbTCpURpQs https://www.youtube.com/watch?v=hNbTCpURpQs Are you aware of any who do it better?
  - mycall 7y ago
    Retinal projection work for display tech. https://www.bynorth.com/focals https://www.bynorth.com/focals
    - xpeon 7y ago
      Pretty cool, but could you project 3d object on these?
- Dylan16807 7y ago
  > the camera system that you need does not exist VR can work at 90Hz. If you spend money on a quality camera and a quality screen you can manage less end-to-end latency for AR. Focal range will be fine if you have a good sensor. Grain will be fine if you have a good screen. Compressing the dynamic range won't break anything. It doesn't have to work in every setting.
taurath 7y ago
My uneducated take - AR in stationary is almost useless except in a few industrial applications. You want AR to be super mobile ideally. VR with reality piped in is a decent way to get what you can get out of stationary AR, but it’s not quite as useful.
gfodor 7y ago
I work in VR and my personal theory is the version you outline above is the long term model for immersive computing. Most people I talk to disagree, but as you mention my argument is that in the limit a system which allows full photonic control will be the final mechanic. Incidentally it’ll maybe happen sooner anyway, since viable VR pass through tech is arguably software gated today on a platform like the Quest. The chicken and egg problem is who is a) going to get it working well and b) have the guts to start wearing a eye covering headset for more and more % of their day. It may be a long iteration cycle and transparent AR may rule the day for a few generations until the world is used to the idea first.
meheleventyone 7y ago
With sufficiently good cameras you can already pass through video. Setting up boundaries in the Oculus Quest has you do so doing exactly that. The current issues are: * A limited field of view. AR headsets are more limited but don’t interfere as much with normal vision. * Latency of video display causing a noticeable mismatch to head movement. * Shutting the rest of the world out. No one can see your face properly so it’s much more invasive than AR might be.
- Robotbeat 7y ago
  Good point about others (not wearing headsets) not being able to see your face properly. I hadn't considered that.
- findthewords 7y ago
  I would like to note that the shortcomings mentioned above can be fixed: - a vr passthrough can achieve a much wider FOV than any ar headset. - latency isssues are debatable, I would say it's a non-issue - eye-tracking and front-facing displays solve gaze opaqueness. we already have cut-out camera holes in consumer grade displays I would say that THE current issues with passthrough vr is solving the convergence issue. varifocal lenses are not consumer level technology yet.
  - meheleventyone 7y ago
    VR pass through can get a wider FOV than an AR display but it still is a lot less wide than natural vision. Which was my point. Latency issues might be subjective but there are clearly going to be population trends. I imagine these will be very similar to latency trends with VR and as such latency will be a significant issue. Front facing displays are even weirder than someone with a headset on. I’m confused that anyone thinks sticking a view of someone’s eyes on a flat screen is going to be anything other than weird. It’s diving head first into the uncanny valley.
abakker 7y ago
Potential legal minefield, too. Take drone flying, for example. It requires that you maintain visual line of sight (most of the time, without special exceptions). But, as a drone flyer in a commercial setting, you frequently want to see through the camera also. Given that, AR suits nicely, since you can overall both. VR doesn't because it is blocking your true visual line of sight. In a perfect world, I agree with you. VR would be nicer, especially once the Cameras include thermal/night vision/other.
- Robotbeat 7y ago
  Good point about the thermal/night vision thing. We effectively do this already with nightvision goggles. It'd be super amazing to use pass-through stereographic ultrasound images for diving in murky waters (or smoke?). A potential game changer in some instances. Imagine directly overlaying a "4D" ultrasound in realtime with careful sensor position registration... it'd make all manner of medical procedures a lot clearer. You could immediately see what way a baby was turned (without training, even), for instance. I guess in those cases, you wouldn't even need conceptually seemless immersion, just precise enough for eye-hand coordination.
- ferongr 7y ago
  >But, as a drone flyer in a commercial setting, you frequently want to see through the camera also Commercial drone photo/videographers should be using a pilot to control the drone, while they themselves control the camera.
  - Robotbeat 7y ago
    I think the idea is that even the pilot may want to see through the camera, i.e. first person flying.
jonas21 7y ago
That's what Varjo is betting on: https://varjo.com/xr-1/ https://varjo.com/xr-1/
- corysama 7y ago
  Tried the XR-1 at SVVR. It's very impressive. The camera is very high rez and low latency. If I look for something to complain about, the colors are not very rich, it doesn't handle glaring light well, and your perspective is shifted a bit forward of your face because of the physical position of the cameras. But, still... I'm nitpicking an awesome piece of kit.

blululu 7y ago

A number of challenges exist for pass through AR. I’m short the camera system that you need does not exist. Cameras add latency. This causes nausea. Camera resolution is low. When the screen is less than 10 cm from your eyes the grain is very noticeable. Cameras only have limited focal range. Camera dynamic range is low. I’m many settings it is hard to capture images that your eye could handle.

Robotbeat 7y ago
Thank you. That makes sense. I know some crazy high resolution lightfield cameras exist, but the hardware is so heavy there's no chance of getting it on your head. (In principle you could address this with a huge array of tiny cameras on a headset, I suppose? Like this: https://www.deccanchronicle.com/150421/technology-mobiles-and-tabs/article/52-megapixel-cameras-smartphones https://www.deccanchronicle.com/150421/technology-mobiles-an... or this: https://newsbeezer.com/austriaeng/lg-patent-shows-smartphone-with-16-cameras/ https://newsbeezer.com/austriaeng/lg-patent-shows-smartphone... ) Apparently display technology is also a limitation. Here's one attempt to pass-through AR (thank you for that vocabulary, BTW): https://www.youtube.com/watch?v=hNbTCpURpQs https://www.youtube.com/watch?v=hNbTCpURpQs Are you aware of any who do it better?
- mycall 7y ago
  Retinal projection work for display tech. https://www.bynorth.com/focals https://www.bynorth.com/focals
  - xpeon 7y ago
    Pretty cool, but could you project 3d object on these?
Dylan16807 7y ago
> the camera system that you need does not exist VR can work at 90Hz. If you spend money on a quality camera and a quality screen you can manage less end-to-end latency for AR. Focal range will be fine if you have a good sensor. Grain will be fine if you have a good screen. Compressing the dynamic range won't break anything. It doesn't have to work in every setting.

taurath 7y ago

My uneducated take - AR in stationary is almost useless except in a few industrial applications. You want AR to be super mobile ideally. VR with reality piped in is a decent way to get what you can get out of stationary AR, but it’s not quite as useful.

gfodor 7y ago

I work in VR and my personal theory is the version you outline above is the long term model for immersive computing. Most people I talk to disagree, but as you mention my argument is that in the limit a system which allows full photonic control will be the final mechanic. Incidentally it’ll maybe happen sooner anyway, since viable VR pass through tech is arguably software gated today on a platform like the Quest. The chicken and egg problem is who is a) going to get it working well and b) have the guts to start wearing a eye covering headset for more and more % of their day. It may be a long iteration cycle and transparent AR may rule the day for a few generations until the world is used to the idea first.

meheleventyone 7y ago

With sufficiently good cameras you can already pass through video. Setting up boundaries in the Oculus Quest has you do so doing exactly that. The current issues are: * A limited field of view. AR headsets are more limited but don’t interfere as much with normal vision. * Latency of video display causing a noticeable mismatch to head movement. * Shutting the rest of the world out. No one can see your face properly so it’s much more invasive than AR might be.

Robotbeat 7y ago
Good point about others (not wearing headsets) not being able to see your face properly. I hadn't considered that.
findthewords 7y ago
I would like to note that the shortcomings mentioned above can be fixed: - a vr passthrough can achieve a much wider FOV than any ar headset. - latency isssues are debatable, I would say it's a non-issue - eye-tracking and front-facing displays solve gaze opaqueness. we already have cut-out camera holes in consumer grade displays I would say that THE current issues with passthrough vr is solving the convergence issue. varifocal lenses are not consumer level technology yet.
- meheleventyone 7y ago
  VR pass through can get a wider FOV than an AR display but it still is a lot less wide than natural vision. Which was my point. Latency issues might be subjective but there are clearly going to be population trends. I imagine these will be very similar to latency trends with VR and as such latency will be a significant issue. Front facing displays are even weirder than someone with a headset on. I’m confused that anyone thinks sticking a view of someone’s eyes on a flat screen is going to be anything other than weird. It’s diving head first into the uncanny valley.

abakker 7y ago

Potential legal minefield, too. Take drone flying, for example. It requires that you maintain visual line of sight (most of the time, without special exceptions). But, as a drone flyer in a commercial setting, you frequently want to see through the camera also. Given that, AR suits nicely, since you can overall both. VR doesn't because it is blocking your true visual line of sight. In a perfect world, I agree with you. VR would be nicer, especially once the Cameras include thermal/night vision/other.

Robotbeat 7y ago
Good point about the thermal/night vision thing. We effectively do this already with nightvision goggles. It'd be super amazing to use pass-through stereographic ultrasound images for diving in murky waters (or smoke?). A potential game changer in some instances. Imagine directly overlaying a "4D" ultrasound in realtime with careful sensor position registration... it'd make all manner of medical procedures a lot clearer. You could immediately see what way a baby was turned (without training, even), for instance. I guess in those cases, you wouldn't even need conceptually seemless immersion, just precise enough for eye-hand coordination.
ferongr 7y ago
>But, as a drone flyer in a commercial setting, you frequently want to see through the camera also Commercial drone photo/videographers should be using a pilot to control the drone, while they themselves control the camera.
- Robotbeat 7y ago
  I think the idea is that even the pilot may want to see through the camera, i.e. first person flying.

jonas21 7y ago

That's what Varjo is betting on: https://varjo.com/xr-1/ https://varjo.com/xr-1/

corysama 7y ago
Tried the XR-1 at SVVR. It's very impressive. The camera is very high rez and low latency. If I look for something to complain about, the colors are not very rich, it doesn't handle glaring light well, and your perspective is shifted a bit forward of your face because of the physical position of the cameras. But, still... I'm nitpicking an awesome piece of kit.