8 ms·

So VR is hard enough - to avoid jitter that makes users feel sick you have to respond to a user's head movement, render a new frame with the new information the

So VR is hard enough - to avoid jitter that makes users feel sick you have to respond to a user's head movement, render a new frame with the new information the user should see, respond to any button presses, then draw the frame, in under 14-20ms. Magic Leap was always a much more difficult problem... they have to respond to a user's head movement, parse the scene the user is looking at (which could be anything), figure out what to draw and where to draw it on the user's environment, then render, all in the same 14-20ms window. Compounding that they have to do it with a much weaker CPU/GPU/battery than Oculus and friends, which use a phone, or are tethered to a PC with a $1,000 GPU. You wear Magic Leap on your head, no cables. On its face I was always sort of surprised to hear about the good demos; it always seemed like such a difficult problem, and I know the VR folks are having a tough enough time with it.

Retric 10y ago
You don't need to parse the environment from scratch 60 times every second. As long as you get head tracking right you can just focus on what's moving and assume it will keep moving the same way. Further the demos all seem to be around a fairly static environmenment. Remeber you don't need to render the background so a simplified internal model works fine with AR. If it works near a closeline blowing in the wind that's really hard and impressive.
ant6n 10y ago
Isn't the head-tracking separate from the graphics rendering? I mean, you could just render onto an area that's somewhat larger than your fov, and select some area every couple of ms based on the head movement.
- jobigoud 10y ago
  You want to integrate rendered objects onto the physical world, so you have to know exactly the pose these objects should be rendered at as if seen from your point of view. The objects transforms are piloted by the head tracking. Usually with pose prediction thrown in the mix to squash latency.
monk_e_boy 10y ago
I could never really figure out how the variable focus lightfield worked. I sort of assumed it was like a stack of images in the Z plane. So not only are they doing everything you mentioned, they are also rendering the 3D scene in slices?
sillysaurus3 10y ago
So VR is hard enough - to avoid jitter that makes users feel sick you have to respond to a user's head movement, render a new frame with the new information the user should see, respond to any button presses, then draw the frame, in under 14-20ms. A bit of a tangent, but: for some people, it's impossible for VR to avoid making them feel sick. It's fundamental to them wearing a VR headset rather than a technical challenge to be overcome. It's related to the fact that VR headsets can't project photons from arbitrary positions with arbitrary angles towards your eyes (i.e. a screen is planar, but the real world is a 3D volume). Turns out, evolution has ensured our bodies are very good at determining the difference between a screen's projection and the real world, resulting in a sick feeling when there's a mismatch. I think that when people accept it's inevitable some subset of users will get sick, the VR ecosystem will grow at a faster rate.
- code-monkey 10y ago
  Technically this would be possible to overcome with a light field display. Light field displays are currently very far off from becoming commercially viable.
  - chris_va 10y ago
    You could approximate this using a standard display if you could dynamically track and respond to focal depth. That might be harder than getting a functional light field display, though.
    - AstralStorm 10y ago
      Much easier, we have the tech in ophthalmology and adaptive lenses that are used in cellphone cameras.
      - chris_va 10y ago
        Out of curiosity, what would the tech stack look like? What I was picturing was: - LCD display - Patterned IR image projected onto eye - Camera to read IR image on the back of retina, to figure out current focal length of cornea - Realtime chip to adjust LCD display based on focal depth, to simulate light field image. ... The real issue is that response latency would have to be <10ms probably to avoid being disorienting.
simcop2387 10y ago
A trick i've heard at least one of the VR devices is doing isn't re-rendering the entire scene but since most of the movement is just head tracking, it renders a larger buffer and scrolls around in that buffer between frames to get a quicker response.
- pfranz 10y ago
  That only works for very, very minimal differences. It's the Oculus that does this. It guesses where your head will be, renders the scene at that location, and distorts the image slightly to match its guess vs reality. It also introduces a bit of softness, but seems to work pretty well. But it does re-render the whole image each frame.
  - simcop2387 10y ago
    I wonder if it does it to create intra-frame changes from rotation and/or motion of the head. If it does then that could help a lot with the experience
  - renox 10y ago
    >It's the Oculus that does this. And the PS no? I think that all the VR system will (optionally) do this to reduce the processing power needed.
    - pfranz 10y ago
      I have no idea about the PSVR. It's probably the VR device I've heard the least about (technically). I own one, and it works pretty well, but I don't know much about it. From my understanding this won't do anything to reduce power processing, it's all about reducing latency. That final transform happens at the last moment, with the most up-to-date head tracking information. I kind of figured it happened in the headset hardware itself in order to reduce the latency as much as possible.
- chrisseaton 10y ago
  That would only work if the scene was a single flat surface and you were only moving in parallel to it. If you were doing that you may as well not use VR and just look at a normal monitor. Otherwise it wouldn't match your motion, which is the problem you're trying to solve in the first place!
highd 10y ago
The latency / motion sickness issues probably aren't as bad, since you still see the surrounding environment to get a stable bearing. The display tech sounds very hard, though.
fossuser 10y ago
Michael Abrash wrote about this a while back and it made me suspect that Magic Leap wasn't where they were pretending to be. http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/ http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-... Two of the major unsolved problems he talks about are latency and the ability to draw black - I would be surprised if magic leap had solved both of these alone and in secret. Their vapid press releases didn't inspire confidence either.
Animats 10y ago
That's not the big unsolved problem with augmented reality. Those are all problems VR systems can already solve with enough money and transistors behind them. The AR big problem is displaying dark. You can put bright things on the display, but not dark ones. Microsoft demos their VR systems only in environments with carefully controlled dim lighting. The same is true of Meta. Laster just puts a dimming filter in front of the real world to dim it out so the overlays show up well. Is there any AR headgear which displays the real world optically and can selectively darken the real world? Magic Leap pretended to do that, but now it appears they can't. You could, of course, do it by focusing the real scene on a focal plane, like a camera, using a monochrome LCD panel as a shutter, and refocusing the scene to infinity. But the optics for that require some length, which means bulky headgear like night vision glasses. Maybe with nonlinear optics or something similarly exotic it might be possible. But if there was a easy way to do this, DoD would be using it for night vision gear.
- Ph0X 10y ago
  The part that's surprising to me is how instantly popular of a startup it became with so little information. Was this "demo" they gave so damn good that all the investors (some really reputable ones such as Google) started throwing money at it, without doing their research to see how real it was? Sounds very suspicious to me.
  - ryanSrich 10y ago
    Especially considering the nightmare of a funding climate we have. Those demos must have been truly life altering to warrant the $500m in investment. Profitable startups with hockey stick like growth can't even raise a couple million...damn.
    - gech 10y ago
      Why is this post greyed out? Did people downvote it? Is not a reasonable question to ask why capital is wasted?
      - 10y ago
psyc 10y ago
I worked on Hololens stuff at Microsoft. It does everything you describe, and IMO does it really well. It's fairly light and is wireless. The tracking is excellent. A lot of the scene parsing is done in special hardware, with low latency. It updates a persistent model of the environment, so it isn't rebuilding everything every frame.
- socmag 10y ago
  Any particular reason you stopped working on it?
  - CardenB 10y ago
    He probably moved to another secretive project in the hardware division. Thats what my friend did.
  - psyc 10y ago
    It was a contract position.
rsp1984 10y ago
The problem is even worse. In VR you're competing on latency with human muscle feedback and what your vestibular system is telling you. In AR, you're competing with on latency with the human visual system that you're trying to augment, which is a race you can't win. The only thing you can do is try to be very fast (<= 10 ms) so the latency becomes unnoticeable. Unfortunately right now this isn't possible unless you custom-engineer everything in your vision path optimized for latency. Fun R&D project but enormously time- and capital intensive with no guarantees for success.
- AstralStorm 10y ago
  AR though leaves much less of a problem with simulator sickness because normal vestibular system and motion perception is used elsewhere. VR is a much tougher nut to crack.
markingram 10y ago
Hello? Hololens did it already and tether-free. Here is my mixed reality capture tape taken on HoloLens. https://m.youtube.com/watch?v=Av3Fdx5RnUI https://m.youtube.com/watch?v=Av3Fdx5RnUI

Retric 10y ago

You don't need to parse the environment from scratch 60 times every second. As long as you get head tracking right you can just focus on what's moving and assume it will keep moving the same way. Further the demos all seem to be around a fairly static environmenment. Remeber you don't need to render the background so a simplified internal model works fine with AR. If it works near a closeline blowing in the wind that's really hard and impressive.

ant6n 10y ago

Isn't the head-tracking separate from the graphics rendering? I mean, you could just render onto an area that's somewhat larger than your fov, and select some area every couple of ms based on the head movement.

jobigoud 10y ago
You want to integrate rendered objects onto the physical world, so you have to know exactly the pose these objects should be rendered at as if seen from your point of view. The objects transforms are piloted by the head tracking. Usually with pose prediction thrown in the mix to squash latency.

monk_e_boy 10y ago

I could never really figure out how the variable focus lightfield worked. I sort of assumed it was like a stack of images in the Z plane. So not only are they doing everything you mentioned, they are also rendering the 3D scene in slices?

sillysaurus3 10y ago

So VR is hard enough - to avoid jitter that makes users feel sick you have to respond to a user's head movement, render a new frame with the new information the user should see, respond to any button presses, then draw the frame, in under 14-20ms. A bit of a tangent, but: for some people, it's impossible for VR to avoid making them feel sick. It's fundamental to them wearing a VR headset rather than a technical challenge to be overcome. It's related to the fact that VR headsets can't project photons from arbitrary positions with arbitrary angles towards your eyes (i.e. a screen is planar, but the real world is a 3D volume). Turns out, evolution has ensured our bodies are very good at determining the difference between a screen's projection and the real world, resulting in a sick feeling when there's a mismatch. I think that when people accept it's inevitable some subset of users will get sick, the VR ecosystem will grow at a faster rate.

code-monkey 10y ago
Technically this would be possible to overcome with a light field display. Light field displays are currently very far off from becoming commercially viable.
- chris_va 10y ago
  You could approximate this using a standard display if you could dynamically track and respond to focal depth. That might be harder than getting a functional light field display, though.
  - AstralStorm 10y ago
    Much easier, we have the tech in ophthalmology and adaptive lenses that are used in cellphone cameras.
    - chris_va 10y ago
      Out of curiosity, what would the tech stack look like? What I was picturing was: - LCD display - Patterned IR image projected onto eye - Camera to read IR image on the back of retina, to figure out current focal length of cornea - Realtime chip to adjust LCD display based on focal depth, to simulate light field image. ... The real issue is that response latency would have to be <10ms probably to avoid being disorienting.

simcop2387 10y ago

A trick i've heard at least one of the VR devices is doing isn't re-rendering the entire scene but since most of the movement is just head tracking, it renders a larger buffer and scrolls around in that buffer between frames to get a quicker response.

pfranz 10y ago
That only works for very, very minimal differences. It's the Oculus that does this. It guesses where your head will be, renders the scene at that location, and distorts the image slightly to match its guess vs reality. It also introduces a bit of softness, but seems to work pretty well. But it does re-render the whole image each frame.
- simcop2387 10y ago
  I wonder if it does it to create intra-frame changes from rotation and/or motion of the head. If it does then that could help a lot with the experience
- renox 10y ago
  >It's the Oculus that does this. And the PS no? I think that all the VR system will (optionally) do this to reduce the processing power needed.
  - pfranz 10y ago
    I have no idea about the PSVR. It's probably the VR device I've heard the least about (technically). I own one, and it works pretty well, but I don't know much about it. From my understanding this won't do anything to reduce power processing, it's all about reducing latency. That final transform happens at the last moment, with the most up-to-date head tracking information. I kind of figured it happened in the headset hardware itself in order to reduce the latency as much as possible.
chrisseaton 10y ago
That would only work if the scene was a single flat surface and you were only moving in parallel to it. If you were doing that you may as well not use VR and just look at a normal monitor. Otherwise it wouldn't match your motion, which is the problem you're trying to solve in the first place!

highd 10y ago

The latency / motion sickness issues probably aren't as bad, since you still see the surrounding environment to get a stable bearing. The display tech sounds very hard, though.

fossuser 10y ago

Michael Abrash wrote about this a while back and it made me suspect that Magic Leap wasn't where they were pretending to be. http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/ http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-... Two of the major unsolved problems he talks about are latency and the ability to draw black - I would be surprised if magic leap had solved both of these alone and in secret. Their vapid press releases didn't inspire confidence either.

Animats 10y ago

That's not the big unsolved problem with augmented reality. Those are all problems VR systems can already solve with enough money and transistors behind them. The AR big problem is displaying dark. You can put bright things on the display, but not dark ones. Microsoft demos their VR systems only in environments with carefully controlled dim lighting. The same is true of Meta. Laster just puts a dimming filter in front of the real world to dim it out so the overlays show up well. Is there any AR headgear which displays the real world optically and can selectively darken the real world? Magic Leap pretended to do that, but now it appears they can't. You could, of course, do it by focusing the real scene on a focal plane, like a camera, using a monochrome LCD panel as a shutter, and refocusing the scene to infinity. But the optics for that require some length, which means bulky headgear like night vision glasses. Maybe with nonlinear optics or something similarly exotic it might be possible. But if there was a easy way to do this, DoD would be using it for night vision gear.

Ph0X 10y ago
The part that's surprising to me is how instantly popular of a startup it became with so little information. Was this "demo" they gave so damn good that all the investors (some really reputable ones such as Google) started throwing money at it, without doing their research to see how real it was? Sounds very suspicious to me.
- ryanSrich 10y ago
  Especially considering the nightmare of a funding climate we have. Those demos must have been truly life altering to warrant the $500m in investment. Profitable startups with hockey stick like growth can't even raise a couple million...damn.
  - gech 10y ago
    Why is this post greyed out? Did people downvote it? Is not a reasonable question to ask why capital is wasted?
    - 10y ago

psyc 10y ago

I worked on Hololens stuff at Microsoft. It does everything you describe, and IMO does it really well. It's fairly light and is wireless. The tracking is excellent. A lot of the scene parsing is done in special hardware, with low latency. It updates a persistent model of the environment, so it isn't rebuilding everything every frame.

socmag 10y ago
Any particular reason you stopped working on it?
- CardenB 10y ago
  He probably moved to another secretive project in the hardware division. Thats what my friend did.
- psyc 10y ago
  It was a contract position.

rsp1984 10y ago

The problem is even worse. In VR you're competing on latency with human muscle feedback and what your vestibular system is telling you. In AR, you're competing with on latency with the human visual system that you're trying to augment, which is a race you can't win. The only thing you can do is try to be very fast (<= 10 ms) so the latency becomes unnoticeable. Unfortunately right now this isn't possible unless you custom-engineer everything in your vision path optimized for latency. Fun R&D project but enormously time- and capital intensive with no guarantees for success.

AstralStorm 10y ago
AR though leaves much less of a problem with simulator sickness because normal vestibular system and motion perception is used elsewhere. VR is a much tougher nut to crack.

markingram 10y ago

Hello? Hololens did it already and tether-free. Here is my mixed reality capture tape taken on HoloLens. https://m.youtube.com/watch?v=Av3Fdx5RnUI https://m.youtube.com/watch?v=Av3Fdx5RnUI