3 ms·

For foveation to go undetected, ideally the central region should update within 5 ms. That's unreachable even with top hardware right now. The MS paper achieve

For foveation to go undetected, ideally the central region should update within 5 ms. That's unreachable even with top hardware right now. The MS paper achieved some nice results using a 120 Hz screen and a very fast tracker, but the rift is far from that. It's interesting, but comes with all kind of issues aside from delay (periphery degradation strategy). I've been trying to adapt shadow mappin to foveation and latency is killing it. Especially since shadows are high contrast information, quality popping and self shadow flicks are so distracting.

doorhammer 12y ago
Yeah. I'm not involved in any kind of dev work with that sort of thing, so you'd probably know better than I would, but it definitely seems like it's a problem that's very much in the research phase. I can imagine the popping would be that much more painful in a VR environment. Game stutters that my brain largely tunes out on a normal monitor become world rocking cataclysmic events in the rift.
angersock 12y ago
So, do path-based rendering (which gets the nice, slow pop-in of noise, right?) and bias the sampling to favor the current point of focus. You naturally end up with fewer samples in places that don't matter, and faster fill-in where they do.
scott_karana 12y ago
Would dual GPUs help the problem? You could have one low-resolution GPU (such as an onboard Intel HD) and another focusing solely on rendering the foevated patch. 5ms per 320x240 section might not be implausible?
- evo 12y ago
  The problem isn't solely in rendering fast enough--it's end-to-end latency. Keep in mind that if your eye is being tracked by a 120Hz camera, that's 1/120 =~ 8.3ms latency just from inter-frame time alone. Add to that all sorts of communication bus overhead, image processing the eye tracking imagery to figure out eye direction, potential buffering in the display adapter, 5ms is not really possible, yet. That said, I don't think there's theoretical limitations that prevent extremely low latency systems from happening, it's just not something we've kept in mind when designing modern consumer hardware.
  - gknoy 12y ago
    Would you be able to throw hardware at it? You could render in high-detail not just the area that your eye can see well, but also the area around it which it could be pointing at if it moved during your measurement latency. (Can your eye look anywhere in your image within 20ms? I honestly have no idea.)
  - scott_karana 12y ago
    Yikes, that makes sense. Thank you for spelling it out a bit! :)
  - doorhammer 12y ago
    Yeah, I'm not an expert in this area, but my casual reading seems to make it out to be similar to the problems you have with a lot of VR stuff, where the hardware just hasn't needed to be ultra-low-latency before now, but that there aren't necessarily hard limits that would prevent it. If they ever get it ironed out, it seems like the kind of thing that could offer pretty huge advantages for gaming, if just one person is looking at the screen (I have no idea if anyone's considering multiple sets of eyes to track or if that introduces particularly novel problems that wouldn't be covered in solving the single set of eyes)

doorhammer 12y ago

Yeah. I'm not involved in any kind of dev work with that sort of thing, so you'd probably know better than I would, but it definitely seems like it's a problem that's very much in the research phase. I can imagine the popping would be that much more painful in a VR environment. Game stutters that my brain largely tunes out on a normal monitor become world rocking cataclysmic events in the rift.

angersock 12y ago

So, do path-based rendering (which gets the nice, slow pop-in of noise, right?) and bias the sampling to favor the current point of focus. You naturally end up with fewer samples in places that don't matter, and faster fill-in where they do.

scott_karana 12y ago

Would dual GPUs help the problem? You could have one low-resolution GPU (such as an onboard Intel HD) and another focusing solely on rendering the foevated patch. 5ms per 320x240 section might not be implausible?

evo 12y ago
The problem isn't solely in rendering fast enough--it's end-to-end latency. Keep in mind that if your eye is being tracked by a 120Hz camera, that's 1/120 =~ 8.3ms latency just from inter-frame time alone. Add to that all sorts of communication bus overhead, image processing the eye tracking imagery to figure out eye direction, potential buffering in the display adapter, 5ms is not really possible, yet. That said, I don't think there's theoretical limitations that prevent extremely low latency systems from happening, it's just not something we've kept in mind when designing modern consumer hardware.
- gknoy 12y ago
  Would you be able to throw hardware at it? You could render in high-detail not just the area that your eye can see well, but also the area around it which it could be pointing at if it moved during your measurement latency. (Can your eye look anywhere in your image within 20ms? I honestly have no idea.)
- scott_karana 12y ago
  Yikes, that makes sense. Thank you for spelling it out a bit! :)
- doorhammer 12y ago
  Yeah, I'm not an expert in this area, but my casual reading seems to make it out to be similar to the problems you have with a lot of VR stuff, where the hardware just hasn't needed to be ultra-low-latency before now, but that there aren't necessarily hard limits that would prevent it. If they ever get it ironed out, it seems like the kind of thing that could offer pretty huge advantages for gaming, if just one person is looking at the screen (I have no idea if anyone's considering multiple sets of eyes to track or if that introduces particularly novel problems that wouldn't be covered in solving the single set of eyes)