5 ms·

I know people are very accustomed to miracles these days but I still find the fact that these guys are comfortably doing foveated rendering kind of crazy. Anyo

by pjs_ 3y ago

I know people are very accustomed to miracles these days but I still find the fact that these guys are comfortably doing foveated rendering kind of crazy. Anyone here work on that stuff and know how it is done in practice? Is it just things like AA and post-processing that are cranked up in the foveated region, or is it also using simpler meshes and whatnot? The former I can easily believe, the latter sounds really hardcore

xsmasher 3y ago
I assume the real challenge here is preventing some kind of "pop" when a pixel leaves or enters the eye's area of focus. Any kind of change in the periphery of your vision would be distracting.
- saiojd 3y ago
  Saccadic masking probably comes into play: https://en.wikipedia.org/wiki/Saccadic_masking https://en.wikipedia.org/wiki/Saccadic_masking (brains "drops frames" while the eyes move, so "popping" during eye movement probably won't be perceptible)
wlesieutre 3y ago
It’s probably rendered at a lower resolution outside of the focused area, so you don’t have fine detail, text becomes blurry, etc. The quest does this but without eye tracking, whatever you’re facing toward is higher detail than stuff toward the edges. Rendering full resolution at high frame rates for 23 million pixels would be too much for an M2 gpu, but most of your retina can’t see that fine detail so you don’t need to.
- MaxikCZ 3y ago
  The question is how much of rendering actually happens in M2 gpu, and how much is delegated to R1.
  - wlesieutre 3y ago
    From how they described it I don't think the R1 does any rendering, it's for handling input from a dozen cameras and other sensors within 12 ms. The one rendering related thing that I'd guess can be offloaded to that is warping the passthrough camera frames into whatever form they need for correct optical alignment.
c7DJTLrn 3y ago
Came here to write the same thing. How have Apple pulled this off? I remember watching one of Meta's researchers describe the extreme challenges, such as the fovea wobbling as it moves. They made it out to be unachievable any time soon, and yet here it is.

by pjs_ 3y ago

xsmasher 3y ago

I assume the real challenge here is preventing some kind of "pop" when a pixel leaves or enters the eye's area of focus. Any kind of change in the periphery of your vision would be distracting.

saiojd 3y ago
Saccadic masking probably comes into play: https://en.wikipedia.org/wiki/Saccadic_masking https://en.wikipedia.org/wiki/Saccadic_masking (brains "drops frames" while the eyes move, so "popping" during eye movement probably won't be perceptible)

wlesieutre 3y ago

It’s probably rendered at a lower resolution outside of the focused area, so you don’t have fine detail, text becomes blurry, etc. The quest does this but without eye tracking, whatever you’re facing toward is higher detail than stuff toward the edges. Rendering full resolution at high frame rates for 23 million pixels would be too much for an M2 gpu, but most of your retina can’t see that fine detail so you don’t need to.

MaxikCZ 3y ago
The question is how much of rendering actually happens in M2 gpu, and how much is delegated to R1.
- wlesieutre 3y ago
  From how they described it I don't think the R1 does any rendering, it's for handling input from a dozen cameras and other sensors within 12 ms. The one rendering related thing that I'd guess can be offloaded to that is warping the passthrough camera frames into whatever form they need for correct optical alignment.

c7DJTLrn 3y ago

Came here to write the same thing. How have Apple pulled this off? I remember watching one of Meta's researchers describe the extreme challenges, such as the fovea wobbling as it moves. They made it out to be unachievable any time soon, and yet here it is.