3 ms·

> > Low Processing Lag Time and High Frame Rate are Necessary but not Sufficient to Solve Visual Issues > I think I read somewhere that Apple are using the eye

> > Low Processing Lag Time and High Frame Rate are Necessary but not Sufficient to Solve Visual Issues > I think I read somewhere that Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. That technique is Foveated Rendering, and my understanding is that it doesn’t change the frame rate. I think if it did you would get obvious visual tearing at the boundaries of the different areas. I believe it normally involves just rendering areas at a lower resolution and then applying a basic blur to it. Additionally, I’m not certain if that will be applied to the pass through video, the most important bit for these visual issues. Adding foveated rendering to that pipeline could even do more harm than good in terms of adding latency, and video processing is typically much faster than the rendering of new graphics from scratch, particularly given the M1’s dedicated video accelerators.

audunw 3y ago
> I think if it did you would get obvious visual tearing at the boundaries of the different areas. They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. You would probably not notice it because the resolution in our peripheral vision is terrible. Even with sharp tear lines it might not be noticeable if the frame rate is fast enough. I don't think this is what they're doing, but it could be viable. I think you're right about the other issue with foveated rendering of pass-through video. You really don't want to do much processing. But imagine this: if you can control which line the camera starts on with its rolling shutter, you could start the camera and the display update a few lines above the point where your eye is focusing. You can blend the first few lines with the ones from the previous frame to make the tear line less obvious, which would require almost no processing. They may be processing a few lines at the time before sending it to the display for color correction, simple transformations, etc. Could be viable.
- danpalmer 3y ago
  > They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. I'm just not sure that's possible for update frequency. For lower/higher resolution, sure, but even with our terrible peripheral vision I think we'd notice that the center of our vision is smooth while our peripheral vision is "sticking", particularly during rapid movement.

audunw 3y ago

> I think if it did you would get obvious visual tearing at the boundaries of the different areas. They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. You would probably not notice it because the resolution in our peripheral vision is terrible. Even with sharp tear lines it might not be noticeable if the frame rate is fast enough. I don't think this is what they're doing, but it could be viable. I think you're right about the other issue with foveated rendering of pass-through video. You really don't want to do much processing. But imagine this: if you can control which line the camera starts on with its rolling shutter, you could start the camera and the display update a few lines above the point where your eye is focusing. You can blend the first few lines with the ones from the previous frame to make the tear line less obvious, which would require almost no processing. They may be processing a few lines at the time before sending it to the display for color correction, simple transformations, etc. Could be viable.

danpalmer 3y ago
> They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. I'm just not sure that's possible for update frequency. For lower/higher resolution, sure, but even with our terrible peripheral vision I think we'd notice that the center of our vision is smooth while our peripheral vision is "sticking", particularly during rapid movement.