3 ms·

There are some good questions asked in this, but I think there are some potential counter arguments missed. > Low Processing Lag Time and High Frame Rate are N

There are some good questions asked in this, but I think there are some potential counter arguments missed. > Low Processing Lag Time and High Frame Rate are Necessary but not Sufficient to Solve Visual Issues I think I read somewhere that Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. > Poor location of main cameras relative to the user’s eye due to the Eyesight Display > The proper location of the cameras would be coaxial with the user’s two eyes. The eyes have two degrees of movement, the camera can't be "coaxial" with them at all times. The proximity of the cameras to the axial direction of your eyes is going to be more important for short distance than long as the effect of misalignment is going to be stronger. We tend to be looking somewhat down when working on things near to us, and in the case of the Vision Pro that's where the cameras are. You are almost exactly looking down through the axis of the cameras when working on something on a desk or table in front of you. This is better than if they were horizontally in front of your eyes. I don't doubt that Apple have considered all these issues in the design, and any trade offs they may have made. Once we get our hands on hardware we will be better informed how well they have addressed them.

danpalmer 3y ago
> > Low Processing Lag Time and High Frame Rate are Necessary but not Sufficient to Solve Visual Issues > I think I read somewhere that Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. That technique is Foveated Rendering, and my understanding is that it doesn’t change the frame rate. I think if it did you would get obvious visual tearing at the boundaries of the different areas. I believe it normally involves just rendering areas at a lower resolution and then applying a basic blur to it. Additionally, I’m not certain if that will be applied to the pass through video, the most important bit for these visual issues. Adding foveated rendering to that pipeline could even do more harm than good in terms of adding latency, and video processing is typically much faster than the rendering of new graphics from scratch, particularly given the M1’s dedicated video accelerators.
- audunw 3y ago
  > I think if it did you would get obvious visual tearing at the boundaries of the different areas. They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. You would probably not notice it because the resolution in our peripheral vision is terrible. Even with sharp tear lines it might not be noticeable if the frame rate is fast enough. I don't think this is what they're doing, but it could be viable. I think you're right about the other issue with foveated rendering of pass-through video. You really don't want to do much processing. But imagine this: if you can control which line the camera starts on with its rolling shutter, you could start the camera and the display update a few lines above the point where your eye is focusing. You can blend the first few lines with the ones from the previous frame to make the tear line less obvious, which would require almost no processing. They may be processing a few lines at the time before sending it to the display for color correction, simple transformations, etc. Could be viable.
  - danpalmer 3y ago
    > They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. I'm just not sure that's possible for update frequency. For lower/higher resolution, sure, but even with our terrible peripheral vision I think we'd notice that the center of our vision is smooth while our peripheral vision is "sticking", particularly during rapid movement.
RealityVoid 3y ago
> I think I read somewhere that > Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. It's called foveated rendering. Everyone following the space should listen to carmack's keynotes. Whe gove over a lot of stuff there and it's really honest and, IMO, technically meaty.

danpalmer 3y ago

> > Low Processing Lag Time and High Frame Rate are Necessary but not Sufficient to Solve Visual Issues > I think I read somewhere that Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. That technique is Foveated Rendering, and my understanding is that it doesn’t change the frame rate. I think if it did you would get obvious visual tearing at the boundaries of the different areas. I believe it normally involves just rendering areas at a lower resolution and then applying a basic blur to it. Additionally, I’m not certain if that will be applied to the pass through video, the most important bit for these visual issues. Adding foveated rendering to that pipeline could even do more harm than good in terms of adding latency, and video processing is typically much faster than the rendering of new graphics from scratch, particularly given the M1’s dedicated video accelerators.

audunw 3y ago
> I think if it did you would get obvious visual tearing at the boundaries of the different areas. They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. You would probably not notice it because the resolution in our peripheral vision is terrible. Even with sharp tear lines it might not be noticeable if the frame rate is fast enough. I don't think this is what they're doing, but it could be viable. I think you're right about the other issue with foveated rendering of pass-through video. You really don't want to do much processing. But imagine this: if you can control which line the camera starts on with its rolling shutter, you could start the camera and the display update a few lines above the point where your eye is focusing. You can blend the first few lines with the ones from the previous frame to make the tear line less obvious, which would require almost no processing. They may be processing a few lines at the time before sending it to the display for color correction, simple transformations, etc. Could be viable.
- danpalmer 3y ago
  > They could gradually blend the area with higher refresh rate to the peripheral images with lower refresh rate. I'm just not sure that's possible for update frequency. For lower/higher resolution, sure, but even with our terrible peripheral vision I think we'd notice that the center of our vision is smooth while our peripheral vision is "sticking", particularly during rapid movement.

RealityVoid 3y ago

> I think I read somewhere that > Apple are using the eye tracking to concentrate processing power and refresh rate on the area you are looking. Eventually a higher refresh rate in a smaller section of the display. It's called foveated rendering. Everyone following the space should listen to carmack's keynotes. Whe gove over a lot of stuff there and it's really honest and, IMO, technically meaty.