5 ms·

Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?

garmaine 6y ago
You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
- KaiserPro 6y ago
  Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
  - chrisweekly 6y ago
    Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
- vlovich123 6y ago
  If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
  - garmaine 6y ago
    Not in VR. You do that and you’ll induce vertigo.
  - dannyw 6y ago
    Add an additional 100ms of latency and people are going to throw up. Pipelining can be used to some degree, but definitely not what you are suggesting.
    - vlovich123 6y ago
      This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR. ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable. Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms. Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver. This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
cheschire 6y ago
Oh interesting. I wonder if you could combine temporal antialiasing techniques with this to get a pseudo upsampling by only upsampling portions of the screen. Maybe focus on edges every other frame, and do different flat surfaces every few frames in between the edge passes. Then use TAA concepts to blend the pixels over frames.

garmaine 6y ago

You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.

KaiserPro 6y ago
Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
- chrisweekly 6y ago
  Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
vlovich123 6y ago
If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
- garmaine 6y ago
  Not in VR. You do that and you’ll induce vertigo.
- dannyw 6y ago
  Add an additional 100ms of latency and people are going to throw up. Pipelining can be used to some degree, but definitely not what you are suggesting.
  - vlovich123 6y ago
    This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR. ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable. Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms. Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver. This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.

cheschire 6y ago

Oh interesting. I wonder if you could combine temporal antialiasing techniques with this to get a pseudo upsampling by only upsampling portions of the screen. Maybe focus on edges every other frame, and do different flat surfaces every few frames in between the edge passes. Then use TAA concepts to blend the pixels over frames.