6 ms·

Relevant performance details: A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out

by daenz 6y ago

Relevant performance details: A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out the 11ms budget you have to render at 90hz (6.9ms for 144hz), and it doesn't appear to include rendering costs at all...that time is purely in upsampling. Cool tech but a ways to go in order to make it useful for VR.

rjeli 6y ago
Yeah, I wonder if they plan to fab an asic (FAIR already has their own accelerator asic) to run this on the next Oculus.
ethanwillis 6y ago
Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?
- garmaine 6y ago
  You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
  - KaiserPro 6y ago
    Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
    - chrisweekly 6y ago
      Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
  - vlovich123 6y ago
    If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
    - garmaine 6y ago
      Not in VR. You do that and you’ll induce vertigo.
    - dannyw 6y ago
      Add an additional 100ms of latency and people are going to throw up. Pipelining can be used to some degree, but definitely not what you are suggesting.
      - vlovich123 6y ago
        This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR. ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable. Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms. Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver. This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
- cheschire 6y ago
  Oh interesting. I wonder if you could combine temporal antialiasing techniques with this to get a pseudo upsampling by only upsampling portions of the screen. Maybe focus on edges every other frame, and do different flat surfaces every few frames in between the edge passes. Then use TAA concepts to blend the pixels over frames.
Vel0cityX 6y ago
My thoughts exactly, I didn't see anything on the paper addressing that. I suppose it's assumed that with the contributions of this one, future work can be done to make it faster.
alkonaut 6y ago
How close are we to working Foveal rendering with eye tracking so that upsampling only needs to be done for a small area of the screen? Spending precious milliseconds perfecting the corners of the image for VR seems like a complete waste.
- RWSen 6y ago
  Foveal rendering is in a weird spot. The software seems to be there, but mostly only in academia. The hardware is almost nowhere to be found, because it is another expense. People prefer spending that extra money on a better computer, since that improves every VR experience, not just the possibility of (part of) future experiences. FVR needs a hook: what can it do that "dumb" VR headsets don't?
onion2k 6y ago
it doesn't appear to include rendering costs at all...that time is purely in upsampling That part wouldn't be an issue if the plan is to render low resolution images in the cloud and stream them to a device that can upsample them locally. There wouldn't be any local rendering costs.
- esperent 6y ago
  That might be the plan, but it seems redundant to require users to have high end gaming hardware just to stream games.
- reitzensteinm 6y ago
  I'd be very surprised if this is what it's to be used for. The technique requires color, depth and motion vectors. That's three separate video channels, and two of them contain data that isn't usually stuffed into videos. Any compression artifacts are going to stick out like a sore thumb, so you'll need to stream very high quality, and you're going to have weird interactions between different layers being compressed differently.

by daenz 6y ago

rjeli 6y ago

Yeah, I wonder if they plan to fab an asic (FAIR already has their own accelerator asic) to run this on the next Oculus.

ethanwillis 6y ago

Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?

garmaine 6y ago
You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
- KaiserPro 6y ago
  Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
  - chrisweekly 6y ago
    Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
- vlovich123 6y ago
  If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
  - garmaine 6y ago
    Not in VR. You do that and you’ll induce vertigo.
  - dannyw 6y ago
    Add an additional 100ms of latency and people are going to throw up. Pipelining can be used to some degree, but definitely not what you are suggesting.
    - vlovich123 6y ago
      This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR. ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable. Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms. Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver. This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
cheschire 6y ago
Oh interesting. I wonder if you could combine temporal antialiasing techniques with this to get a pseudo upsampling by only upsampling portions of the screen. Maybe focus on edges every other frame, and do different flat surfaces every few frames in between the edge passes. Then use TAA concepts to blend the pixels over frames.

Vel0cityX 6y ago

My thoughts exactly, I didn't see anything on the paper addressing that. I suppose it's assumed that with the contributions of this one, future work can be done to make it faster.

alkonaut 6y ago

How close are we to working Foveal rendering with eye tracking so that upsampling only needs to be done for a small area of the screen? Spending precious milliseconds perfecting the corners of the image for VR seems like a complete waste.

RWSen 6y ago
Foveal rendering is in a weird spot. The software seems to be there, but mostly only in academia. The hardware is almost nowhere to be found, because it is another expense. People prefer spending that extra money on a better computer, since that improves every VR experience, not just the possibility of (part of) future experiences. FVR needs a hook: what can it do that "dumb" VR headsets don't?

onion2k 6y ago

it doesn't appear to include rendering costs at all...that time is purely in upsampling That part wouldn't be an issue if the plan is to render low resolution images in the cloud and stream them to a device that can upsample them locally. There wouldn't be any local rendering costs.

esperent 6y ago
That might be the plan, but it seems redundant to require users to have high end gaming hardware just to stream games.
reitzensteinm 6y ago
I'd be very surprised if this is what it's to be used for. The technique requires color, depth and motion vectors. That's three separate video channels, and two of them contain data that isn't usually stuffed into videos. Any compression artifacts are going to stick out like a sore thumb, so you'll need to stream very high quality, and you're going to have weird interactions between different layers being compressed differently.