6 ms·

Neural Supersampling for Real-Time Rendering

102 points by jsheard 6y ago

warvstar 6y ago
Would love to add this to my game engine.
- Vel0cityX 6y ago
  Did you read the paper? Or the benchmarks at least? In its fastest mode, it takes like 18ms. Not even usable in real time if you target 30fps. Great start but definitely needs additional work to be usable in games.
  - pseudosavant 6y ago
    There is a big difference between latency and throughput. FPS is throughput. If you assume the entire system is producing only the current frame then those numbers are directly correlated. But most systems, especially game engines/hardware, always have multiple things going in parallel simultaneously. The H.264 encoder on my CPU introduces >16.7ms of latency into a video stream, but it can encode hundreds of frames per second of SD video all day. Adding ~1 more frame of latency may be worth a quadrupling in image quality/resolution in most circumstances.
bigdict 6y ago
Nvidia recently demoed a similar technique running in real time: https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-your-questions-answered/ https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-your-q...
- jsheard 6y ago
  More than demoed, they've shipped DLSS in quite a few games now. The 1.0 version was underwhelming but the 2.0 version works extremely well in practice. However Nvidia are treating DLSS as their secret sauce and not publishing any details, so Facebook's more open research is interesting even if it's not as refined yet.
  - Scaevolus 6y ago
    This appears to be more accurate than DLSS according to Fig. 11 in the paper, and has the 16x mode as well as 4x.
    - dannyw 6y ago
      Meanwhile DLSS is real time and this takes 100ms a frame.
      - goldenkey 6y ago
        DLSS is possibly baked into the graphics cards. It is what the RT cores are for. Facebook is just releasing a POC. I doubt the code is handwritten CUDA optimized for performance.
        gerardvivancos 6y ago
        On Nvidia cards the RT cores are the ones handling BVH traversal and ray-triangle intersection operations. The part of the hardware that is running the DLSS ML model are the tensor cores. But the algorithm and the model itself is not baked in, it is provided in the driver and/or game
    - bcatanzaro 6y ago
      That was DLSS 1.0 they compared with.
  - cma 6y ago
    > However Nvidia are treating DLSS as their secret sauce and not publishing any details https://developer.nvidia.com/gtc/2020/video/s22698 https://developer.nvidia.com/gtc/2020/video/s22698
    - jsheard 6y ago
      There are no real details in that presentation, they explain what input data DLSS needs, and demonstrate the outputs it can produce, but gloss over how it actually gets there. From the inputs you can glean that DLSS is using temporal integration but that's hardly a new idea, the novel part is in how it performs the integration and Nvidia isn't sharing that.
daenz 6y ago
Relevant performance details: A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out the 11ms budget you have to render at 90hz (6.9ms for 144hz), and it doesn't appear to include rendering costs at all...that time is purely in upsampling. Cool tech but a ways to go in order to make it useful for VR.
- rjeli 6y ago
  Yeah, I wonder if they plan to fab an asic (FAIR already has their own accelerator asic) to run this on the next Oculus.
- ethanwillis 6y ago
  Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?
  - garmaine 6y ago
    You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
    - KaiserPro 6y ago
      Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
      - chrisweekly 6y ago
        Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
    - vlovich123 6y ago
      If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).
tanilama 6y ago
Isn't this literally NVIDIA's dlss? And it has already been productized.
- Vel0cityX 6y ago
  Except Nvidia has published pretty much nothing about their method.
  - pixelhorse 6y ago
    They did publish a video on how it works, but I can't find it right now. The inputs are similar: https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/news/nvidia-dlss-2-0/nvidia-dlss-2-0-architecture.png https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/n... In contrast to DLSS1, the output of the NN is not color values, but sampling locations and weights, to look up the color values from the previous low-resolution frames.
carrolldunham 6y ago
scanning through it for the clause that gives away what slight of hand was used to correctly get BERLIN from nothing. Suspects > and combines the additional auxiliary information > multiple frames In other words, the label "Low Resolution Input" on the blurry images is misleading. The image should be labelled "some of the input".
- Vel0cityX 6y ago
  No idea what "some of the input" means, or why you thought "Low Resolution Input" is disingenuous? It uses color, depth and subpixel motion vectors of 1-4 previous frames. All things that modern game engines can easily calculate. You didn't even need to read the paper to get this info, it's literally in a picture on the blog post.
  - carrolldunham 6y ago
    Right - so a single low-res image should not be paired with the high-res one and labelled as input and output, because that implies the algorithm turned the one data into the other, which it did not do.
    - pseudosavant 6y ago
      This isn't about upsampling low-res bitmaps. It is a technique for upsampling the output of a game engine. The low-res image is itself output generated from a lot of other input the game engine generates. That same input that is already being generated anyway can also be fed into this to improve the post-processing. Finding ways to productively reuse already existing/generated data is the hallmark of any top (graphically) game. I read a detailed write-up on the graphics pipeline for GTA:V on Xbox 360. It blew my mind how many different ways they reused every single bit that ever hit the RAM. Which explains how they pulled off those graphics on a system with half as much RAM as an Apple Watch.

102 points by jsheard 6y ago

warvstar 6y ago

Would love to add this to my game engine.

Vel0cityX 6y ago
Did you read the paper? Or the benchmarks at least? In its fastest mode, it takes like 18ms. Not even usable in real time if you target 30fps. Great start but definitely needs additional work to be usable in games.
- pseudosavant 6y ago
  There is a big difference between latency and throughput. FPS is throughput. If you assume the entire system is producing only the current frame then those numbers are directly correlated. But most systems, especially game engines/hardware, always have multiple things going in parallel simultaneously. The H.264 encoder on my CPU introduces >16.7ms of latency into a video stream, but it can encode hundreds of frames per second of SD video all day. Adding ~1 more frame of latency may be worth a quadrupling in image quality/resolution in most circumstances.

bigdict 6y ago

Nvidia recently demoed a similar technique running in real time: https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-your-questions-answered/ https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-your-q...

jsheard 6y ago
More than demoed, they've shipped DLSS in quite a few games now. The 1.0 version was underwhelming but the 2.0 version works extremely well in practice. However Nvidia are treating DLSS as their secret sauce and not publishing any details, so Facebook's more open research is interesting even if it's not as refined yet.
- Scaevolus 6y ago
  This appears to be more accurate than DLSS according to Fig. 11 in the paper, and has the 16x mode as well as 4x.
  - dannyw 6y ago
    Meanwhile DLSS is real time and this takes 100ms a frame.
    - goldenkey 6y ago
      DLSS is possibly baked into the graphics cards. It is what the RT cores are for. Facebook is just releasing a POC. I doubt the code is handwritten CUDA optimized for performance.
      - gerardvivancos 6y ago
        On Nvidia cards the RT cores are the ones handling BVH traversal and ray-triangle intersection operations. The part of the hardware that is running the DLSS ML model are the tensor cores. But the algorithm and the model itself is not baked in, it is provided in the driver and/or game
  - bcatanzaro 6y ago
    That was DLSS 1.0 they compared with.
- cma 6y ago
  > However Nvidia are treating DLSS as their secret sauce and not publishing any details https://developer.nvidia.com/gtc/2020/video/s22698 https://developer.nvidia.com/gtc/2020/video/s22698
  - jsheard 6y ago
    There are no real details in that presentation, they explain what input data DLSS needs, and demonstrate the outputs it can produce, but gloss over how it actually gets there. From the inputs you can glean that DLSS is using temporal integration but that's hardly a new idea, the novel part is in how it performs the integration and Nvidia isn't sharing that.

daenz 6y ago

Relevant performance details: A Titan V GPU, using the 4x4 upsampling, at a target resolution of 1080p takes 24.42ms or 18.25ms for "fast" mode. This blows out the 11ms budget you have to render at 90hz (6.9ms for 144hz), and it doesn't appear to include rendering costs at all...that time is purely in upsampling. Cool tech but a ways to go in order to make it useful for VR.

rjeli 6y ago
Yeah, I wonder if they plan to fab an asic (FAIR already has their own accelerator asic) to run this on the next Oculus.
ethanwillis 6y ago
Do you need every single frame to be perfectly upsampled? Maybe there's a proportion of frames that could be rendered faster but with a less accurate method?
- garmaine 6y ago
  You need every frame to take less than 11ms (for 90Hz) or 8.3ms (120Hz) to render, or else you will get stuttering.
  - KaiserPro 6y ago
    Indeed, but you only need the part you are actively looking at in high resolution. the bit of your eye that need high resolution is surprisingly small.
    - chrisweekly 6y ago
      Others have also mentioned foveal rendering with eye tracking, which sounds brilliant but is apparently mostly limited to academia at present...
  - vlovich123 6y ago
    If you’re talking about the old synchronous model. The name of the game today is how to pipeline the shit out of everything. You’re frame can end up taking nearly 100ms to get to you’re eye from whatever starting point you want to talk about. As long as you’re delivering a frame every ~11ms then it’s fine. I wouldn’t be surprised to see future GPU architectures that end up driving individual sections of the display as content becomes available. Then even if it takes 50ms to update the entire screen, you’ve got a rolling shutter effect giving the illusion of a higher FPS (eg if each individual section is updating at 90fps and you have intelligent combination of the tiles to minimize graphical artifacts).

tanilama 6y ago

Isn't this literally NVIDIA's dlss? And it has already been productized.

Vel0cityX 6y ago
Except Nvidia has published pretty much nothing about their method.
- pixelhorse 6y ago
  They did publish a video on how it works, but I can't find it right now. The inputs are similar: https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/news/nvidia-dlss-2-0/nvidia-dlss-2-0-architecture.png https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/n... In contrast to DLSS1, the output of the NN is not color values, but sampling locations and weights, to look up the color values from the previous low-resolution frames.

carrolldunham 6y ago

scanning through it for the clause that gives away what slight of hand was used to correctly get BERLIN from nothing. Suspects > and combines the additional auxiliary information > multiple frames In other words, the label "Low Resolution Input" on the blurry images is misleading. The image should be labelled "some of the input".

Vel0cityX 6y ago
No idea what "some of the input" means, or why you thought "Low Resolution Input" is disingenuous? It uses color, depth and subpixel motion vectors of 1-4 previous frames. All things that modern game engines can easily calculate. You didn't even need to read the paper to get this info, it's literally in a picture on the blog post.
- carrolldunham 6y ago
  Right - so a single low-res image should not be paired with the high-res one and labelled as input and output, because that implies the algorithm turned the one data into the other, which it did not do.
  - pseudosavant 6y ago
    This isn't about upsampling low-res bitmaps. It is a technique for upsampling the output of a game engine. The low-res image is itself output generated from a lot of other input the game engine generates. That same input that is already being generated anyway can also be fed into this to improve the post-processing. Finding ways to productively reuse already existing/generated data is the hallmark of any top (graphically) game. I read a detailed write-up on the graphics pipeline for GTA:V on Xbox 360. It blew my mind how many different ways they reused every single bit that ever hit the RAM. Which explains how they pulled off those graphics on a system with half as much RAM as an Apple Watch.