6 ms·

Arm Accuracy Super Resolution

122 points by rcarmo 2y ago

a1o 2y ago
Why do results need a license?
- Piskvorrr 2y ago
  That's a misparse, I think: "We are very proud of the results of this work and want to share it" They want to share the work, not the results (otherwise the sentence would be "we want to share THEM").
- Y-bar 2y ago
  The results of the research and development effort, e.g. the code and resources like documentation is how I was reading that sentence.
meindnoch 2y ago
When did we reach such a plateau with GPU tech, that we have to resort to this fake super-resolution stuff instead of just rendering more pixels?
- Almondsetat 2y ago
  explain what's fake about it without invalidating the entire history of video game performance tricks
  - doctorhandshake 2y ago
    I love the fake nature of graphics tricks. It’s smoke and mirrors but the mirrors and the smoke are fake too!
- doctorhandshake 2y ago
  Not every frame is rendered on the highest powered hardware with the highest settings. On mobile, battery life is important. On desktop, in gaming, many players don’t have the highest-end graphics cards. And even for those who do, GPU resources are often committed to other tasks beyond rendering like simulation, NPC AI, etc. Super resolution allows for apparently higher pixel count with (in some cases) acceptable image quality while still using GPU for compute or not burning up the battery.
- WithinReason 2y ago
  Faking is what real time computer graphics is all about. You should rather ask: Why should we perform a long chain of complex computations just to produce the same pixel that was already there on the previous frame, instead of using our resources to produce something new?
  - keyringlight 2y ago
    Another aspect to the temporal-based techniques is that they jitter where the sample is done within the pixel, so over a few frames it accumulates more information than purely having that one sample per frame as the truth of what that pixel is. Plus if the renderer knows about the world and how its moving, and passes that to the TAA function you can track the rendered history for that element of the scene when it's moved. There's more information available to create a higher quality image for a low cost. Render resolution has been less tightly linked to display resolution for a long time, and some games have done that on different elements in a scene for a long time (I remember seeing it in Psychonauts from 2005). Variable Rate Shading is another tool developers have to pull resources away from areas it's not going to be noticed. What I've been wondering about for a while is what's the furthest a developer could push it to smartly spend performance instead of doing it on the full frame, how much effort does it take to hint to these systems that different elements are more or less important, and then compose them together in a way that doesn't have obvious flaws.
- klodolph 2y ago
  If you remember back in the day when we just had MSAA and nothing else, it was horribly slow. “Just render more pixels” turns out to give you poor efficiency. Better efficiency = you can spend the cycles doing something more interesting. In other words—if GPUs were powerful enough to just push more pixels to the screen, then we would still use these upsampling techniques anyway, save a bunch of computational power, and then spend that computational power improving the graphics in other ways.
- dudegpus 2y ago
  This is not fake... We literally generate 30-60 pictures every single second. Every picture has information in it which combined gives you more unfaked information. Your viewpoint doesn't change it's content every single frame. This only happens when you rotate super fast 180degree or warp through a portal or whatever.
  - chrisjj 2y ago
    Actually 1deg is enough.
    - dudegpus 2y ago
      It doesn't change completely. We take the information from the previous frame and now have more information than before as long as it's the same object we render
- GrantMoyer 2y ago
  All anti-aliasing techniques are essentially faking the effect of subsampling a higher resolution render. Temporal anti-aliasing is a pretty darn good anti-aliasing method, since it uses more real information than other methods except for (well implemented) brute force supersampling. It's also a little unique, since it can be tweaked to actually produce that intermediate higher resolution render, which is how we get temporal super resolution.
- numpad0 2y ago
  Grainy 60fps is better experience than choppy FHD, to both play and to deliver to consumers. Right now, GPU performance on lower end phones is a problem that both differentiates high end ARM chips from the rest, and is also harming app growths. 60fps on non-SD8 SoCs is going to sell more Play Store cards, and also co-incidentally more ARM licenses. Maybe it could also preemptively save Google Tensor before its public reputation points run out. Game players also gets better experience, but that's maybe entirely coincidental.
- archagon 2y ago
  FWIW, DLSS feels like magic. I get something like 50% perf improvement with almost no loss in fidelity. Lets me play modern games at 4k/120hz on my 3080 without too many problems.
- sitkack 2y ago
  TFA shows how much energy this saves.
- kevingadd 2y ago
  It's less a plateau and more scaling boundaries. Mobile chips especially are limited in power and die space so if you want to produce a high-resolution image at acceptable frame rates, either quality-per-pixel or pixel count have to drop compared to stuff that's plugged into the wall. Each generation of chip on desktop or mobile is still much faster than the previous, but that's battling with increased quality-per-pixel and higher screen resolutions/refresh rates. DLSS/XeSS allow dropping the pixel count and then reconstructing a satisfactory image, which means you can either cut power draw for near-equivalent quality, or use the newly available headroom to improve quality-per-pixel further or deliver higher framerates. One dirty secret when it comes to mobile phones and tablets is that games were already rendering below native resolution - Apple basically told developers to do this in the early Retina era, since it wasn't feasible to render at native with good framerates - since it saves power and the screens are so high-res that most users won't be able to tell the difference. I can't remember the last time I launched a game on my phone and it was actually rendering at native resolution. So in that context, techniques like FSR or this new upscaler are just replacing the existing bilinear/bicubic upscale filter.
rendaw 2y ago
The article lists two methods: * Spatial - Inferring data from nearby pixels in a single frame (?) * Temporal - Inferring data from a point on previous and next frames I guess this is super resolution in the statistical sense, rather than "AI super resolution" which would infer data from similar data in other photos/videos. I'm surprised, I thought this was a dead area for research and AI super resolution had fully supplanted it. Are there any open source implementations of this, ex: for photography? I was digging into this a few years ago and all I could find was the commercial "PhotoAcute" which is basically dead but I somehow managed to get a key for and it... barely works.
- GrantMoyer 2y ago
  Temporal super resalution needs to accumulate data from multiple frames to work. Essentially, each frame renders only a subset of pixels, but crucially each frame renders a different subset, so you can combine information over a few frames to compute what should be at each pixel. There're some complications to account for things moving around the scene, but that's the basic idea. So there's no way the technique or similar techniques can be used to upscale a single photo in isolation. However, conceptually it's similar to capturing a long exposure image of dark scene, like the night sky, to get a low-noise, brighter photo. Temporal super resolution accumulates information from rendered pixels over time, while a long exposure photo accumulates information from captured photons over time.
  - rendaw 2y ago
    PhotoAcute uses several photos with some jitter, like you might get from burst shots on a phone. I'd expect other software (and possibly phone upscalers transparently) might do this too.
- jlouis 2y ago
  A key component is that when a game generates pixels, it also generates other data. Such as a Z-buffer for pixel depth and motion vectors (for motion blur, among other things). Feeding this to the upscaling algorithm vastly improves image quality. The larger problem to solve with these upscalers are temporal artifacts when using the above information. This lead to a number of heuristics, but if you train a neural network to do the work, it tends to perform better than the heuristics. There's still a ton of research going into non-ML solutions however, because the current console generation doesn't have AI-acceleration available, nor does a lot of hardware out there. So there's some longevity to be had by doing a good job for all that hardware out there.
kibwen 2y ago
Graphical fidelity isn't the reason I don't play mobile games. I don't play mobile games because touchscreens are terribly deficient compared to a controller, and because the monetization of mobile games is invariably psychotic.
- jsheard 2y ago
  There's no getting around the control issues without a clunky peripheral, but this site helps to find the rare examples which aren't designed to bleed you dry at least: https://nobsgames.stavros.io https://nobsgames.stavros.io Predictably many of the high quality games in that category are ones which started their life on PC or consoles and were ported to mobile later. Mobile-first game development is rotten to the core.
  - Helmut10001 2y ago
    Too bad they don't have a kids category. Thanks for the tipp though!
    - evilduck 2y ago
      Apple Arcade, a modest subscription in itself, is filled with mostly kid friendly games and none of them are allowed to have iAPs (or the iAPs are made free and then bundled in order to be featured in the Arcade catalog).
    - recursivecaveat 2y ago
      I would suggest getting a last gen handheld, since they're pretty cheap now, have good libraries of actual proper games (especially with emulation), and physical controls just can't be beat. I got a near-perfect vita from Japan for $90 and I'm quite happy with it. Not as convenient as the phone in your pocket, but if you're carrying around a backpack anyways it's not bad, and for kids maybe encouraging them off smartphones ain't so bad (:
  - asdff 2y ago
    The Nintendo DS light fits comfortably in a pocket on the other hand and had great games made for it that took advantage of that platform's featureset (I personally think the legend of zelda games for that platform have some of the best controls out of any handheld nintendo game with the stylus gestures). Clunky peripheral is from a lack of imagination not from a technological impossibility.
coldcode 2y ago
The one comparison example is sort of poor, as the "improved" part is bright, and the "original" part is dark and rather limited. It's hard to tell much from this.
KronisLV 2y ago
Honestly, I don't get why all the popular upscaling methods (DLSS, FSR, XeSS, ...) aren't integrated in all of the mainstream engines: Unity, Unreal, Godot... As far as I'm concerned, on the end of a game developer, that should be a single checkbox in the project options, or something similarly easy to choosing between 2X and 4X AA, both on the desktop and mobile, where supported by the hardware (or simpler generic upscaling, if nothing else suffices). The performance gains (at the expense of graphical fidelity, which may or may not be acceptable) would be staggering - you could quite literally squeeze out just enough performance to enjoy a game or an application even from older generation hardware for more modern titles.
- jsheard 2y ago
  Microsoft is pushing something along those lines with DirectSR, the idea is that game engines integrate one set of generic hooks which can then be routed to any of the vendor upscalers transparently. https://devblogs.microsoft.com/directx/directsr-preview/ https://devblogs.microsoft.com/directx/directsr-preview/
  - KronisLV 2y ago
    If it's done well, then that sounds exactly like what we need! Wonder whether something similar will land in Vulkan too, at some point.
- chrisjj 2y ago
  Because devs find them far less attractive in practice than the sales pitch implies.
  - keyringlight 2y ago
    I'd guess that a lot of devs it's not their main strength and their main concern is getting a working product out the door. If you're making "Bunnies go berserk" with the adventures of berserk rabbits as your main selling point, it seems unlikely that a smaller list of graphics fidelity options is going to dissuade anyone but the most hardcore graphics enthusiast from buying. All those options are also going to need time allocated for a graphics developer to tune to suit your game if the out-of-the-box defaults aren't great, and as it's an area still in progress upgrading APIs and revalidating for your game on a range of hardware. If you're doing multiplatform with consoles/mobile that's even more in the mix, and you need to consider what will reward the time spent on it.
LisaST 2y ago
[flagged]
chrisjj 2y ago
> Accuracy Super Resolution That would have been worth first checking with a native English speaker.
- saagarjha 2y ago
  What’s wrong with it
  - chrisjj 2y ago
    For starters, accuracy is not an adjective.
- deleted 2y ago
  [deleted]
DeathArrow 2y ago
Nvidia has the hardware power to use AI for upscaling. How many tensor cores do ARM socs have?
- cubefox 2y ago
  Well, the term "tensor core" is only used by Nvidia, and Nvidia does make Arm SoCs (Tegra), which may help you find the answer.
- caseyy 2y ago
  Doesn't matter. ASR is FSR2-based. It's just upscaling and sharpening with a proprietary shader. No neural nets are involved.
miohtama 2y ago
Is this based on some acquisition ARM has done? Because the sector is very un-ARM’y.
- enragedcacti 2y ago
  It seems right in their wheelhouse imo. They've just taken all of the hard work AMD did in the space with FSR and optimized it for their GPU architecture. I'd be pretty worried if ARM didn't know how to write a performant shader for their own GPU.
  - chrisjj 2y ago
    > I'd be pretty worried if ARM didn't know how to write a performant shader for their own GPU. We should probably worry why ARM feels it has to. If this was a great idea, game and game engine devs would have done it already.
asdff 2y ago
So tired of the dragon chasing with gaming graphics. The fruit they are chasing today is so low hanging yet demands such powerful hardware just to do something like render a beam of light through the woods. Meanwhile the game hardly looks better than crysis outside of that, especially when you are, you know, focused on playing the game and not pixel peeping. I wish developers put more effort into compatibility and cross platform support but I guess its not the developers making the engines after all, they are kind of beholden to them.
- deleted 2y ago
  [deleted]

122 points by rcarmo 2y ago

a1o 2y ago

Why do results need a license?

Piskvorrr 2y ago
That's a misparse, I think: "We are very proud of the results of this work and want to share it" They want to share the work, not the results (otherwise the sentence would be "we want to share THEM").
Y-bar 2y ago
The results of the research and development effort, e.g. the code and resources like documentation is how I was reading that sentence.

meindnoch 2y ago

When did we reach such a plateau with GPU tech, that we have to resort to this fake super-resolution stuff instead of just rendering more pixels?

Almondsetat 2y ago
explain what's fake about it without invalidating the entire history of video game performance tricks
- doctorhandshake 2y ago
  I love the fake nature of graphics tricks. It’s smoke and mirrors but the mirrors and the smoke are fake too!
doctorhandshake 2y ago
Not every frame is rendered on the highest powered hardware with the highest settings. On mobile, battery life is important. On desktop, in gaming, many players don’t have the highest-end graphics cards. And even for those who do, GPU resources are often committed to other tasks beyond rendering like simulation, NPC AI, etc. Super resolution allows for apparently higher pixel count with (in some cases) acceptable image quality while still using GPU for compute or not burning up the battery.
WithinReason 2y ago
Faking is what real time computer graphics is all about. You should rather ask: Why should we perform a long chain of complex computations just to produce the same pixel that was already there on the previous frame, instead of using our resources to produce something new?
- keyringlight 2y ago
  Another aspect to the temporal-based techniques is that they jitter where the sample is done within the pixel, so over a few frames it accumulates more information than purely having that one sample per frame as the truth of what that pixel is. Plus if the renderer knows about the world and how its moving, and passes that to the TAA function you can track the rendered history for that element of the scene when it's moved. There's more information available to create a higher quality image for a low cost. Render resolution has been less tightly linked to display resolution for a long time, and some games have done that on different elements in a scene for a long time (I remember seeing it in Psychonauts from 2005). Variable Rate Shading is another tool developers have to pull resources away from areas it's not going to be noticed. What I've been wondering about for a while is what's the furthest a developer could push it to smartly spend performance instead of doing it on the full frame, how much effort does it take to hint to these systems that different elements are more or less important, and then compose them together in a way that doesn't have obvious flaws.
klodolph 2y ago
If you remember back in the day when we just had MSAA and nothing else, it was horribly slow. “Just render more pixels” turns out to give you poor efficiency. Better efficiency = you can spend the cycles doing something more interesting. In other words—if GPUs were powerful enough to just push more pixels to the screen, then we would still use these upsampling techniques anyway, save a bunch of computational power, and then spend that computational power improving the graphics in other ways.
dudegpus 2y ago
This is not fake... We literally generate 30-60 pictures every single second. Every picture has information in it which combined gives you more unfaked information. Your viewpoint doesn't change it's content every single frame. This only happens when you rotate super fast 180degree or warp through a portal or whatever.
- chrisjj 2y ago
  Actually 1deg is enough.
  - dudegpus 2y ago
    It doesn't change completely. We take the information from the previous frame and now have more information than before as long as it's the same object we render
GrantMoyer 2y ago
All anti-aliasing techniques are essentially faking the effect of subsampling a higher resolution render. Temporal anti-aliasing is a pretty darn good anti-aliasing method, since it uses more real information than other methods except for (well implemented) brute force supersampling. It's also a little unique, since it can be tweaked to actually produce that intermediate higher resolution render, which is how we get temporal super resolution.
numpad0 2y ago
Grainy 60fps is better experience than choppy FHD, to both play and to deliver to consumers. Right now, GPU performance on lower end phones is a problem that both differentiates high end ARM chips from the rest, and is also harming app growths. 60fps on non-SD8 SoCs is going to sell more Play Store cards, and also co-incidentally more ARM licenses. Maybe it could also preemptively save Google Tensor before its public reputation points run out. Game players also gets better experience, but that's maybe entirely coincidental.
archagon 2y ago
FWIW, DLSS feels like magic. I get something like 50% perf improvement with almost no loss in fidelity. Lets me play modern games at 4k/120hz on my 3080 without too many problems.
sitkack 2y ago
TFA shows how much energy this saves.
kevingadd 2y ago
It's less a plateau and more scaling boundaries. Mobile chips especially are limited in power and die space so if you want to produce a high-resolution image at acceptable frame rates, either quality-per-pixel or pixel count have to drop compared to stuff that's plugged into the wall. Each generation of chip on desktop or mobile is still much faster than the previous, but that's battling with increased quality-per-pixel and higher screen resolutions/refresh rates. DLSS/XeSS allow dropping the pixel count and then reconstructing a satisfactory image, which means you can either cut power draw for near-equivalent quality, or use the newly available headroom to improve quality-per-pixel further or deliver higher framerates. One dirty secret when it comes to mobile phones and tablets is that games were already rendering below native resolution - Apple basically told developers to do this in the early Retina era, since it wasn't feasible to render at native with good framerates - since it saves power and the screens are so high-res that most users won't be able to tell the difference. I can't remember the last time I launched a game on my phone and it was actually rendering at native resolution. So in that context, techniques like FSR or this new upscaler are just replacing the existing bilinear/bicubic upscale filter.

rendaw 2y ago

The article lists two methods: * Spatial - Inferring data from nearby pixels in a single frame (?) * Temporal - Inferring data from a point on previous and next frames I guess this is super resolution in the statistical sense, rather than "AI super resolution" which would infer data from similar data in other photos/videos. I'm surprised, I thought this was a dead area for research and AI super resolution had fully supplanted it. Are there any open source implementations of this, ex: for photography? I was digging into this a few years ago and all I could find was the commercial "PhotoAcute" which is basically dead but I somehow managed to get a key for and it... barely works.

GrantMoyer 2y ago
Temporal super resalution needs to accumulate data from multiple frames to work. Essentially, each frame renders only a subset of pixels, but crucially each frame renders a different subset, so you can combine information over a few frames to compute what should be at each pixel. There're some complications to account for things moving around the scene, but that's the basic idea. So there's no way the technique or similar techniques can be used to upscale a single photo in isolation. However, conceptually it's similar to capturing a long exposure image of dark scene, like the night sky, to get a low-noise, brighter photo. Temporal super resolution accumulates information from rendered pixels over time, while a long exposure photo accumulates information from captured photons over time.
- rendaw 2y ago
  PhotoAcute uses several photos with some jitter, like you might get from burst shots on a phone. I'd expect other software (and possibly phone upscalers transparently) might do this too.
jlouis 2y ago
A key component is that when a game generates pixels, it also generates other data. Such as a Z-buffer for pixel depth and motion vectors (for motion blur, among other things). Feeding this to the upscaling algorithm vastly improves image quality. The larger problem to solve with these upscalers are temporal artifacts when using the above information. This lead to a number of heuristics, but if you train a neural network to do the work, it tends to perform better than the heuristics. There's still a ton of research going into non-ML solutions however, because the current console generation doesn't have AI-acceleration available, nor does a lot of hardware out there. So there's some longevity to be had by doing a good job for all that hardware out there.

kibwen 2y ago

Graphical fidelity isn't the reason I don't play mobile games. I don't play mobile games because touchscreens are terribly deficient compared to a controller, and because the monetization of mobile games is invariably psychotic.

jsheard 2y ago
There's no getting around the control issues without a clunky peripheral, but this site helps to find the rare examples which aren't designed to bleed you dry at least: https://nobsgames.stavros.io https://nobsgames.stavros.io Predictably many of the high quality games in that category are ones which started their life on PC or consoles and were ported to mobile later. Mobile-first game development is rotten to the core.
- Helmut10001 2y ago
  Too bad they don't have a kids category. Thanks for the tipp though!
  - evilduck 2y ago
    Apple Arcade, a modest subscription in itself, is filled with mostly kid friendly games and none of them are allowed to have iAPs (or the iAPs are made free and then bundled in order to be featured in the Arcade catalog).
  - recursivecaveat 2y ago
    I would suggest getting a last gen handheld, since they're pretty cheap now, have good libraries of actual proper games (especially with emulation), and physical controls just can't be beat. I got a near-perfect vita from Japan for $90 and I'm quite happy with it. Not as convenient as the phone in your pocket, but if you're carrying around a backpack anyways it's not bad, and for kids maybe encouraging them off smartphones ain't so bad (:
- asdff 2y ago
  The Nintendo DS light fits comfortably in a pocket on the other hand and had great games made for it that took advantage of that platform's featureset (I personally think the legend of zelda games for that platform have some of the best controls out of any handheld nintendo game with the stylus gestures). Clunky peripheral is from a lack of imagination not from a technological impossibility.

coldcode 2y ago

The one comparison example is sort of poor, as the "improved" part is bright, and the "original" part is dark and rather limited. It's hard to tell much from this.

KronisLV 2y ago

Honestly, I don't get why all the popular upscaling methods (DLSS, FSR, XeSS, ...) aren't integrated in all of the mainstream engines: Unity, Unreal, Godot... As far as I'm concerned, on the end of a game developer, that should be a single checkbox in the project options, or something similarly easy to choosing between 2X and 4X AA, both on the desktop and mobile, where supported by the hardware (or simpler generic upscaling, if nothing else suffices). The performance gains (at the expense of graphical fidelity, which may or may not be acceptable) would be staggering - you could quite literally squeeze out just enough performance to enjoy a game or an application even from older generation hardware for more modern titles.

jsheard 2y ago
Microsoft is pushing something along those lines with DirectSR, the idea is that game engines integrate one set of generic hooks which can then be routed to any of the vendor upscalers transparently. https://devblogs.microsoft.com/directx/directsr-preview/ https://devblogs.microsoft.com/directx/directsr-preview/
- KronisLV 2y ago
  If it's done well, then that sounds exactly like what we need! Wonder whether something similar will land in Vulkan too, at some point.
chrisjj 2y ago
Because devs find them far less attractive in practice than the sales pitch implies.
- keyringlight 2y ago
  I'd guess that a lot of devs it's not their main strength and their main concern is getting a working product out the door. If you're making "Bunnies go berserk" with the adventures of berserk rabbits as your main selling point, it seems unlikely that a smaller list of graphics fidelity options is going to dissuade anyone but the most hardcore graphics enthusiast from buying. All those options are also going to need time allocated for a graphics developer to tune to suit your game if the out-of-the-box defaults aren't great, and as it's an area still in progress upgrading APIs and revalidating for your game on a range of hardware. If you're doing multiplatform with consoles/mobile that's even more in the mix, and you need to consider what will reward the time spent on it.

LisaST 2y ago

[flagged]

chrisjj 2y ago

> Accuracy Super Resolution That would have been worth first checking with a native English speaker.

saagarjha 2y ago
What’s wrong with it
- chrisjj 2y ago
  For starters, accuracy is not an adjective.
deleted 2y ago
[deleted]

DeathArrow 2y ago

Nvidia has the hardware power to use AI for upscaling. How many tensor cores do ARM socs have?

cubefox 2y ago
Well, the term "tensor core" is only used by Nvidia, and Nvidia does make Arm SoCs (Tegra), which may help you find the answer.
caseyy 2y ago
Doesn't matter. ASR is FSR2-based. It's just upscaling and sharpening with a proprietary shader. No neural nets are involved.

miohtama 2y ago

Is this based on some acquisition ARM has done? Because the sector is very un-ARM’y.

enragedcacti 2y ago
It seems right in their wheelhouse imo. They've just taken all of the hard work AMD did in the space with FSR and optimized it for their GPU architecture. I'd be pretty worried if ARM didn't know how to write a performant shader for their own GPU.
- chrisjj 2y ago
  > I'd be pretty worried if ARM didn't know how to write a performant shader for their own GPU. We should probably worry why ARM feels it has to. If this was a great idea, game and game engine devs would have done it already.

asdff 2y ago

So tired of the dragon chasing with gaming graphics. The fruit they are chasing today is so low hanging yet demands such powerful hardware just to do something like render a beam of light through the woods. Meanwhile the game hardly looks better than crysis outside of that, especially when you are, you know, focused on playing the game and not pixel peeping. I wish developers put more effort into compatibility and cross platform support but I guess its not the developers making the engines after all, they are kind of beholden to them.

deleted 2y ago
[deleted]