3 ms·

The article lists two methods: * Spatial - Inferring data from nearby pixels in a single frame (?) * Temporal - Inferring data from a point on previous and nex

by rendaw 2y ago

The article lists two methods: * Spatial - Inferring data from nearby pixels in a single frame (?) * Temporal - Inferring data from a point on previous and next frames I guess this is super resolution in the statistical sense, rather than "AI super resolution" which would infer data from similar data in other photos/videos. I'm surprised, I thought this was a dead area for research and AI super resolution had fully supplanted it. Are there any open source implementations of this, ex: for photography? I was digging into this a few years ago and all I could find was the commercial "PhotoAcute" which is basically dead but I somehow managed to get a key for and it... barely works.

GrantMoyer 2y ago
Temporal super resalution needs to accumulate data from multiple frames to work. Essentially, each frame renders only a subset of pixels, but crucially each frame renders a different subset, so you can combine information over a few frames to compute what should be at each pixel. There're some complications to account for things moving around the scene, but that's the basic idea. So there's no way the technique or similar techniques can be used to upscale a single photo in isolation. However, conceptually it's similar to capturing a long exposure image of dark scene, like the night sky, to get a low-noise, brighter photo. Temporal super resolution accumulates information from rendered pixels over time, while a long exposure photo accumulates information from captured photons over time.
- rendaw 2y ago
  PhotoAcute uses several photos with some jitter, like you might get from burst shots on a phone. I'd expect other software (and possibly phone upscalers transparently) might do this too.
jlouis 2y ago
A key component is that when a game generates pixels, it also generates other data. Such as a Z-buffer for pixel depth and motion vectors (for motion blur, among other things). Feeding this to the upscaling algorithm vastly improves image quality. The larger problem to solve with these upscalers are temporal artifacts when using the above information. This lead to a number of heuristics, but if you train a neural network to do the work, it tends to perform better than the heuristics. There's still a ton of research going into non-ML solutions however, because the current console generation doesn't have AI-acceleration available, nor does a lot of hardware out there. So there's some longevity to be had by doing a good job for all that hardware out there.
- rendaw 2y ago
  So the use case is on old gaming hardware, upscaling even older games?
cubefox 2y ago
> I'm surprised, I thought this was a dead area for research and AI super resolution had fully supplanted it. There is no open source solution of the AI variant. There are currently only DLSS (Nvidia) and XeSS (Intel), and both are proprietary. The one open source solution is AMD's FSR (MIT license), which doesn't use machine learning. So naturally Arm and Apple build on FSR for their products instead of developing an AI solution from scratch.

by rendaw 2y ago

GrantMoyer 2y ago

Temporal super resalution needs to accumulate data from multiple frames to work. Essentially, each frame renders only a subset of pixels, but crucially each frame renders a different subset, so you can combine information over a few frames to compute what should be at each pixel. There're some complications to account for things moving around the scene, but that's the basic idea. So there's no way the technique or similar techniques can be used to upscale a single photo in isolation. However, conceptually it's similar to capturing a long exposure image of dark scene, like the night sky, to get a low-noise, brighter photo. Temporal super resolution accumulates information from rendered pixels over time, while a long exposure photo accumulates information from captured photons over time.

rendaw 2y ago
PhotoAcute uses several photos with some jitter, like you might get from burst shots on a phone. I'd expect other software (and possibly phone upscalers transparently) might do this too.

jlouis 2y ago

A key component is that when a game generates pixels, it also generates other data. Such as a Z-buffer for pixel depth and motion vectors (for motion blur, among other things). Feeding this to the upscaling algorithm vastly improves image quality. The larger problem to solve with these upscalers are temporal artifacts when using the above information. This lead to a number of heuristics, but if you train a neural network to do the work, it tends to perform better than the heuristics. There's still a ton of research going into non-ML solutions however, because the current console generation doesn't have AI-acceleration available, nor does a lot of hardware out there. So there's some longevity to be had by doing a good job for all that hardware out there.

rendaw 2y ago
So the use case is on old gaming hardware, upscaling even older games?

cubefox 2y ago

> I'm surprised, I thought this was a dead area for research and AI super resolution had fully supplanted it. There is no open source solution of the AI variant. There are currently only DLSS (Nvidia) and XeSS (Intel), and both are proprietary. The one open source solution is AMD's FSR (MIT license), which doesn't use machine learning. So naturally Arm and Apple build on FSR for their products instead of developing an AI solution from scratch.