19 ms·

Every time someone brings up Super-resolution, I like to pull up this hilarious example : https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-

Every time someone brings up Super-resolution, I like to pull up this hilarious example : https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-goslings-face-to-this-photo/ https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a... Super-resolution is only guessing. It's ok for art, not for critical tasks.

aktuel 6y ago
There are serious use cases for super resolution in medical imaging for example.
- sgerenser 6y ago
  I’d think medical imaging is exactly where you wouldn’t want to use this technique. If something isn’t clear on an X-ray, you don’t want to fill in the details by guessing like this software does.
Datenstrom 6y ago
Unfortunately someone is going to wrap up super-resolution for critical tasks and sell it likely causing many people harm or at least inconvenience. I have already tried to talk some companies out of using it for police/surveillance type work. People who do not understand the technology are determined to use it and someone is going to.
savant_penguin 6y ago
What I find interesting about that is that after seeing the face in the super-resolution image you can kinda see it in the original
jonplackett 6y ago
Every time someone brings up enhancing images I like to pull up this Red Dwarf clip: https://www.youtube.com/watch?v=2aINa6tg3fo https://www.youtube.com/watch?v=2aINa6tg3fo
- opo 6y ago
  That is great. I like to bring up this clip from Castle: https://www.youtube.com/watch?v=PaMdXjTn9rc https://www.youtube.com/watch?v=PaMdXjTn9rc
  - drewzero1 6y ago
    I really loved the way Castle played around with typical police drama tropes, though I got a little confused about what the show was trying to be/do in later seasons.
    - jonplackett 6y ago
      Haha. I’d never even heard of this show. Will be checking it out.
- 14 6y ago
  Such a good show and highly underrated and unknown by so many. What a great clip.
  - jonplackett 6y ago
    I love the gag at the end too. “Wouldn’t it have been easier to just look them up in the phone book?” Pure genius
- geoduck14 6y ago
  Lolol!
danShumway 6y ago
I wish Adobe would use a different name for this that made it more obvious what was happening, something like "detail fill" or "detail interpolation". I worry this is going to be a case where the marketing is at direct odds with public education efforts.
timthorn 6y ago
There are forms of super-resolution that certainly aren't guessing. For example, you can take a video of a subject and integrate over time, so that the motion of the subject over the sensor allows you to infer sub-pixel detail. https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResolutionSequences.pdf https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResoluti...
- Judgmentality 6y ago
  Turning a regular video into a super photo is different from turning a regular photo into a super photo.
- liuliu 6y ago
  They started their model with RAW format, so the model should encoded some interactions between red / blue / green light sensors and that can help generate genuine sub-pixel details. OTOH, this is machine learning, unless you specifically has some discriminators (just an idea) to counteract, you don't really know how much these are genuine sub-pixel details and how much are hallucinations.
greggturkington 6y ago
Is this single example of someone using different software (by Topaz Labs) relevant to this article specifically? Or just every article about enhancement?
ryanwhitney 6y ago
>Photographer Jomppe Vaarakallio has been a professional retoucher for 30 years… >To be clear, this isn’t a knock on the Gigapixel software. Vaarakallio tells PetaPixel that the software is “amazing” and he uses it all the time.
- danShumway 6y ago
  I don't think that it's a knock on the software, I think it's a knock on the common interpretation of what that software is doing. Professional photo retouching is art. It's okay to use Gigapixel for an artistic task, it's not OK to use it to enhance a photo that you're going to show to a jury. That's what GP means by 'critical': use cases where it matters whether or not the pixels being added map to an objective reality rather than an algorithmic guess about what would look good.
ska 6y ago
One of those annoying things is that the name "super resolution" stuck, here. Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Some modern photographic systems actually do this directly (piezo motors?) on the sensor. Of course this only works if what you are imaging is reasonably static over the time needed to take all the images. You can do an approximate version of this with video, with caveats because you don't control the motion. The key thing is, though, you actually have more data to work with. This idea ran in parallel with image processing people attempting to estimate higher resolution from a single image for a while, and unfortunately the terminology stuck in image processing also. Something like resolution extrapolation is probably better but that ship sailed ages ago.
- anon_tor_12345 6y ago
  Citation? I've read over 50 papers in MISR and SISR (I wrote a lit review) and I have never seen mention of actual hardware that would shift the imaging system; indeed such a system would have been my dissertation topic if I hadn't switched areas.
  - CarVac 6y ago
    Certain Hasselblad cameras have had this for a long time, and Pentax, Olympus, Panasonic, and Sony have it on various models, using the sensor shift image stabilization to implement it.
  - galago 6y ago
    I think what he's talking about is what Sony calls Pixel Shift Multi Shooting. Other camera manufacturers do it too. https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7rm3/en/index.html https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...
    - 6y ago
tshaddox 6y ago
Is there any proof that Ryan Gosling's face (or perhaps a photograph of Ryan Gosling's face) was in fact not there when the original photo was taken? :)
jonas21 6y ago
Lots of things are "only guessing." Auto color correction is only guessing. Unsharp mask is only guessing. Smart selection is only guessing. Content-aware fill is only guessing. They're still useful tools to have in your toolbox as a photographer or designer, even for critical tasks, and I don't really see how this is different. There may be certain failure cases, but everything has failure cases.
- jonplackett 6y ago
  The difference is that those kind of guesses are not inventive. It’s like how modern word-guessing while texting can have really weird results because the guess of what you meant to say has been turned into real words which creates new meaning. Where previously you’d just have had a few mangled words it’s now been corrected into proper words, often with an unfortunate sexual innuendo.
nojokes 6y ago
For me super resolution mean a combination of multiple lower resolution images to gain additional information and from that higher resolution. It is especially something that some cameras can do by deliberately doing sensor shifting. They should not call it super resolution or at best emulated super resolution or artificial super resolution.
oivey 6y ago
Guessing I think is too strong of a way to phrase what super resolution does. The broader concept of, for example, regularized solving of inverse problems is used widely in things like CT and MRI where the reconstructed imagery is used for analysis. The regularization is effectively the part you’re saying is guessing, but I would phrase it as enforcing assumptions about the data. Neural network-based approaches are similarly learning the distribution of the output data.
kthartic 6y ago
From the linked article: >you may want to uncheck detect faces… unless you want Ryan Gosling popping up all over the place. Sooo not really a case against super-resolution, just a funny result of having used the wrong settings
sorenjan 6y ago
> Super-resolution is only guessing. Machine learning is educated guessing based on previously seen data. As mentioned by others there are ways to do super resolution that only uses the data available. I can't think of any that can upscale a single image, although I have vague memories of having seen something about using moiré patterns to infer the higher resolution texture of some features.
ravi-delia 6y ago
I feel like it's also ok for critical tasks if you're willing to accept that it isn't perfect. If all you have is a grainy photo, you'll only be able to make guesses yourself; why not have a superhuman guess too? (Because the people putting it to use would be morons about it, I know, let me dream)
rom1v 6y ago
Or https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801
natemo 6y ago
Nitpicky, but I think "It's okay for x, not for y" when describing nascent technology is a bit shortsighted. Who knows how this evolves and what new applications people may devise? For today, I agree: it's just art.
porphyra 6y ago
A personal favourite is white Obama: https://www.theverge.com/21298762/face-depixelizer-ai-machine-learning-tool-pulse-stylegan-obama-bias https://www.theverge.com/21298762/face-depixelizer-ai-machin...
cycrutchfield 6y ago
Nobody has billed it as anything more than just guessing. In the literature, it is frequently mentioned as “perceptually plausible” upscaling.
dirtyid 6y ago
Example of critical tasks?
ineedasername 6y ago
That reminds me of the old CSI episodes where they'd have grainy footage from a CCTV running around SIF/240p, and the lead investigator would say "Enhance... Enhance... Enhance... There!" And the face of the killer would be clear as day from inside a moving car a block away.

aktuel 6y ago

There are serious use cases for super resolution in medical imaging for example.

sgerenser 6y ago
I’d think medical imaging is exactly where you wouldn’t want to use this technique. If something isn’t clear on an X-ray, you don’t want to fill in the details by guessing like this software does.

Datenstrom 6y ago

Unfortunately someone is going to wrap up super-resolution for critical tasks and sell it likely causing many people harm or at least inconvenience. I have already tried to talk some companies out of using it for police/surveillance type work. People who do not understand the technology are determined to use it and someone is going to.

savant_penguin 6y ago

What I find interesting about that is that after seeing the face in the super-resolution image you can kinda see it in the original

jonplackett 6y ago

Every time someone brings up enhancing images I like to pull up this Red Dwarf clip: https://www.youtube.com/watch?v=2aINa6tg3fo https://www.youtube.com/watch?v=2aINa6tg3fo

opo 6y ago
That is great. I like to bring up this clip from Castle: https://www.youtube.com/watch?v=PaMdXjTn9rc https://www.youtube.com/watch?v=PaMdXjTn9rc
- drewzero1 6y ago
  I really loved the way Castle played around with typical police drama tropes, though I got a little confused about what the show was trying to be/do in later seasons.
  - jonplackett 6y ago
    Haha. I’d never even heard of this show. Will be checking it out.
14 6y ago
Such a good show and highly underrated and unknown by so many. What a great clip.
- jonplackett 6y ago
  I love the gag at the end too. “Wouldn’t it have been easier to just look them up in the phone book?” Pure genius
geoduck14 6y ago
Lolol!

danShumway 6y ago

I wish Adobe would use a different name for this that made it more obvious what was happening, something like "detail fill" or "detail interpolation". I worry this is going to be a case where the marketing is at direct odds with public education efforts.

timthorn 6y ago

There are forms of super-resolution that certainly aren't guessing. For example, you can take a video of a subject and integrate over time, so that the motion of the subject over the sensor allows you to infer sub-pixel detail. https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResolutionSequences.pdf https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResoluti...

Judgmentality 6y ago
Turning a regular video into a super photo is different from turning a regular photo into a super photo.
liuliu 6y ago
They started their model with RAW format, so the model should encoded some interactions between red / blue / green light sensors and that can help generate genuine sub-pixel details. OTOH, this is machine learning, unless you specifically has some discriminators (just an idea) to counteract, you don't really know how much these are genuine sub-pixel details and how much are hallucinations.

greggturkington 6y ago

Is this single example of someone using different software (by Topaz Labs) relevant to this article specifically? Or just every article about enhancement?

ryanwhitney 6y ago

>Photographer Jomppe Vaarakallio has been a professional retoucher for 30 years… >To be clear, this isn’t a knock on the Gigapixel software. Vaarakallio tells PetaPixel that the software is “amazing” and he uses it all the time.

danShumway 6y ago
I don't think that it's a knock on the software, I think it's a knock on the common interpretation of what that software is doing. Professional photo retouching is art. It's okay to use Gigapixel for an artistic task, it's not OK to use it to enhance a photo that you're going to show to a jury. That's what GP means by 'critical': use cases where it matters whether or not the pixels being added map to an objective reality rather than an algorithmic guess about what would look good.

ska 6y ago

One of those annoying things is that the name "super resolution" stuck, here. Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Some modern photographic systems actually do this directly (piezo motors?) on the sensor. Of course this only works if what you are imaging is reasonably static over the time needed to take all the images. You can do an approximate version of this with video, with caveats because you don't control the motion. The key thing is, though, you actually have more data to work with. This idea ran in parallel with image processing people attempting to estimate higher resolution from a single image for a while, and unfortunately the terminology stuck in image processing also. Something like resolution extrapolation is probably better but that ship sailed ages ago.

anon_tor_12345 6y ago
Citation? I've read over 50 papers in MISR and SISR (I wrote a lit review) and I have never seen mention of actual hardware that would shift the imaging system; indeed such a system would have been my dissertation topic if I hadn't switched areas.
- CarVac 6y ago
  Certain Hasselblad cameras have had this for a long time, and Pentax, Olympus, Panasonic, and Sony have it on various models, using the sensor shift image stabilization to implement it.
- galago 6y ago
  I think what he's talking about is what Sony calls Pixel Shift Multi Shooting. Other camera manufacturers do it too. https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7rm3/en/index.html https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...
  - 6y ago

tshaddox 6y ago

Is there any proof that Ryan Gosling's face (or perhaps a photograph of Ryan Gosling's face) was in fact not there when the original photo was taken? :)

jonas21 6y ago

Lots of things are "only guessing." Auto color correction is only guessing. Unsharp mask is only guessing. Smart selection is only guessing. Content-aware fill is only guessing. They're still useful tools to have in your toolbox as a photographer or designer, even for critical tasks, and I don't really see how this is different. There may be certain failure cases, but everything has failure cases.

jonplackett 6y ago
The difference is that those kind of guesses are not inventive. It’s like how modern word-guessing while texting can have really weird results because the guess of what you meant to say has been turned into real words which creates new meaning. Where previously you’d just have had a few mangled words it’s now been corrected into proper words, often with an unfortunate sexual innuendo.

nojokes 6y ago

For me super resolution mean a combination of multiple lower resolution images to gain additional information and from that higher resolution. It is especially something that some cameras can do by deliberately doing sensor shifting. They should not call it super resolution or at best emulated super resolution or artificial super resolution.

oivey 6y ago

Guessing I think is too strong of a way to phrase what super resolution does. The broader concept of, for example, regularized solving of inverse problems is used widely in things like CT and MRI where the reconstructed imagery is used for analysis. The regularization is effectively the part you’re saying is guessing, but I would phrase it as enforcing assumptions about the data. Neural network-based approaches are similarly learning the distribution of the output data.

kthartic 6y ago

From the linked article: >you may want to uncheck detect faces… unless you want Ryan Gosling popping up all over the place. Sooo not really a case against super-resolution, just a funny result of having used the wrong settings

sorenjan 6y ago

> Super-resolution is only guessing. Machine learning is educated guessing based on previously seen data. As mentioned by others there are ways to do super resolution that only uses the data available. I can't think of any that can upscale a single image, although I have vague memories of having seen something about using moiré patterns to infer the higher resolution texture of some features.

ravi-delia 6y ago

I feel like it's also ok for critical tasks if you're willing to accept that it isn't perfect. If all you have is a grainy photo, you'll only be able to make guesses yourself; why not have a superhuman guess too? (Because the people putting it to use would be morons about it, I know, let me dream)

rom1v 6y ago

Or https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801

natemo 6y ago

Nitpicky, but I think "It's okay for x, not for y" when describing nascent technology is a bit shortsighted. Who knows how this evolves and what new applications people may devise? For today, I agree: it's just art.

porphyra 6y ago

A personal favourite is white Obama: https://www.theverge.com/21298762/face-depixelizer-ai-machine-learning-tool-pulse-stylegan-obama-bias https://www.theverge.com/21298762/face-depixelizer-ai-machin...

cycrutchfield 6y ago

Nobody has billed it as anything more than just guessing. In the literature, it is frequently mentioned as “perceptually plausible” upscaling.

dirtyid 6y ago

Example of critical tasks?

ineedasername 6y ago

That reminds me of the old CSI episodes where they'd have grainy footage from a CCTV running around SIF/240p, and the lead investigator would say "Enhance... Enhance... Enhance... There!" And the face of the killer would be clear as day from inside a moving car a block away.