7 ms·

A smooth and sharp image interpolation you probably haven't heard of

102 points by okaleniuk 2y ago

esafak 2y ago
There was a shareware site popular with the photo enthusiasts twenty years ago that implemented numerous resizing algorithms and allowed you to compare them visually. Unfortunately I can not recall its name.
ImHereToVote 2y ago
Why aren't the results of different interpolation methods shown in the website?
- mtekman 2y ago
  probably because the visual differences are so slight that there is no benefit in showing them to someone who isn't deeply entrenched in the topic
  - ImHereToVote 2y ago
    I think that would be me. Or anyone who would be interested in viewing this website. Would anyone not interested in this topic not want to ever see the differences?
    - szundi 2y ago
      Agreed
- okaleniuk 2y ago
  Good point. Why not? I'll add them.
  - WithinReason 2y ago
    Don't forget Smoothstep: https://en.wikipedia.org/wiki/Smoothstep https://en.wikipedia.org/wiki/Smoothstep
  - ImHereToVote 2y ago
    Thank you very much.
roger_ 2y ago
Sounds like this: https://en.m.wikipedia.org/wiki/Inverse_distance_weighting https://en.m.wikipedia.org/wiki/Inverse_distance_weighting
- ur-whale 2y ago
  I note that the "example 1 on one dimension" diagram in the link you provided produces a very strange curve, typically not the one you'd draw intuitively by hand.
Asooka 2y ago
I am a bit confused how the formula is different from a simple lerp. k(x) = 1/x y_i * k(x - x_i) + y_i+1 * k(x_i+1 - x) F = --------------------------------------- = k(x - x_i) + k(x_i+1 - x) y_i / (x - x_i) + y_i+1 / (x_i+1 - x) = --------------------------------------- = 1 / (x - x_i) + 1 / (x_i+1 - x) y_i * (x_i+1 - x) / A + y_i+1 * (x - x_i) / A = --------------------------------------------- (x_i+1 - x) / A + (x - x_i) / A where A = (x - x_i) * (x_i+1 - x) scaling to x_i+1 - x_i = 1, and using a = (x - x_i) F = (1 - a) * y_i + a * y_i+1 The only time this wouldn't hold is when A = 0, but then F is not defined anyway. However, the graph of F shown is decidedly not that of the lerp function.
- okaleniuk 2y ago
  Oh, it seems that a square fell off. It should have been 1/x^2. 1/x works fine for bitmaps, but in 1-dimensional case 1/x indeed turns the whole thing into linear interpolation, you're right.
- okaleniuk 2y ago
  Yep, there is a square in the source: // interpolation/extrapolation with no basis functions var weight2d1 = 2; const wf2d1 = function(x) {return 1./Math.pow(x, weight2d1);}
GrantMoyer 2y ago
With k = 1/x, the inverse weight interpolation is equivalent to bilinear. Perhaps it's better to show the result for k = 1/x^2. Neat aside, k = 1/x^∞ seems to be equivalent to nearest neighbor. (please excuse the abuse of notation)
- mark-r 2y ago
  Bilinear isn't 1/x, it's 1-x.
  - GrantMoyer 2y ago
    Right, but the more complicated expression ends up simplifying to the bilinear expression.
- MereInterest 2y ago
  The abuse of notation is a useful one, as it also captures the sense that as the exponent approaches infinity, the interpolation approaches nearest neighbor interpolation. This seems similar to the “max norm”, described in the wiki link below, which also uses infinity to describe the relationship with other distance formulas. https://en.wikipedia.org/wiki/Uniform_norm https://en.wikipedia.org/wiki/Uniform_norm
- lloeki 2y ago
  Could be nice if there were a log-scale slider to interactively vary the exponent from 1 to ∞-ish and see how that affects the frame-cropped preview.
deadbabe 2y ago
AI based interpolation will soon make these mathematical interpolation methods obsolete right?
- NayamAmarshe 2y ago
  Yes, with Upscayl, we’re already doing that (https://upscayl.org https://upscayl.org). It’s not as fast yet but it’s extremely good. Good enough that you could print a low resolution image after upscaling.
  - deadbabe 2y ago
    Thank you for everything you do!!
- amelius 2y ago
  If you prefer hands with six fingers ...
- deleted 2y ago
  [deleted]
- edflsafoiewq 2y ago
  In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. You can't use them for stuff like scaling an <img> in a browser. Incidentally, the way I always thought it would work is the huge and slow AI would be used to discover a tiny algorithm that just so happens to be an amazing image rescaler. Instead we just ship the whole AI. Anybody know of any work on the former strategy?
  - eru 2y ago
    > In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. Aren't games using these for real time upscaling many dozen times a second already?
- krapp 2y ago
  No, why would they? What would you expect an "AI" to do besides somehow implement an interpolation algorithm? The math is unavoidable.
Dwedit 2y ago
My go-to for any kind of very fast image upscaling is the SuperXBR algorithm. It's fast, a pixel shader implementation runs at over 60FPS on Skylake IGP. SuperXBR is in the family of Edge-Directed interpolation algorithms, so diagonal or curved lines will properly maintain their shape when upscaled, and not produce "staircase" artifacts that you see with other classical interpolation. It does not improve the sharpness though, images will still look blurry after being upscaled, just like when you perform bilinear upscaling. But if you are doing any kind of non-realtime image upscaling today, you'd want to use the modern AI algorithms, such as Waifu2x or ESRGAN.
- amelius 2y ago
  Ok. How can I apt-get install it? PS: is it me or has it become more difficult to find cool stuff in your typical Linux distro?
  - Dwedit 2y ago
    It's not a standalone program, it's usually a set of pixel shaders that can be integrated into another program. I once made an image viewer that would apply the effect, but it's not publicly released. There is an extremely roundabout way to see the effect in action on a Linux system using a package you can apt-get. It involves installing RetroArch, starting the "Image Viewer" core, loading an image, loading the shader preset named "super-xbr-2p.glslp", changing the number of shaders from 3 to 2 to exclude the "custom Jinc resharper" step and selecting Apply, then scaling the window to be exactly 2x the size of the original image. You probably don't want to do this. But if you really want a standalone program, I could build a Windows .NET program that will apply the shader to images you paste in from the clipboard.
    - amelius 2y ago
      That is very kind of you. But my statement was meant to be more general. I just have no time to dive into all the inevitable configure and compile issues anymore and am hoping that my Linux distro will catch up some day.
- deleted 2y ago
jasonjmcghee 2y ago
I found the original "pixelated" image to look higher resolution / crisper / more detailed. Really at any size I tried, I found the "upscale" to look blurry / low quality. Am I an outlier here?
- chefandy 2y ago
  There's no universal best option, here. Even two different pictures of the same thing at slightly different angles might look best with different upscaling algorithms. There are also variants on the smoother one to favor certain kinds of transitions. Really, though, all of those options suck. AI-powered options in Photoshop (much more controllable and reliable than SD or the like) actually invent details, so if provenance or accuracy is not important, (as it would be in a historic artifact, legal document, piece of visual art, or unique artisinal product,) that's always a possibility, but most projects have too many images to make tweaking each one individually viable. Except in extremely rare situations, I flat-out refuse to work with low-quality source images in my design jobs. They can either give me a proper source image, pay me to rebuild it in a vector program if it's a logo or similar, or go down to the local sign shop and get someone to "design" something for them using a template. Most people end up having a higher quality copy around and just didn't know the difference between that and the screenshot they took of it, or the thumbnail they downloaded. At 300DPI with 4 colors, those flaws are glaringly obvious compared to screen resolution and sRGB color representation.
- dheera 2y ago
  I think at this point diffusion model super-resolution out-beats everything else.
- eddd-ddde 2y ago
  You are right. It looks more pixelated, but better quality at the same time somehow?
Dylan16807 2y ago
When I think of "smooth and sharp" my mind goes to, uh, I guess "sharp bilinear" is the most common term? It's like nearest neighbor interpolation but with less artifacting when the zoom is not an integer. The middle of each source pixel becomes a solid color block, and then bilinear filtering is applied to the single-pixel-wide seams between each block. It's equivalent to doing a nearest neighbor upscale to floor(scale), then finishing with bilinear.

102 points by okaleniuk 2y ago

There was a shareware site popular with the photo enthusiasts twenty years ago that implemented numerous resizing algorithms and allowed you to compare them visually. Unfortunately I can not recall its name.

ImHereToVote 2y ago

Why aren't the results of different interpolation methods shown in the website?

mtekman 2y ago
probably because the visual differences are so slight that there is no benefit in showing them to someone who isn't deeply entrenched in the topic
- ImHereToVote 2y ago
  I think that would be me. Or anyone who would be interested in viewing this website. Would anyone not interested in this topic not want to ever see the differences?
  - szundi 2y ago
    Agreed
okaleniuk 2y ago
Good point. Why not? I'll add them.
- WithinReason 2y ago
  Don't forget Smoothstep: https://en.wikipedia.org/wiki/Smoothstep https://en.wikipedia.org/wiki/Smoothstep
- ImHereToVote 2y ago
  Thank you very much.

roger_ 2y ago

Sounds like this: https://en.m.wikipedia.org/wiki/Inverse_distance_weighting https://en.m.wikipedia.org/wiki/Inverse_distance_weighting

ur-whale 2y ago
I note that the "example 1 on one dimension" diagram in the link you provided produces a very strange curve, typically not the one you'd draw intuitively by hand.

Asooka 2y ago

I am a bit confused how the formula is different from a simple lerp. k(x) = 1/x y_i * k(x - x_i) + y_i+1 * k(x_i+1 - x) F = --------------------------------------- = k(x - x_i) + k(x_i+1 - x) y_i / (x - x_i) + y_i+1 / (x_i+1 - x) = --------------------------------------- = 1 / (x - x_i) + 1 / (x_i+1 - x) y_i * (x_i+1 - x) / A + y_i+1 * (x - x_i) / A = --------------------------------------------- (x_i+1 - x) / A + (x - x_i) / A where A = (x - x_i) * (x_i+1 - x) scaling to x_i+1 - x_i = 1, and using a = (x - x_i) F = (1 - a) * y_i + a * y_i+1 The only time this wouldn't hold is when A = 0, but then F is not defined anyway. However, the graph of F shown is decidedly not that of the lerp function.

okaleniuk 2y ago
Oh, it seems that a square fell off. It should have been 1/x^2. 1/x works fine for bitmaps, but in 1-dimensional case 1/x indeed turns the whole thing into linear interpolation, you're right.
okaleniuk 2y ago
Yep, there is a square in the source: // interpolation/extrapolation with no basis functions var weight2d1 = 2; const wf2d1 = function(x) {return 1./Math.pow(x, weight2d1);}

GrantMoyer 2y ago

With k = 1/x, the inverse weight interpolation is equivalent to bilinear. Perhaps it's better to show the result for k = 1/x^2. Neat aside, k = 1/x^∞ seems to be equivalent to nearest neighbor. (please excuse the abuse of notation)

mark-r 2y ago
Bilinear isn't 1/x, it's 1-x.
- GrantMoyer 2y ago
  Right, but the more complicated expression ends up simplifying to the bilinear expression.
MereInterest 2y ago
The abuse of notation is a useful one, as it also captures the sense that as the exponent approaches infinity, the interpolation approaches nearest neighbor interpolation. This seems similar to the “max norm”, described in the wiki link below, which also uses infinity to describe the relationship with other distance formulas. https://en.wikipedia.org/wiki/Uniform_norm https://en.wikipedia.org/wiki/Uniform_norm
lloeki 2y ago
Could be nice if there were a log-scale slider to interactively vary the exponent from 1 to ∞-ish and see how that affects the frame-cropped preview.

deadbabe 2y ago

AI based interpolation will soon make these mathematical interpolation methods obsolete right?

NayamAmarshe 2y ago
Yes, with Upscayl, we’re already doing that (https://upscayl.org https://upscayl.org). It’s not as fast yet but it’s extremely good. Good enough that you could print a low resolution image after upscaling.
- deadbabe 2y ago
  Thank you for everything you do!!
amelius 2y ago
If you prefer hands with six fingers ...
deleted 2y ago
[deleted]
edflsafoiewq 2y ago
In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. You can't use them for stuff like scaling an <img> in a browser. Incidentally, the way I always thought it would work is the huge and slow AI would be used to discover a tiny algorithm that just so happens to be an amazing image rescaler. Instead we just ship the whole AI. Anybody know of any work on the former strategy?
- eru 2y ago
  > In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. Aren't games using these for real time upscaling many dozen times a second already?
krapp 2y ago
No, why would they? What would you expect an "AI" to do besides somehow implement an interpolation algorithm? The math is unavoidable.

Dwedit 2y ago

My go-to for any kind of very fast image upscaling is the SuperXBR algorithm. It's fast, a pixel shader implementation runs at over 60FPS on Skylake IGP. SuperXBR is in the family of Edge-Directed interpolation algorithms, so diagonal or curved lines will properly maintain their shape when upscaled, and not produce "staircase" artifacts that you see with other classical interpolation. It does not improve the sharpness though, images will still look blurry after being upscaled, just like when you perform bilinear upscaling. But if you are doing any kind of non-realtime image upscaling today, you'd want to use the modern AI algorithms, such as Waifu2x or ESRGAN.

amelius 2y ago
Ok. How can I apt-get install it? PS: is it me or has it become more difficult to find cool stuff in your typical Linux distro?
- Dwedit 2y ago
  It's not a standalone program, it's usually a set of pixel shaders that can be integrated into another program. I once made an image viewer that would apply the effect, but it's not publicly released. There is an extremely roundabout way to see the effect in action on a Linux system using a package you can apt-get. It involves installing RetroArch, starting the "Image Viewer" core, loading an image, loading the shader preset named "super-xbr-2p.glslp", changing the number of shaders from 3 to 2 to exclude the "custom Jinc resharper" step and selecting Apply, then scaling the window to be exactly 2x the size of the original image. You probably don't want to do this. But if you really want a standalone program, I could build a Windows .NET program that will apply the shader to images you paste in from the clipboard.
  - amelius 2y ago
    That is very kind of you. But my statement was meant to be more general. I just have no time to dive into all the inevitable configure and compile issues anymore and am hoping that my Linux distro will catch up some day.
deleted 2y ago

jasonjmcghee 2y ago

I found the original "pixelated" image to look higher resolution / crisper / more detailed. Really at any size I tried, I found the "upscale" to look blurry / low quality. Am I an outlier here?

chefandy 2y ago
There's no universal best option, here. Even two different pictures of the same thing at slightly different angles might look best with different upscaling algorithms. There are also variants on the smoother one to favor certain kinds of transitions. Really, though, all of those options suck. AI-powered options in Photoshop (much more controllable and reliable than SD or the like) actually invent details, so if provenance or accuracy is not important, (as it would be in a historic artifact, legal document, piece of visual art, or unique artisinal product,) that's always a possibility, but most projects have too many images to make tweaking each one individually viable. Except in extremely rare situations, I flat-out refuse to work with low-quality source images in my design jobs. They can either give me a proper source image, pay me to rebuild it in a vector program if it's a logo or similar, or go down to the local sign shop and get someone to "design" something for them using a template. Most people end up having a higher quality copy around and just didn't know the difference between that and the screenshot they took of it, or the thumbnail they downloaded. At 300DPI with 4 colors, those flaws are glaringly obvious compared to screen resolution and sRGB color representation.
dheera 2y ago
I think at this point diffusion model super-resolution out-beats everything else.
eddd-ddde 2y ago
You are right. It looks more pixelated, but better quality at the same time somehow?

Dylan16807 2y ago

When I think of "smooth and sharp" my mind goes to, uh, I guess "sharp bilinear" is the most common term? It's like nearest neighbor interpolation but with less artifacting when the zoom is not an integer. The middle of each source pixel becomes a solid color block, and then bilinear filtering is applied to the single-pixel-wide seams between each block. It's equivalent to doing a nearest neighbor upscale to floor(scale), then finishing with bilinear.