3 ms·

Bilinear down/upsampling, aligning pixel grids, and that infamous GPU half pixel (2021)

136 points by fanf2 2y ago

deleted 2y ago
[deleted]
Demiurge 2y ago
what would it look like with nearest, or gaussian filters? Would be interesting to see the same sample graphs with different kernels.
calibas 2y ago
Bilinear is nice for performance, but if you're concerned about image quality you generally want to use bicubic resampling instead.
- jeffreygoesto 2y ago
  Or Lanczos resampling.
  - magicalhippo 2y ago
    Lanczos if you want sharpness above all else. There's also the well-known Mitchell-Netravali filters[1], based on piece-wise cubic splines and with two tunable paramerers. [1]: https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_filters https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_fil...
    - Dwedit 2y ago
      Whenever I want to upscale, I use an Edge-Directed algorithm, such as SuperXBR. Diagonal lines stay diagonal and don't turn into blurry boxes.
    - jorvi 2y ago
      I don't know the fine details of it but afaik FSR <3.0 is Lanczos with a few tricks to both speed it up and make the visuals more stable.
    - mark-r 2y ago
      There's a pair of parameters that make Mitchell-Netravali nearly identical to Lanczos-3, but my grep-fu is failing me at the moment. I'll try to find it later.
      - mark-r 2y ago
        Found it. The parameters are B=0 C=0.5458 in the Mitchell-Netravali formula to emulate Lanczos-3. It's not a perfect match, but I dare you to see the difference.
        magicalhippo 2y ago
        Cool, will check it out.
  - daneel_w 2y ago
    For upscaling, sure. For downscaling it has a tendency to both amplify and distort (rather than preserve) detail and sharpness, "imagining" detail that never existed, leaving you with a worse result. Bicubic and sometimes even bilinear does a better job, truer to the original, when you're scaling down.
    - mark-r 2y ago
      I think what you're seeing is aliasing. You need to filter for the Nyquist frequency of whichever is smallest, the input or the output. When upscaling the input is always smallest so it's easy. But when downscaling you need to adjust the width - for a 0.5x scale you need a filter 2x the size. I have my own hand coded Lanczos-5 that I use for all my scaling needs, and it's perfect.
waynecochran 2y ago
Great article. That half pixel offset is crucial for understanding proper rasterization of detailed 3D meshes. No seams, no holes, no overlaps on shared triangle edges rely on exactly where the pixel centers are. This carries over to image processing and sampling. There are a few of us that lose sleep over these tiny details -- I am happy when I see kindred spirits with the same attention to detail.
- deleted 2y ago
  [deleted]
lcnielsen 2y ago
I spent a substantial part of my Ph.D. dealing with this stuff, existing frameworks and algorithms were not suitable for our exact use-case and implementation. I had more bugs/artefacts than I'd like to admit due to some off-by-less-than-0.5 in some particular direction in some particular frame...
Const-me 2y ago
Good article, but IMO none of the discussed downsampling methods are actually ideal. The ideal is trilinear sampling instead of bilinear. Easy to do with GPU APIs, in D3D11 it’s ID3D11DeviceContext.GenerateMips API call to generate full set of mip levels of the input texture, then generate each output pixel with trilinear sampler. When doing non-uniform downsampling, use anisotropic sampler instead of trilinear. Not sure high level image processing libraries like the mentioned numpy, PIL or TensorFlow are doing anything like that, though.
- bartwr 2y ago
  Original author here and it's been a while since I have read such word salad nonsense, sorry. Why people who have no idea or expertise comment on articles? GenerateMips API constructs a mip chain by using box/bilinear (equivalent for factors of two) log N times. Trilinear interpolates across three dimensions, such as 3D textures or mip chains. It is not a method for downsampling, but a method for filtering that interpolates two bilinear results, such as two bilinear filters of mip levels that were generated with "some" downsampling filter (which can be anything from box to Lanczos). Anisotropic is a hybrid between trilinear across a smaller interpolation axis under perspective projection of a 3D asset and multiple taps along the longer axis. (More expensive)
  - Const-me 2y ago
    > Trilinear interpolates across three dimensions, such as 3D textures or mip chains I meant trilinear interpolation across the mip chain. > generated with "some" downsampling filter (which can be anything from box to Lanczos) In practice, whichever method is implemented in user-mode half of GPU drivers is pretty good. > It is not a method for downsampling No, but it can be applied for downsampling as well. > under perspective projection of a 3D asset Texture samplers don’t know or care about projections. They only take 2D texture coordinates, and screen-space derivatives of these. This is precisely what enables to use texture samplers to downsample images. The only caveat, if you do that by dispatching a compute shader as opposed to rendering a full-screen triangle, you’ll have to supply screen-space derivatives manually in the arguments of Texture2D.SampleGrad method. When doing non-uniform downsampling without perspective projections, these ddx/ddy numbers are the same for all output pixels, and are trivial to compute on CPU before dispatching the shader. > More expensive On modern computers, the performance overhead of anisotropic sampling compared to trilinear is just barely measurable.
qwertox 2y ago
On a similar note, yesterday I stumbled across a 4-days old video related to dithering which is really worth watching. "Surface-Stable Fractal Dithering" [0] by "Rune Skovbo Johansen, indie game developer and procedural generation enthusiast" [0] https://www.youtube.com/watch?v=HPqGaIMVuLs https://www.youtube.com/watch?v=HPqGaIMVuLs
- magicalhippo 2y ago
  It was also covered here 4 days ago[1], again I might add. Very cool effect, and one of my absolute favorite games. [1]: https://news.ycombinator.com/item?id=42808889 https://news.ycombinator.com/item?id=42808889
johncostella 2y ago
I'm biased, but Magic Kernel Sharp does a good job for many applications. :) https://johncostella.com/magic/ https://johncostella.com/magic/

136 points by fanf2 2y ago

deleted 2y ago

[deleted]

what would it look like with nearest, or gaussian filters? Would be interesting to see the same sample graphs with different kernels.

calibas 2y ago

Bilinear is nice for performance, but if you're concerned about image quality you generally want to use bicubic resampling instead.

jeffreygoesto 2y ago
Or Lanczos resampling.
- magicalhippo 2y ago
  Lanczos if you want sharpness above all else. There's also the well-known Mitchell-Netravali filters[1], based on piece-wise cubic splines and with two tunable paramerers. [1]: https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_filters https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_fil...
  - Dwedit 2y ago
    Whenever I want to upscale, I use an Edge-Directed algorithm, such as SuperXBR. Diagonal lines stay diagonal and don't turn into blurry boxes.
  - jorvi 2y ago
    I don't know the fine details of it but afaik FSR <3.0 is Lanczos with a few tricks to both speed it up and make the visuals more stable.
  - mark-r 2y ago
    There's a pair of parameters that make Mitchell-Netravali nearly identical to Lanczos-3, but my grep-fu is failing me at the moment. I'll try to find it later.
    - mark-r 2y ago
      Found it. The parameters are B=0 C=0.5458 in the Mitchell-Netravali formula to emulate Lanczos-3. It's not a perfect match, but I dare you to see the difference.
      - magicalhippo 2y ago
        Cool, will check it out.
- daneel_w 2y ago
  For upscaling, sure. For downscaling it has a tendency to both amplify and distort (rather than preserve) detail and sharpness, "imagining" detail that never existed, leaving you with a worse result. Bicubic and sometimes even bilinear does a better job, truer to the original, when you're scaling down.
  - mark-r 2y ago
    I think what you're seeing is aliasing. You need to filter for the Nyquist frequency of whichever is smallest, the input or the output. When upscaling the input is always smallest so it's easy. But when downscaling you need to adjust the width - for a 0.5x scale you need a filter 2x the size. I have my own hand coded Lanczos-5 that I use for all my scaling needs, and it's perfect.

waynecochran 2y ago

Great article. That half pixel offset is crucial for understanding proper rasterization of detailed 3D meshes. No seams, no holes, no overlaps on shared triangle edges rely on exactly where the pixel centers are. This carries over to image processing and sampling. There are a few of us that lose sleep over these tiny details -- I am happy when I see kindred spirits with the same attention to detail.

deleted 2y ago
[deleted]

lcnielsen 2y ago

I spent a substantial part of my Ph.D. dealing with this stuff, existing frameworks and algorithms were not suitable for our exact use-case and implementation. I had more bugs/artefacts than I'd like to admit due to some off-by-less-than-0.5 in some particular direction in some particular frame...

Const-me 2y ago

Good article, but IMO none of the discussed downsampling methods are actually ideal. The ideal is trilinear sampling instead of bilinear. Easy to do with GPU APIs, in D3D11 it’s ID3D11DeviceContext.GenerateMips API call to generate full set of mip levels of the input texture, then generate each output pixel with trilinear sampler. When doing non-uniform downsampling, use anisotropic sampler instead of trilinear. Not sure high level image processing libraries like the mentioned numpy, PIL or TensorFlow are doing anything like that, though.

bartwr 2y ago
Original author here and it's been a while since I have read such word salad nonsense, sorry. Why people who have no idea or expertise comment on articles? GenerateMips API constructs a mip chain by using box/bilinear (equivalent for factors of two) log N times. Trilinear interpolates across three dimensions, such as 3D textures or mip chains. It is not a method for downsampling, but a method for filtering that interpolates two bilinear results, such as two bilinear filters of mip levels that were generated with "some" downsampling filter (which can be anything from box to Lanczos). Anisotropic is a hybrid between trilinear across a smaller interpolation axis under perspective projection of a 3D asset and multiple taps along the longer axis. (More expensive)
- Const-me 2y ago
  > Trilinear interpolates across three dimensions, such as 3D textures or mip chains I meant trilinear interpolation across the mip chain. > generated with "some" downsampling filter (which can be anything from box to Lanczos) In practice, whichever method is implemented in user-mode half of GPU drivers is pretty good. > It is not a method for downsampling No, but it can be applied for downsampling as well. > under perspective projection of a 3D asset Texture samplers don’t know or care about projections. They only take 2D texture coordinates, and screen-space derivatives of these. This is precisely what enables to use texture samplers to downsample images. The only caveat, if you do that by dispatching a compute shader as opposed to rendering a full-screen triangle, you’ll have to supply screen-space derivatives manually in the arguments of Texture2D.SampleGrad method. When doing non-uniform downsampling without perspective projections, these ddx/ddy numbers are the same for all output pixels, and are trivial to compute on CPU before dispatching the shader. > More expensive On modern computers, the performance overhead of anisotropic sampling compared to trilinear is just barely measurable.

qwertox 2y ago

On a similar note, yesterday I stumbled across a 4-days old video related to dithering which is really worth watching. "Surface-Stable Fractal Dithering" [0] by "Rune Skovbo Johansen, indie game developer and procedural generation enthusiast" [0] https://www.youtube.com/watch?v=HPqGaIMVuLs https://www.youtube.com/watch?v=HPqGaIMVuLs

magicalhippo 2y ago
It was also covered here 4 days ago[1], again I might add. Very cool effect, and one of my absolute favorite games. [1]: https://news.ycombinator.com/item?id=42808889 https://news.ycombinator.com/item?id=42808889

johncostella 2y ago

I'm biased, but Magic Kernel Sharp does a good job for many applications. :) https://johncostella.com/magic/ https://johncostella.com/magic/