6 ms·

However, image is not a wave either. I have mixed feelings about this memo. It's right about practical aspects of resampling filters, but tries too hard to jus

by pornel 6y ago

However, image is not a wave either. I have mixed feelings about this memo. It's right about practical aspects of resampling filters, but tries too hard to justify that with sampling theory. For example, pixel-aligned sharp edges exist and are meaningful in images, unlike perfectly square waves in sampling theory.

sudosysgen 6y ago
Pixel aligned sharp edges do not actually exist and are not meaningful in images, because a perfectly sharp lens does not exist (and cannot exist) and as a result you can never form a sharp edge in an image. You also have de-focus that prevents you from doing so, and a lens that has a wider depth of field has an immediately noticeable limit in sharpness. Even if you were somehow able to create a perfect lens, you would not be able to create a perfectly sharp edge with real world objects.
- wtallis 6y ago
  See, here you're committing the same error that the paper does: pretending pixels are all about photography and optics, and ignoring that some computer-generated graphics actually are supposed to represent perfect squares. Sometimes, pixels really are little squares. Not always, but not never, either.
  - chaorace 6y ago
    I disagree. Would you say a specific address in an array has edges? No, because it's an address, a fixed point. Even if that array accidentally described a square, the individual array addresses would have nothing to do with that. It's not a question of the representation, it's a question of quanta. Pixels are data, the little squares are the artifacts your LCD eventually produces with the help of that data.
    - wtallis 6y ago
      When your GPU is rasterizing the edges of polygons, it computes (sometimes just approximates) how much of a little square is covered by that polygon and uses that as the weight when averaging what color to assign to that pixel. The resulting rendered image is most correctly interpreted as an array of little squares, not point samples and definitely not truncated gaussians.
      - sudosysgen 6y ago
        Actually no, that's not the case, rasterization is on-off at the hardware level. You need anti-aliasing for the behaviour you are describing, which very rarely works the way you describe - the best we have right now as far as quality uses multisampling.
        Jasper_ 6y ago
        2D graphics uses coverage-based antialiasing which computes the coverage over the pixel square. https://nothings.org/gamedev/rasterize/ https://nothings.org/gamedev/rasterize/
        sudosysgen 6y ago
        And that is not done by the GPU. The top comment started with "your GPU does...." In any case, that is an extreme edge case of software renderers that doesn't even come close to a significant part of 2D graphics in real life. Indeed, most 2D graphics is really flat 3D graphics done using GPU routines and does not the work that way. I know that some extreme edge cases do use coverage based rasterization, but : >You need anti-aliasing for the behaviour you are describing, which very rarely works the way you describe This is a case of anti-aliasing (read the title of the article) and is extremely rarely used. It's essentially irrelevant when discussing how graphics work in real life. I really cannot overstate just how rarely software rasterizers are used for interactive graphics in 2020, coverage based rasterizers are an even smaller subset of that. It really makes a ton more sense to use a GPU rasterizer and use MSAA or oversample the whole image.
        6y ago
    - function_seven 6y ago
      The data often "knows" where it will be displayed, and is designed with that knowledge. To go to the absurd extreme here, the data is just 1s and 0s, not pixels or samples or anything more. Of course that's nonsense, because the data has context and the arrangement of those samples or pixels has a purpose. Sometimes that purpose is to serve as a sampling of a real-world continuous image, other times it is to describe the arrangement and color of tiny little squares on an LCD screen. Depending on what context you're working in pixels can be squares, or may not be.
      - mark-r 6y ago
        They may be captured as squares, and displayed as squares. But mathematically everything works better if they're treated as dimensionless points.
    - coldtea 6y ago
      >Pixels are data, the little squares are the artifacts your LCD eventually produces with the help of that data. Only we don't care about the data, except when they are rendered as "artifacts" at our screens. So prioritising the abstract data is putting the cart before the horse. We only captured/generated those data to display them on our LCD.
  - sudosysgen 6y ago
    A UI element will almost never be perfectly square, exactly aligned on a pixel. And even if it was, LCD pixels aren't even squares. They are three rectangles with different colors and gaps between them. This fact means that if you want to have proper representation of even a pixel-perfect square, because of the gap between pixels and because of sub-pixels, a perfect representation would not be such.
    - megameter 6y ago
      Most pixel art today is designed around square pixels. The pixels are typically blown up to some multiple for viewing - and in many cases, not even an integer multiple! This has resulted in the design of filter kernels that make some trade-off between the true sharpness of square pixels and the aliasing of nearest neighbor interpolation. Therefore, even before images hit the display there is a rationale to avoid using photographic resampling techniques, just because the method of their authoring defined the meaning otherwise. See also: The gradual evolution of font rendering techniques. Earlier versions of Windows aimed for pixel-grid snapping to produce clean, sharp edges. Newer ones introduced anti-aliasing techniques, but again made trade-offs towards sharp edges, including exploitation of the LCD display format, while the contemporary Apple rendering favored the photographic approach. With the introduction of high DPI the differences have become less pronounced, but there's still disagreement about how best to render vector fonts into pixels.
    - jack1243star 6y ago
      The UI elements I use most, such as the mouse cursor and window borders are perfectly aligned to the pixel grid.
      - sudosysgen 6y ago
        I don't know about yours, but my mouse cursor is not a perfectly straight 2+ pixel wide solid-colored rectangle, so that doesn't really matter. As for windows edges, I'll give you that one. However, the edge of my window happens not to be green, so it doesn't actually align with the elements of my LCD :) I just so happen not to notice it, because my eyes don't have individual pixel resolution, almost as if there was a low-pass filter of the order > 2 lambda... Food for thought!
- comex 6y ago
  True, but the resolution needed to perceive the edge as not sharp on a typical display is far higher than the resolution of the image itself. Or reformulated: On a low-resolution LCD, an axis-aligned square will appear much sharper than a square drawn at any other angle.
  - sudosysgen 6y ago
    That's true, but we're really getting there fast. My very sharp Minolta F1.4 full-frame lens is diffraction limited already at f/8 to f/11 at a definition of 6000 x 4000. It's also much, much sharper than the vast majority of pictures people take. While yes, 6000x4000 is a lot, monitors are already coming out with higher resolutions, so it's relevant right now. The fraction of images taken at an actual resolution so high that it has to be down scaled on a 6K or 8K monitor in practice is exceedingly thin. Even with a Sony A7RIV, an insane camera, and mind boggingly sharp lenses, most of your pictures after the Bayer filter and taking into account sampling (which is very real in photography due to Moire), most of your pictures either due to depth of field, optical abberations or motion blur, will not be at the level where you can create a truly sharp line of frequency higher than that between two pixels. So while it is often true, this is increasingly not the case.
contravariant 6y ago
There's no problem with viewing an image as a wave, however if you do so you also need to accept that it contains frequency components well beyond the Nyquist frequency so the sampling theorem will be of no help in reconstructing the image. Still lots of operations on image can be stated in terms of convolution (or a regularized inverse of it) so it's not like Fourier analysis is entirely useless.
blt 6y ago
Pixel-aligned sharp edges were super important on low-DPI displays, but they are increasingly less important. A one-pixel line on a phone screen is tiny, so most designs will use lines of more than one pixel. A pixel-perfect 3-pixel line doesn't look very different from an anti-aliased 3-pixel line with arbitrary floating-point coordinates on a phone.
- Avamander 6y ago
  I must have a low-DPI display, but it happens way too often on the web that some text is off by a tiny bit and blurred by antialiasing.
  - marcan_42 6y ago
    Twitter? Because Twitter has been doing that a lot for me recently. It's like all their devs are on mobile or Retina screens and overpowered CPUs and nobody noticed the problem (and resulting processing overhead).
    - Avamander 6y ago
      Twitter, Facebook, Telegram Web, there have been others but none as noticeable.

by pornel 6y ago

sudosysgen 6y ago

Pixel aligned sharp edges do not actually exist and are not meaningful in images, because a perfectly sharp lens does not exist (and cannot exist) and as a result you can never form a sharp edge in an image. You also have de-focus that prevents you from doing so, and a lens that has a wider depth of field has an immediately noticeable limit in sharpness. Even if you were somehow able to create a perfect lens, you would not be able to create a perfectly sharp edge with real world objects.

wtallis 6y ago
See, here you're committing the same error that the paper does: pretending pixels are all about photography and optics, and ignoring that some computer-generated graphics actually are supposed to represent perfect squares. Sometimes, pixels really are little squares. Not always, but not never, either.
- chaorace 6y ago
  I disagree. Would you say a specific address in an array has edges? No, because it's an address, a fixed point. Even if that array accidentally described a square, the individual array addresses would have nothing to do with that. It's not a question of the representation, it's a question of quanta. Pixels are data, the little squares are the artifacts your LCD eventually produces with the help of that data.
  - wtallis 6y ago
    When your GPU is rasterizing the edges of polygons, it computes (sometimes just approximates) how much of a little square is covered by that polygon and uses that as the weight when averaging what color to assign to that pixel. The resulting rendered image is most correctly interpreted as an array of little squares, not point samples and definitely not truncated gaussians.
    - sudosysgen 6y ago
      Actually no, that's not the case, rasterization is on-off at the hardware level. You need anti-aliasing for the behaviour you are describing, which very rarely works the way you describe - the best we have right now as far as quality uses multisampling.
      - Jasper_ 6y ago
        2D graphics uses coverage-based antialiasing which computes the coverage over the pixel square. https://nothings.org/gamedev/rasterize/ https://nothings.org/gamedev/rasterize/
        sudosysgen 6y ago
        And that is not done by the GPU. The top comment started with "your GPU does...." In any case, that is an extreme edge case of software renderers that doesn't even come close to a significant part of 2D graphics in real life. Indeed, most 2D graphics is really flat 3D graphics done using GPU routines and does not the work that way. I know that some extreme edge cases do use coverage based rasterization, but : >You need anti-aliasing for the behaviour you are describing, which very rarely works the way you describe This is a case of anti-aliasing (read the title of the article) and is extremely rarely used. It's essentially irrelevant when discussing how graphics work in real life. I really cannot overstate just how rarely software rasterizers are used for interactive graphics in 2020, coverage based rasterizers are an even smaller subset of that. It really makes a ton more sense to use a GPU rasterizer and use MSAA or oversample the whole image.
        6y ago
  - function_seven 6y ago
    The data often "knows" where it will be displayed, and is designed with that knowledge. To go to the absurd extreme here, the data is just 1s and 0s, not pixels or samples or anything more. Of course that's nonsense, because the data has context and the arrangement of those samples or pixels has a purpose. Sometimes that purpose is to serve as a sampling of a real-world continuous image, other times it is to describe the arrangement and color of tiny little squares on an LCD screen. Depending on what context you're working in pixels can be squares, or may not be.
    - mark-r 6y ago
      They may be captured as squares, and displayed as squares. But mathematically everything works better if they're treated as dimensionless points.
  - coldtea 6y ago
    >Pixels are data, the little squares are the artifacts your LCD eventually produces with the help of that data. Only we don't care about the data, except when they are rendered as "artifacts" at our screens. So prioritising the abstract data is putting the cart before the horse. We only captured/generated those data to display them on our LCD.
- sudosysgen 6y ago
  A UI element will almost never be perfectly square, exactly aligned on a pixel. And even if it was, LCD pixels aren't even squares. They are three rectangles with different colors and gaps between them. This fact means that if you want to have proper representation of even a pixel-perfect square, because of the gap between pixels and because of sub-pixels, a perfect representation would not be such.
  - megameter 6y ago
    Most pixel art today is designed around square pixels. The pixels are typically blown up to some multiple for viewing - and in many cases, not even an integer multiple! This has resulted in the design of filter kernels that make some trade-off between the true sharpness of square pixels and the aliasing of nearest neighbor interpolation. Therefore, even before images hit the display there is a rationale to avoid using photographic resampling techniques, just because the method of their authoring defined the meaning otherwise. See also: The gradual evolution of font rendering techniques. Earlier versions of Windows aimed for pixel-grid snapping to produce clean, sharp edges. Newer ones introduced anti-aliasing techniques, but again made trade-offs towards sharp edges, including exploitation of the LCD display format, while the contemporary Apple rendering favored the photographic approach. With the introduction of high DPI the differences have become less pronounced, but there's still disagreement about how best to render vector fonts into pixels.
  - jack1243star 6y ago
    The UI elements I use most, such as the mouse cursor and window borders are perfectly aligned to the pixel grid.
    - sudosysgen 6y ago
      I don't know about yours, but my mouse cursor is not a perfectly straight 2+ pixel wide solid-colored rectangle, so that doesn't really matter. As for windows edges, I'll give you that one. However, the edge of my window happens not to be green, so it doesn't actually align with the elements of my LCD :) I just so happen not to notice it, because my eyes don't have individual pixel resolution, almost as if there was a low-pass filter of the order > 2 lambda... Food for thought!
comex 6y ago
True, but the resolution needed to perceive the edge as not sharp on a typical display is far higher than the resolution of the image itself. Or reformulated: On a low-resolution LCD, an axis-aligned square will appear much sharper than a square drawn at any other angle.
- sudosysgen 6y ago
  That's true, but we're really getting there fast. My very sharp Minolta F1.4 full-frame lens is diffraction limited already at f/8 to f/11 at a definition of 6000 x 4000. It's also much, much sharper than the vast majority of pictures people take. While yes, 6000x4000 is a lot, monitors are already coming out with higher resolutions, so it's relevant right now. The fraction of images taken at an actual resolution so high that it has to be down scaled on a 6K or 8K monitor in practice is exceedingly thin. Even with a Sony A7RIV, an insane camera, and mind boggingly sharp lenses, most of your pictures after the Bayer filter and taking into account sampling (which is very real in photography due to Moire), most of your pictures either due to depth of field, optical abberations or motion blur, will not be at the level where you can create a truly sharp line of frequency higher than that between two pixels. So while it is often true, this is increasingly not the case.

contravariant 6y ago

There's no problem with viewing an image as a wave, however if you do so you also need to accept that it contains frequency components well beyond the Nyquist frequency so the sampling theorem will be of no help in reconstructing the image. Still lots of operations on image can be stated in terms of convolution (or a regularized inverse of it) so it's not like Fourier analysis is entirely useless.

blt 6y ago

Pixel-aligned sharp edges were super important on low-DPI displays, but they are increasingly less important. A one-pixel line on a phone screen is tiny, so most designs will use lines of more than one pixel. A pixel-perfect 3-pixel line doesn't look very different from an anti-aliased 3-pixel line with arbitrary floating-point coordinates on a phone.

Avamander 6y ago
I must have a low-DPI display, but it happens way too often on the web that some text is off by a tiny bit and blurred by antialiasing.
- marcan_42 6y ago
  Twitter? Because Twitter has been doing that a lot for me recently. It's like all their devs are on mobile or Retina screens and overpowered CPUs and nobody noticed the problem (and resulting processing overhead).
  - Avamander 6y ago
    Twitter, Facebook, Telegram Web, there have been others but none as noticeable.