5 ms·

I imagine this came up partly as a result of the recent alpha compositing discussion. We desperately need some research, based in user studies and using modern

I imagine this came up partly as a result of the recent alpha compositing discussion. We desperately need some research, based in user studies and using modern display technology, to settle some basic questions: * What reconstruction filter gives the best results? Is it the same for vector (text) and natural images? By "best" I do mean contrast (sharpness) and lack of visible artifacts. * For rendering of very thin lines (relevant to text), what gamma curve gives the perception of equal line thickness across the range of subpixel phase? How does it vary with display dpi? (Hint: it's likely not linear luminance) * What gamma curve yields the perception of equal width of black-on-white and white-on-black thin lines (also relevant for text)? (Hint: likely not linear luminance) I've seen a number of discussions where people feel they are able to answer these questions from first principles, and corresponding arguments that doing these in a "correct" way gives results that are less visually appealing than the common assumptions of treating a pixel as a little square (so doing a box filter for reconstruction) and ignoring gamma so effectively using a perceptual color space for the purposes of alpha compositing. I posit that these questions cannot be solved by argumentation. I think user studies might not be very difficult to do; you could probably get "good enough" results by doing online surveys, though this wouldn't pass standards of academic rigor.

BubRoss 7y ago
I can tell you right now that you will have a very difficult time beating a normalized gauss filter with a diameter of around 2.2 pixels in a general case. Color and luminance is a separate and orthogonal issue from filtering. I also know that people get away with compositing without converting to linear space, but I'm skeptical that any benefits they see aren't just a matter of getting the color curve they want for free, as opposed to doing a similar correction after something has been composited correctly.
- raphlinus 7y ago
  Reeeaallly? I haven't made the images, but my intuition is telling me that text will be noticeably blurry compared with a box filter. I basically agree with your points regarding compositing in a linear space, except that I suspect that thin black-on-white lines will come out looking thin and spindly.
  - BubRoss 7y ago
    It sounds like you are talking about text that doesn't move. If you want a general filter that can give a result without visible aliasing while sacrificing as little sharpness as possible, a 2.2 gauss filter is very hard to beat and I have spent a lot of time trying. Box filters can be sharper, lancoz filters can be better for scaling down a final image, etc. but they will alias in a general sense. You might not see it in static text which is fine. Also thin black on white lines are an extreme outlier, since what you percieve is relative and the entire image matters. It is more into the realm of optical illusion that play off our relative sensitivity.
    - raphlinus 7y ago
      So obviously text is one of the things I care about a lot. I'm willing to accept your filter as being very good for a lot of stuff other than presenting a GUI. Does this mean that there is not one true right answer for images, as there is for audio[0]? That you should use different reconstruction filters depending on the application. Is it accurate to say that a pixel is not a little square unless you're rendering text, in which case it is? [0] I know this is a slight oversimplification, if you care deeply about latency you might choose between linear and minimal phase, etc. But for consumer applications it's true enough.
    - jacobolus 7y ago
      A Gaussian kernel will reduce ringing artifacts, but is much blurrier than alternative resampling kernels. https://www.imagemagick.org/Usage/filter/nicolas/ https://www.imagemagick.org/Usage/filter/nicolas/ Does “diameter” here mean “twice the standard deviation”?
      - BubRoss 7y ago
        No, diameter means the width of the filter in pixels. Each side would extend 1.1 pixels for a 2.2 diameter filter. If you compare filters at the same width, gauss will be softer, which is why you can lower the diameter.
- 0815test 7y ago
  Lanczos interpolation seems to beat a gauss filter for me. It has some overshooting/ringing effects of course, but the improved sharpness is a great tradeoff compared to the blurriness of gauss filtering. Of course, gauss filtering might still play a residual role in analog systems, such as the one that physically displays stuff via a CRT screen.
- mrob 7y ago
  True, but how often do you care about the general case? Most images are representations of real objects, or symbols or diagrams designed for human comprehension. Ignoring cases that in practice are very unlikely to occur lets you optimize for the more common cases. Techniques like resizing in a sigmoidal color space (see http://www.imagemagick.org/Usage/resize/#resize_sigmoidal http://www.imagemagick.org/Usage/resize/#resize_sigmoidal ) don't have any rigorous mathematical basis, but they're tuned subjectively to have good results in common cases.
- dahart 7y ago
  I was working at PDI Dreamworks during one of the semi annual investigations into which filter kernel was best. And I was completely blown away by how quickly the lighting sup could identify and react to various filters. Gaussian was voted down reliably and repeatedly for being too blurry. Personally, I like the extra blur I get (and extra safety and guarantees) you get with Gaussian. Back in the NTSC days I discovered that vertically blurring interlaced video made it noticeably more clear and visible, even though it was softer. But, If you do really spend time with sharper filters, it is true that Gaussian is softer and some pros really do want sharper images than Gaussian can provide.
  - BubRoss 7y ago
    Were all the filters being compared with the same width? At the same width it will be softer, at a smaller pixel radius is where you can compare aliasing with the same visual sharpness as something like catmull-rom.
    - dahart 7y ago
      There were a variety of widths, it was one axis of the study. But I don’t remember the details, it is certainly possible you’re talking about something we didn’t test. Personally, at first I couldn’t even see the differences they were discussing. Having studied graphics and signal processing for a few years in graduate school before that job, I thought I would be good at seeing the differences, and I was a bit shocked how good they were at it, and how not that good I was. :) Truncating the Gaussian too closely though, and it’s not exactly a Gaussian anymore, you lose the best antialiasing properties. I can totally see how it will be sharper and more comparable to other popular filters. (Normalized & truncated at 1.1 radius is just slightly outside the 1 std dev line, right?) Gaussian is my personal choice for large format prints of images with extreme aliasing problems.
      - blevin 7y ago
        My experience has likewise been that film DP’s have extremely impressive visual acuity, memory for color, etc. Talented artists often have developed whole sets of skills that the rest of us are unaware are even skills. In the same way a programmer might have thought about cache line false sharing as it affects memory hierarchy throughput, visual art often hides lots of expertise you cannot directly perceive, even as you can sense the quality of the whole.
anderskaseorg 7y ago
> * What gamma curve yields the perception of equal width of black-on-white and white-on-black thin lines (also relevant for text)? (Hint: likely not linear luminance) Gamma curves do not affect black and white themselves, only intermediate grays. It’s true that grays are usually used to draw antialiased black and white lines, but we can also think about “ideal” (axis-aligned, pixel-centered, non-antialiased) black and white lines. By framing this as a question about gamma, you’ve implicitly assumed that “ideal” black lines on white would have equal perceived thickness to “ideal” white lines on black. This is not the case, as typographers have known for decades. The right way to draw perceptually equal-thickness black-on-white and white-on-black lines is to vary the line width (perhaps even by more than a full pixel if the lines are thick enough!). Gamma only comes in afterwards, to help us reproduce the varied widths accurately, and the accurate way to do that is in a linear color space.
- raphlinus 7y ago
  I should have clarified that I meant lines with subpixel phase and possibly widths that are not integral numbers of pixels, ie the antialiased case. Also I totally agree that we need to take into account the perceptual differences between black-on-white and white-on-black even assuming the display technology is perfect. That's one reason doing these studies is not trivial!
  - anderskaseorg 7y ago
    The non-antialiased “case” is one spot among the continuum of antialiased “cases”; it doesn’t satisfy totally different perceptual rules. What gamma curve would you propose to correct for the perceptual equality of a 9.3px white-on-black line with a 11.6px black-on-white line? What I’m saying is that you’re presupposing the wrong layer to make the correction (antialiasing or no).
    - raphlinus 7y ago
      You are correct. Change my third question to, "after applying a correct gamma curve to achieve perceptually uniform line widths in the unipolar case, what is the correspondence of line widths when inverting the polarity to preserve the perception of width matching?" Then you want to break that down into the contribution assuming a perfect display and the effect from lower dpi. This question is less important than the other two, and plausibly the best place to address the effect is in design, rather than rendering.
Someone 7y ago
There likely is tons of research on this in psychophysics (https://en.wikipedia.org/wiki/Psychophysics https://en.wikipedia.org/wiki/Psychophysics). I’m not familiar with it, but I do know that you’ll find that it isn’t as simple as what you state. For example, Google gave me http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.5122&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.... I don’t know whether it’s good, or when it was written. The perception of equal line thickness may depend on orientation of the lines (https://res.mdpi.com/vision/vision-03-00001/article_deploy/vision-03-00001.pdf https://res.mdpi.com/vision/vision-03-00001/article_deploy/v...), distance from the fovea, distance from where your attention lies (a can of worms even deeper. I don’t think there’s agreement on whether one can attend to more than one visual location at a time, or what ‘attention’ even is), direction of those distances (vertical will almost certainly be different from horizontal, sight could be better or worse in the nasal direction vs the temporal one), light/dark adaptation of the eyes, whether subjects are color-blind, etc. I fear that, to qet something that’s better than Weber’s law (https://en.wikipedia.org/wiki/Weber–Fechner_law https://en.wikipedia.org/wiki/Weber–Fechner_law), you need a model so complex that it doesn’t make sense to start using it.
- raphlinus 7y ago
  It's possible this research has been done, but if so I haven't seen it yet. Basically the reason I feel confident asserting it is that I'm talking very specifically about the stimulus produced by modern displays. I know there's some work by Avi Naiman on CRT displays from the very early 90's[0]. There might be some other work done on low-resolution LCD's, but even aside from resolution it's only modern, high quality displays that have good contrast and low dependence on viewing angle (which affects gamma greatly). Another place to look is Kevin Larson's work on subpixel rendering, which informed Microsoft's ClearType efforts. But that was done mostly around 10 years ago, when displays also were different. A good representative is [1]. Here's another pretty good paper I found[2], but it focuses more on the display technology than the perception side. So what I'm looking for is adjacent to general psychophysics results on visual perception, but much more specific to what real displays do. That literature is pretty thin on the ground. [0]: Avi C. Naiman and Walter Makous. Spatial nonlinearities of gray-scale CRT pixels, 1992 [1]: https://www.researchgate.net/publication/264485766_Optimizing_subpixel_rendering_using_a_perceptual_metric https://www.researchgate.net/publication/264485766_Optimizin... [2]: https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a967752620e25d4.pdf https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a...

BubRoss 7y ago

I can tell you right now that you will have a very difficult time beating a normalized gauss filter with a diameter of around 2.2 pixels in a general case. Color and luminance is a separate and orthogonal issue from filtering. I also know that people get away with compositing without converting to linear space, but I'm skeptical that any benefits they see aren't just a matter of getting the color curve they want for free, as opposed to doing a similar correction after something has been composited correctly.

raphlinus 7y ago
Reeeaallly? I haven't made the images, but my intuition is telling me that text will be noticeably blurry compared with a box filter. I basically agree with your points regarding compositing in a linear space, except that I suspect that thin black-on-white lines will come out looking thin and spindly.
- BubRoss 7y ago
  It sounds like you are talking about text that doesn't move. If you want a general filter that can give a result without visible aliasing while sacrificing as little sharpness as possible, a 2.2 gauss filter is very hard to beat and I have spent a lot of time trying. Box filters can be sharper, lancoz filters can be better for scaling down a final image, etc. but they will alias in a general sense. You might not see it in static text which is fine. Also thin black on white lines are an extreme outlier, since what you percieve is relative and the entire image matters. It is more into the realm of optical illusion that play off our relative sensitivity.
  - raphlinus 7y ago
    So obviously text is one of the things I care about a lot. I'm willing to accept your filter as being very good for a lot of stuff other than presenting a GUI. Does this mean that there is not one true right answer for images, as there is for audio[0]? That you should use different reconstruction filters depending on the application. Is it accurate to say that a pixel is not a little square unless you're rendering text, in which case it is? [0] I know this is a slight oversimplification, if you care deeply about latency you might choose between linear and minimal phase, etc. But for consumer applications it's true enough.
  - jacobolus 7y ago
    A Gaussian kernel will reduce ringing artifacts, but is much blurrier than alternative resampling kernels. https://www.imagemagick.org/Usage/filter/nicolas/ https://www.imagemagick.org/Usage/filter/nicolas/ Does “diameter” here mean “twice the standard deviation”?
    - BubRoss 7y ago
      No, diameter means the width of the filter in pixels. Each side would extend 1.1 pixels for a 2.2 diameter filter. If you compare filters at the same width, gauss will be softer, which is why you can lower the diameter.
0815test 7y ago
Lanczos interpolation seems to beat a gauss filter for me. It has some overshooting/ringing effects of course, but the improved sharpness is a great tradeoff compared to the blurriness of gauss filtering. Of course, gauss filtering might still play a residual role in analog systems, such as the one that physically displays stuff via a CRT screen.
mrob 7y ago
True, but how often do you care about the general case? Most images are representations of real objects, or symbols or diagrams designed for human comprehension. Ignoring cases that in practice are very unlikely to occur lets you optimize for the more common cases. Techniques like resizing in a sigmoidal color space (see http://www.imagemagick.org/Usage/resize/#resize_sigmoidal http://www.imagemagick.org/Usage/resize/#resize_sigmoidal ) don't have any rigorous mathematical basis, but they're tuned subjectively to have good results in common cases.
dahart 7y ago
I was working at PDI Dreamworks during one of the semi annual investigations into which filter kernel was best. And I was completely blown away by how quickly the lighting sup could identify and react to various filters. Gaussian was voted down reliably and repeatedly for being too blurry. Personally, I like the extra blur I get (and extra safety and guarantees) you get with Gaussian. Back in the NTSC days I discovered that vertically blurring interlaced video made it noticeably more clear and visible, even though it was softer. But, If you do really spend time with sharper filters, it is true that Gaussian is softer and some pros really do want sharper images than Gaussian can provide.
- BubRoss 7y ago
  Were all the filters being compared with the same width? At the same width it will be softer, at a smaller pixel radius is where you can compare aliasing with the same visual sharpness as something like catmull-rom.
  - dahart 7y ago
    There were a variety of widths, it was one axis of the study. But I don’t remember the details, it is certainly possible you’re talking about something we didn’t test. Personally, at first I couldn’t even see the differences they were discussing. Having studied graphics and signal processing for a few years in graduate school before that job, I thought I would be good at seeing the differences, and I was a bit shocked how good they were at it, and how not that good I was. :) Truncating the Gaussian too closely though, and it’s not exactly a Gaussian anymore, you lose the best antialiasing properties. I can totally see how it will be sharper and more comparable to other popular filters. (Normalized & truncated at 1.1 radius is just slightly outside the 1 std dev line, right?) Gaussian is my personal choice for large format prints of images with extreme aliasing problems.
    - blevin 7y ago
      My experience has likewise been that film DP’s have extremely impressive visual acuity, memory for color, etc. Talented artists often have developed whole sets of skills that the rest of us are unaware are even skills. In the same way a programmer might have thought about cache line false sharing as it affects memory hierarchy throughput, visual art often hides lots of expertise you cannot directly perceive, even as you can sense the quality of the whole.

anderskaseorg 7y ago

> * What gamma curve yields the perception of equal width of black-on-white and white-on-black thin lines (also relevant for text)? (Hint: likely not linear luminance) Gamma curves do not affect black and white themselves, only intermediate grays. It’s true that grays are usually used to draw antialiased black and white lines, but we can also think about “ideal” (axis-aligned, pixel-centered, non-antialiased) black and white lines. By framing this as a question about gamma, you’ve implicitly assumed that “ideal” black lines on white would have equal perceived thickness to “ideal” white lines on black. This is not the case, as typographers have known for decades. The right way to draw perceptually equal-thickness black-on-white and white-on-black lines is to vary the line width (perhaps even by more than a full pixel if the lines are thick enough!). Gamma only comes in afterwards, to help us reproduce the varied widths accurately, and the accurate way to do that is in a linear color space.

raphlinus 7y ago
I should have clarified that I meant lines with subpixel phase and possibly widths that are not integral numbers of pixels, ie the antialiased case. Also I totally agree that we need to take into account the perceptual differences between black-on-white and white-on-black even assuming the display technology is perfect. That's one reason doing these studies is not trivial!
- anderskaseorg 7y ago
  The non-antialiased “case” is one spot among the continuum of antialiased “cases”; it doesn’t satisfy totally different perceptual rules. What gamma curve would you propose to correct for the perceptual equality of a 9.3px white-on-black line with a 11.6px black-on-white line? What I’m saying is that you’re presupposing the wrong layer to make the correction (antialiasing or no).
  - raphlinus 7y ago
    You are correct. Change my third question to, "after applying a correct gamma curve to achieve perceptually uniform line widths in the unipolar case, what is the correspondence of line widths when inverting the polarity to preserve the perception of width matching?" Then you want to break that down into the contribution assuming a perfect display and the effect from lower dpi. This question is less important than the other two, and plausibly the best place to address the effect is in design, rather than rendering.

Someone 7y ago

There likely is tons of research on this in psychophysics (https://en.wikipedia.org/wiki/Psychophysics https://en.wikipedia.org/wiki/Psychophysics). I’m not familiar with it, but I do know that you’ll find that it isn’t as simple as what you state. For example, Google gave me http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.5122&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.... I don’t know whether it’s good, or when it was written. The perception of equal line thickness may depend on orientation of the lines (https://res.mdpi.com/vision/vision-03-00001/article_deploy/vision-03-00001.pdf https://res.mdpi.com/vision/vision-03-00001/article_deploy/v...), distance from the fovea, distance from where your attention lies (a can of worms even deeper. I don’t think there’s agreement on whether one can attend to more than one visual location at a time, or what ‘attention’ even is), direction of those distances (vertical will almost certainly be different from horizontal, sight could be better or worse in the nasal direction vs the temporal one), light/dark adaptation of the eyes, whether subjects are color-blind, etc. I fear that, to qet something that’s better than Weber’s law (https://en.wikipedia.org/wiki/Weber–Fechner_law https://en.wikipedia.org/wiki/Weber–Fechner_law), you need a model so complex that it doesn’t make sense to start using it.

raphlinus 7y ago
It's possible this research has been done, but if so I haven't seen it yet. Basically the reason I feel confident asserting it is that I'm talking very specifically about the stimulus produced by modern displays. I know there's some work by Avi Naiman on CRT displays from the very early 90's[0]. There might be some other work done on low-resolution LCD's, but even aside from resolution it's only modern, high quality displays that have good contrast and low dependence on viewing angle (which affects gamma greatly). Another place to look is Kevin Larson's work on subpixel rendering, which informed Microsoft's ClearType efforts. But that was done mostly around 10 years ago, when displays also were different. A good representative is [1]. Here's another pretty good paper I found[2], but it focuses more on the display technology than the perception side. So what I'm looking for is adjacent to general psychophysics results on visual perception, but much more specific to what real displays do. That literature is pretty thin on the ground. [0]: Avi C. Naiman and Walter Makous. Spatial nonlinearities of gray-scale CRT pixels, 1992 [1]: https://www.researchgate.net/publication/264485766_Optimizing_subpixel_rendering_using_a_perceptual_metric https://www.researchgate.net/publication/264485766_Optimizin... [2]: https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a967752620e25d4.pdf https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a...