7 ms·
A Pixel Is Not A Little Square (1995) [pdf]
- sctb 7y agoSome previous discussions: https://news.ycombinator.com/item?id=8614159 https://news.ycombinator.com/item?id=8614159 https://news.ycombinator.com/item?id=1472175 https://news.ycombinator.com/item?id=1472175
- raphlinus 7y agoI imagine this came up partly as a result of the recent alpha compositing discussion. We desperately need some research, based in user studies and using modern display technology, to settle some basic questions: * What reconstruction filter gives the best results? Is it the same for vector (text) and natural images? By "best" I do mean contrast (sharpness) and lack of visible artifacts. * For rendering of very thin lines (relevant to text), what gamma curve gives the perception of equal line thickness across the range of subpixel phase? How does it vary with display dpi? (Hint: it's likely not linear luminance) * What gamma curve yields the perception of equal width of black-on-white and white-on-black thin lines (also relevant for text)? (Hint: likely not linear luminance) I've seen a number of discussions where people feel they are able to answer these questions from first principles, and corresponding arguments that doing these in a "correct" way gives results that are less visually appealing than the common assumptions of treating a pixel as a little square (so doing a box filter for reconstruction) and ignoring gamma so effectively using a perceptual color space for the purposes of alpha compositing. I posit that these questions cannot be solved by argumentation. I think user studies might not be very difficult to do; you could probably get "good enough" results by doing online surveys, though this wouldn't pass standards of academic rigor.
- BubRoss 7y agoI can tell you right now that you will have a very difficult time beating a normalized gauss filter with a diameter of around 2.2 pixels in a general case. Color and luminance is a separate and orthogonal issue from filtering. I also know that people get away with compositing without converting to linear space, but I'm skeptical that any benefits they see aren't just a matter of getting the color curve they want for free, as opposed to doing a similar correction after something has been composited correctly.
- raphlinus 7y agoReeeaallly? I haven't made the images, but my intuition is telling me that text will be noticeably blurry compared with a box filter. I basically agree with your points regarding compositing in a linear space, except that I suspect that thin black-on-white lines will come out looking thin and spindly.
- BubRoss 7y agoIt sounds like you are talking about text that doesn't move. If you want a general filter that can give a result without visible aliasing while sacrificing as little sharpness as possible, a 2.2 gauss filter is very hard to beat and I have spent a lot of time trying. Box filters can be sharper, lancoz filters can be better for scaling down a final image, etc. but they will alias in a general sense. You might not see it in static text which is fine. Also thin black on white lines are an extreme outlier, since what you percieve is relative and the entire image matters. It is more into the realm of optical illusion that play off our relative sensitivity.
- raphlinus 7y agoSo obviously text is one of the things I care about a lot. I'm willing to accept your filter as being very good for a lot of stuff other than presenting a GUI. Does this mean that there is not one true right answer for images, as there is for audio[0]? That you should use different reconstruction filters depending on the application. Is it accurate to say that a pixel is not a little square unless you're rendering text, in which case it is? [0] I know this is a slight oversimplification, if you care deeply about latency you might choose between linear and minimal phase, etc. But for consumer applications it's true enough.
- jacobolus 7y agoA Gaussian kernel will reduce ringing artifacts, but is much blurrier than alternative resampling kernels. https://www.imagemagick.org/Usage/filter/nicolas/ https://www.imagemagick.org/Usage/filter/nicolas/ Does “diameter” here mean “twice the standard deviation”?
- anderskaseorg 7y ago> * What gamma curve yields the perception of equal width of black-on-white and white-on-black thin lines (also relevant for text)? (Hint: likely not linear luminance) Gamma curves do not affect black and white themselves, only intermediate grays. It’s true that grays are usually used to draw antialiased black and white lines, but we can also think about “ideal” (axis-aligned, pixel-centered, non-antialiased) black and white lines. By framing this as a question about gamma, you’ve implicitly assumed that “ideal” black lines on white would have equal perceived thickness to “ideal” white lines on black. This is not the case, as typographers have known for decades. The right way to draw perceptually equal-thickness black-on-white and white-on-black lines is to vary the line width (perhaps even by more than a full pixel if the lines are thick enough!). Gamma only comes in afterwards, to help us reproduce the varied widths accurately, and the accurate way to do that is in a linear color space.
- raphlinus 7y agoI should have clarified that I meant lines with subpixel phase and possibly widths that are not integral numbers of pixels, ie the antialiased case. Also I totally agree that we need to take into account the perceptual differences between black-on-white and white-on-black even assuming the display technology is perfect. That's one reason doing these studies is not trivial!
- anderskaseorg 7y agoThe non-antialiased “case” is one spot among the continuum of antialiased “cases”; it doesn’t satisfy totally different perceptual rules. What gamma curve would you propose to correct for the perceptual equality of a 9.3px white-on-black line with a 11.6px black-on-white line? What I’m saying is that you’re presupposing the wrong layer to make the correction (antialiasing or no).
- raphlinus 7y agoYou are correct. Change my third question to, "after applying a correct gamma curve to achieve perceptually uniform line widths in the unipolar case, what is the correspondence of line widths when inverting the polarity to preserve the perception of width matching?" Then you want to break that down into the contribution assuming a perfect display and the effect from lower dpi. This question is less important than the other two, and plausibly the best place to address the effect is in design, rather than rendering.
- Someone 7y agoThere likely is tons of research on this in psychophysics (https://en.wikipedia.org/wiki/Psychophysics https://en.wikipedia.org/wiki/Psychophysics). I’m not familiar with it, but I do know that you’ll find that it isn’t as simple as what you state. For example, Google gave me http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.5122&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.... I don’t know whether it’s good, or when it was written. The perception of equal line thickness may depend on orientation of the lines (https://res.mdpi.com/vision/vision-03-00001/article_deploy/vision-03-00001.pdf https://res.mdpi.com/vision/vision-03-00001/article_deploy/v...), distance from the fovea, distance from where your attention lies (a can of worms even deeper. I don’t think there’s agreement on whether one can attend to more than one visual location at a time, or what ‘attention’ even is), direction of those distances (vertical will almost certainly be different from horizontal, sight could be better or worse in the nasal direction vs the temporal one), light/dark adaptation of the eyes, whether subjects are color-blind, etc. I fear that, to qet something that’s better than Weber’s law (https://en.wikipedia.org/wiki/Weber–Fechner_law https://en.wikipedia.org/wiki/Weber–Fechner_law), you need a model so complex that it doesn’t make sense to start using it.
- raphlinus 7y agoIt's possible this research has been done, but if so I haven't seen it yet. Basically the reason I feel confident asserting it is that I'm talking very specifically about the stimulus produced by modern displays. I know there's some work by Avi Naiman on CRT displays from the very early 90's[0]. There might be some other work done on low-resolution LCD's, but even aside from resolution it's only modern, high quality displays that have good contrast and low dependence on viewing angle (which affects gamma greatly). Another place to look is Kevin Larson's work on subpixel rendering, which informed Microsoft's ClearType efforts. But that was done mostly around 10 years ago, when displays also were different. A good representative is [1]. Here's another pretty good paper I found[2], but it focuses more on the display technology than the perception side. So what I'm looking for is adjacent to general psychophysics results on visual perception, but much more specific to what real displays do. That literature is pretty thin on the ground. [0]: Avi C. Naiman and Walter Makous. Spatial nonlinearities of gray-scale CRT pixels, 1992 [1]: https://www.researchgate.net/publication/264485766_Optimizing_subpixel_rendering_using_a_perceptual_metric https://www.researchgate.net/publication/264485766_Optimizin... [2]: https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a967752620e25d4.pdf https://pdfs.semanticscholar.org/8fc4/ecc59380f24e363c6a5d6a...
- shawnz 7y agoA great video which touches on this idea, but mostly in the context of audio: https://youtu.be/cIQ9IXSUzuM https://youtu.be/cIQ9IXSUzuM (relevant part around 8 minutes)
- raphlinus 7y agoBe careful generalizing the audio results to pixels. The central lesson of xiph's work is that people simply cannot hear frequencies above, let's say 20kHz. Therefore, as long as your sampling rate is above the Nyquist limit (and under the assumption the signal chain is linear), any reconstruction filter that passes frequencies through 20kHz is effectively "perfect." There are two ways this is not true for pixels. First, even for "retina" displays the human visual system can make out spatial frequencies beyond the Nyquist limit of the display (this will vary by viewing distance, so is more of an issue for young people who can get close to their displays). Second, even assuming perfect gamma, the display must clip at black and white because of physical device limitations. Thus, especially for text rendering, only a reconstruction filter with nonnegative support is generally useful. Such a reconstruction filter would be an extremely poor choice for audio. It is true that many of the underlying signal processing principles are the same, and I encourage people to learn and understand those :)
- shawnz 7y agoThanks for the heads up. There is one specific section where he compares pixels to lollipop graphs and that is mainly what I was referring to, I didn't mean to suggest that all the principles in the video apply to graphics in the same way that they apply to audio.
- mktmkr 7y agoIs the intense background noise of that video part of some inside joke?
- pjs_ 7y agoA pixel is basically a little square dude, come on.
- BubRoss 7y agoNo, your pixels are displayed using little squares (sort of).
- jcelerier 7y ago> A Pixel Is Not a Little Square! Microsoft Tech Memo 6 is this the reason why they forced this horrible bilinear filter on the windows image viewer for so long (guess they still do) ? It made me crazy, it's so ugly
- BubRoss 7y agoNo, this was written by Alvy Ray Smith 25 years ago in response to incorrect filtering in computer graphics in general. This has nothing to do with whatever you are seeing.
- jmull 7y agoWell, it's an overloaded term (like pretty much every term in software development). But there's good information in this article. In many contexts you should be thinking in terms of point-samples, not squares or other areas. I think it's not such a great idea to try to simplify the definition of pixel in this article because it distracts from the useful info.
- draw_down 7y agoSure it is!
- robbrown451 7y agoI wrote this on Quora several years ago, answering the questions of why pixels are square (and quickly clarified that I interpreted it as "why are they laid out on a rectangular grid?"): https://www.quora.com/Why-are-computer-pixels-square/answer/Rob-Brown-13 https://www.quora.com/Why-are-computer-pixels-square/answer/... The more I've thought about it since, the more I think we should represent images with hexagonal pixels (i.e., laid out on a hex grid), and that color images should treat the center points of red, green, and blue subpixels as not being right on top of each other. (the third image in the post shows how they would be arranged, which is actually similar to how they are on some displays) It would be a little harder to deal with for graphics programmers who are working at a pixel level, or at the least, it would require a bit of relearning. But it makes more sense in so many ways. Hex is just a better way of "circle packing" (as you notice if you arrange a bunch of pennies on a table), and of course real world displays tend to have the red, green and blue subpixels offset from one another anyway. (are there any that don't?) Obviously it isn't easy to change something like this at this point, but still, I find the idea fascinating, and appealing to the OCD efficiency fan in me.
- osamagirl69 7y agoYou might want to look into pentile displays. You are correct in that they provide higher resolution than rectilinear displays (see the controversy tab in [1]), but they have largely fallen out of style because it is impossible to draw a straight line so everything looks blurry upon close inspection. One cool trick is that you can sprinkle in more efficient white pixels to allow the display to reduce its power consumption. [1] https://en.wikipedia.org/wiki/PenTile_matrix_family https://en.wikipedia.org/wiki/PenTile_matrix_family
- want2know 7y agoThe same is true for digital audio. We are so used to see the visual presentation of samples that look like a bar diagram, that a lot of people think analog sounds better because the curves are smoother. Chris Montgomery has a great talk about this.
- nothis 7y ago>We are so used to see the visual presentation of samples that look like a bar diagram, that a lot of people think analog sounds better because the curves are smoother. Except for a philosophical debate about continuity, isn't that true?
- arundelo 7y agohttp://productionadvice.co.uk/no-stair-steps-in-digital-audio/ http://productionadvice.co.uk/no-stair-steps-in-digital-audi... The “stair-steps” you see in your DAW when you zoom up on a digital waveform only exist inside the computer. [...] When digital audio is played back in the Real World, the reconstruction filter doesn’t reproduce those stair-steps – and the audio becomes truly analogue again.
- tzakrajs 7y agoYeah, it's strange people are thinking the display of a spectrum analyzer is somehow 1:1 with the underlying thing which they are measuring. As if a digital clock with only the hour and minutes displayed implies that seconds don't exist.
- planteen 7y agoNo, you put a low pass filter on the other side of the DAC and it looks equally smooth. Those stair steps are high frequency. This is the filter is called the image rejection or reconstruction filter. Once you learn about the frequency domain, you realize how silly most digital versus analog debates are and how very few really understand what is going on.
- mhh__ 7y ago
- bjourne 7y agoA pixel is a picture element. An element of a picture. Hence the name... It turns out that thinking of them as little boxes arranged in rectangular grids is very useful. Because that is how computers deal with them. Not as point samples. The article reminds me of the many mathematical text I've read insisting on that vectors are not tuples of numbers. That thinking of them as anything other than directions with magnitudes is wrong. Technically, that might be correct but vectors-as-numbers is much more useful when calculating with them. When you get into more abstract mathematics, and your vectors contain other kinds of algebraic objects, such as polynomials, you are already so accustomed with them that you can think of them as flying burritos if you like. When I teach graphics programming, I will continue to tell students that pixels are like little boxes.
- msla 7y ago> The article reminds me of the many mathematical text I've read insisting on that vectors are not tuples of numbers. That thinking of them as anything other than directions with magnitudes is wrong. I break every mathematical object down into three things: 1. The intuition. Why do we have this concept to begin with? What underlying idea are we trying to capture? 2. The definition. These are the axioms. 3. The implementation. This includes every way to communicate the idea, from natural language words to notation to source code. Without the intuition, you have nothing but a symbol game. It's hollow. Something with rules and notation but no deeper intuition is, arguably, chess. Without the definitions, you can't think rigorously about your ideas and you don't know if they lead to internal contradiction. You can dump the axioms without losing the intuition; we did this with set theory at the turn of the previous century, when Russell proved that the previous axioms were inconsistent. We saved set theory without having to abandon the notion, the intuition, of sets entirely. Without implementation, it's just thought, and you can't communicate with anyone. Moreover, without some intuition, the implementation is meaningless, because you have no cognitive frame to use to interpret it. So the tuple of numbers is one implementation of a vector. It allows you to communicate some aspects of a vector, but without the underlying idea of what a vector means, what concept we're trying to get across, it's just a list of numbers. They might as well be box scores or something.
- 7y ago
- dahart 7y agoThis piece is a classic and a must-read for graphics people, but do remember that this was written before LCD displays. Today’s pixels actually are little squares to a much greater degree than CRTs in 1995. That doesn’t change the theory or truth in Alvy Ray’s paper, but it does mean that the perfect reconstruction isn’t the same now that it was then.
- PorterDuff 7y agoHeck, if you've written color film recorder software, they're not even particularly square
- saboot 7y agoThis is very relevant for image reconstruction for medical imaging systems. When determining how much a beam of radiation is attenuated through an image, it really matters what the representation you use when translating a 2D matrix of values of material parameters, to an image representation. Are they boxes, trapezoids (bilinear interpolation), 2D gaussians, spheres? Each technique has some drawbacks, but for getting down to millimeter precision in scans it matters.
- pornel 7y agoHowever, image is not a wave. Sampling theorem applies beautifully to audio, but only to a limited degree to images. Some filters make sense in frequency domain, eyes are sensitive to certain frequencies more than others, but it all breaks down on hard edges, which don't behave like square waves. The problem is that in images the Gibb's effect is visible and annoying (ringing artifact). If sampling theorem applied, people wouldn't be able to see it, like they can't hear the difference between square waves shifted by a half of a sample.
- nullc 7y agoThis is mostly only true because our display technology lacks the resolution to reliably saturate the visual system in the same way that our audio technology does.
- kazinator 7y agoPixel and voxel are not commensurable. A pixel is a display hardware concept. A voxel is more comparable to a texel. https://en.wikipedia.org/wiki/Texel_(graphics) https://en.wikipedia.org/wiki/Texel_(graphics) Texels and Voxels can be square/cubic, such as in video game applications where it is accepted and exploited as a fundamental esthetic: worlds are textured with tiled mosaics which reveal their square unit when approached closely, and ditto for worlds made of voxels. A voxel as a sample of a solid, for instance from a computed tomography, where the fidelity of the reconstruction matters, is subject to different requirements.
- jacobolus 7y agoBoth the word “pixel” and the word “voxel” are commonly used in both of these different senses. I believe the word “voxel” was created to be explicitly a 3D analog of a 2D “pixel”. For example, you can also have “pixel art”, drawings made up of little squares, which only loosely have to do with pixels as samples for a raster image or physical camera detector elements or physical display elements. https://en.wikipedia.org/wiki/Pixel_art https://en.wikipedia.org/wiki/Pixel_art
- ctdonath 7y agoI remember the fanfare & hoopla around the introduction of the IBM PS/2 featuring square pixels. Made computing shapes much easier.
- deleted 7y ago[deleted]
- stretchwithme 7y agoOf course not. It's a little teacup.
- perrygeo 7y ago> [title repeats 3 times] > rid the world of the misconception that a pixel is a little geometric square. > The little square model is simply incorrect. It harms. > I show why it is wrong in general. Why the bluster? He makes a decent case for this point in the domain of graphics processing; what he calls "correct image (sprite) computing". But the narrow focus on computer graphics undermines these broad generalizations. There are many other domains that can be represented in pixel-based data models. Climate, terrain, population and land cover mapping are just a few domains where the use of a pixel as a "little geometric square" is a perfectly viable approach. Ultimately, if the message is "think about how your data model maps to reality" - I agree. But why the hyperbole? Why shit on an entire model because it doesn't fit for your very specific use case?