3 ms·

Excellent video, though I do wonder about the treatment of pixels as point samples. I've seen it before, and it appears to be a popular viewpoint among code des

by sesqu 14y ago

Excellent video, though I do wonder about the treatment of pixels as point samples. I've seen it before, and it appears to be a popular viewpoint among code designers, but it doesn't agree with how I use and create pixels, or with my understanding of how they are sampled by cameras. So it feels like a convenient misrepresentation. Given that, I also wonder about the point sample nature of acoustic signals, espoused in this video. I know nothing about the sampling, but imagine a stairstep function could be more accurate for that, just reasonably represented as point samples transformed with the use of theoretical understanding of sound. The square wave example provides a further interesting representational quandary. Given that it's a simple function, the bandlimiting results in a surprisingly large signal loss at a notable increase in bit depth. Combined with the signal loss that results from assuming the frequencies a priori, I wonder if there is enough signal lost and converted to noise in the digitization process to spend processing power looking for non-sinusoidal, perhaps merely skewed, signals.

mpyne 14y ago
The math treats pixels as point samples, because that's what sampling is. You can quibble about how a CCD cell in a camera converts its narrow field-of-view into its output but at the conclusion of the snapshot you will have a finite set of data points from that cell (the "pixel"), nothing more or less. His point is that after this sampling process all you can say about the original data is that the cell at X,Y recorded a value of RRGGBB (expand to as many bits as appropriate for your camera). That is a "point sample" by definition: An ordered pair of [ Point, Sample-Value ]. It doesn't matter how special your code that uses or creates the pixels, this is a simple fact-of-life of the physical sampling process itself. Given that you admittedly know nothing about the sampling, I would imagine that you would be careful about how you judge the adequacy of the fit for a mapping from the sampled data to arbitrary functions intersecting with the sample.
- sesqu 14y ago
  Fair enough wrt/ point samples. But I didn't see his point as being that; I saw him arguing that pixels are correctly interpreted as representing an area of zero, which is very convenient when you're fitting a function to interpolate between them (since that way you don't have to correct for integrals). I would argue that this approach is what gets you JPEG artifacts, and that there might be an analogous effect in audio to explore, since the bitrate avenue has been exhausted. It's perfectly possible that more accurate bases for the decomposition step would result in unjustifiable bitrates, but I want to hear of it. As for judging the fit, my aim is to judge the mapping from input to output, not from sample to output (the ideal mapping from sample to output would of course be an identity function). This naturally requires recognizing some assumptions.
  - chipsy 14y ago
    JPEG artifacts are unrelated to the definition of pixels. They're an artifact of using lossy compression, much as how MP3 exhibits some distinctive artifacts at lower bitrates. Monty is only talking about plain old uncompressed sampled data. Since you're focused on the "size of a pixel" you might also be thinking of resampling artifacts, which is also beyond the scope of the video. Resampling is the process of changing the effective sample rate of data by reconstructing a new, larger or smaller set of samples from the original data. Monty alludes to this when he says "the samples are not stairstepped", because when we go to reconstruct samples we have to figure out a method that is NOT stairstepped, but rather: 1. Reasonably represents the original signal 2. Remains band-limited Fortunately there's theory that explains what a mathematically ideal resampler should look like. It's good study material for thinking about sampling theory in more depth and understanding some outcomes that develop from defining samples as infinitely small points. It also explains why pixel art doesn't respond well to standard image resampling algorithms.
    - sesqu 14y ago
      JPEG artifacts are separate from the definition of a pixel, yes, but not to the extent you propose. The artifacts are specifically a result of pixels becoming corrupted during the compress-decompress cycle, and feature most prominently where the source function has discontinuities. Treating a pixel as a point sample of a smooth function removes our ability to consider such things. Quantization is a form of lossy compression, which is why it adds noise. Monty certainly spoke of that, and of randomization as a recourse in the face of systematic error. I wasn't thinking about resampling, no, but it certainly fits the topic. Could you expand on how the ideal resampler explains why standard resamplers are occasionally poor, and whether this can be affected by treating the sample as filtered?
      - mpyne 14y ago
        > The artifacts are specifically a result of pixels becoming corrupted during the compress-decompress cycle, and feature most prominently where the source function has discontinuities. Treating a pixel as a point sample of a smooth function removes our ability to consider such things. The pixels are not "corrupted", the source pixels are instead deliberately replaced by a pixel set that compresses better. The way the JPEG encoder chooses this replacement pixel set is why you see the artifacts in areas with significant edges: JPEG deliberately removes the image counterpart to high-frequency data, much the same way that MPEG audio encoding removes high-frequency audio. What you see is the "Gibbs effect" in the visual realm from deliberately throwing away that high frequency data, which is not at all due to "treating a pixel as a point sample" (since as before, that's the very definition of a pixel, a value at a point). The solution is easy enough: force JPEG to keep the extra data, by driving the quality to the maximum permitted level (at the penalty of horrible compression). JPEG has other artifacts too (e.g. you can see the block size used by the encoder), but those are also not inherent to treating pixels as point samples but are instead inherent to the design of the encoder.
ssp 14y ago
A camera might produce a pixel by computing the average light input over a square, and in that sense you could consider the pixel itself a square. But in signal processing you interpret it as a two step procedure: First, for every mathematical point of the scene, compute the average light input of the surrounding square to produce a band limited signal. Second, for every pixel report the value at the corresponding point in the band limited image. The first step is filtering (in this case with a box filter), the second is point sampling.

by sesqu 14y ago

mpyne 14y ago

The math treats pixels as point samples, because that's what sampling is. You can quibble about how a CCD cell in a camera converts its narrow field-of-view into its output but at the conclusion of the snapshot you will have a finite set of data points from that cell (the "pixel"), nothing more or less. His point is that after this sampling process all you can say about the original data is that the cell at X,Y recorded a value of RRGGBB (expand to as many bits as appropriate for your camera). That is a "point sample" by definition: An ordered pair of [ Point, Sample-Value ]. It doesn't matter how special your code that uses or creates the pixels, this is a simple fact-of-life of the physical sampling process itself. Given that you admittedly know nothing about the sampling, I would imagine that you would be careful about how you judge the adequacy of the fit for a mapping from the sampled data to arbitrary functions intersecting with the sample.

sesqu 14y ago
Fair enough wrt/ point samples. But I didn't see his point as being that; I saw him arguing that pixels are correctly interpreted as representing an area of zero, which is very convenient when you're fitting a function to interpolate between them (since that way you don't have to correct for integrals). I would argue that this approach is what gets you JPEG artifacts, and that there might be an analogous effect in audio to explore, since the bitrate avenue has been exhausted. It's perfectly possible that more accurate bases for the decomposition step would result in unjustifiable bitrates, but I want to hear of it. As for judging the fit, my aim is to judge the mapping from input to output, not from sample to output (the ideal mapping from sample to output would of course be an identity function). This naturally requires recognizing some assumptions.
- chipsy 14y ago
  JPEG artifacts are unrelated to the definition of pixels. They're an artifact of using lossy compression, much as how MP3 exhibits some distinctive artifacts at lower bitrates. Monty is only talking about plain old uncompressed sampled data. Since you're focused on the "size of a pixel" you might also be thinking of resampling artifacts, which is also beyond the scope of the video. Resampling is the process of changing the effective sample rate of data by reconstructing a new, larger or smaller set of samples from the original data. Monty alludes to this when he says "the samples are not stairstepped", because when we go to reconstruct samples we have to figure out a method that is NOT stairstepped, but rather: 1. Reasonably represents the original signal 2. Remains band-limited Fortunately there's theory that explains what a mathematically ideal resampler should look like. It's good study material for thinking about sampling theory in more depth and understanding some outcomes that develop from defining samples as infinitely small points. It also explains why pixel art doesn't respond well to standard image resampling algorithms.
  - sesqu 14y ago
    JPEG artifacts are separate from the definition of a pixel, yes, but not to the extent you propose. The artifacts are specifically a result of pixels becoming corrupted during the compress-decompress cycle, and feature most prominently where the source function has discontinuities. Treating a pixel as a point sample of a smooth function removes our ability to consider such things. Quantization is a form of lossy compression, which is why it adds noise. Monty certainly spoke of that, and of randomization as a recourse in the face of systematic error. I wasn't thinking about resampling, no, but it certainly fits the topic. Could you expand on how the ideal resampler explains why standard resamplers are occasionally poor, and whether this can be affected by treating the sample as filtered?
    - mpyne 14y ago
      > The artifacts are specifically a result of pixels becoming corrupted during the compress-decompress cycle, and feature most prominently where the source function has discontinuities. Treating a pixel as a point sample of a smooth function removes our ability to consider such things. The pixels are not "corrupted", the source pixels are instead deliberately replaced by a pixel set that compresses better. The way the JPEG encoder chooses this replacement pixel set is why you see the artifacts in areas with significant edges: JPEG deliberately removes the image counterpart to high-frequency data, much the same way that MPEG audio encoding removes high-frequency audio. What you see is the "Gibbs effect" in the visual realm from deliberately throwing away that high frequency data, which is not at all due to "treating a pixel as a point sample" (since as before, that's the very definition of a pixel, a value at a point). The solution is easy enough: force JPEG to keep the extra data, by driving the quality to the maximum permitted level (at the penalty of horrible compression). JPEG has other artifacts too (e.g. you can see the block size used by the encoder), but those are also not inherent to treating pixels as point samples but are instead inherent to the design of the encoder.

ssp 14y ago

A camera might produce a pixel by computing the average light input over a square, and in that sense you could consider the pixel itself a square. But in signal processing you interpret it as a two step procedure: First, for every mathematical point of the scene, compute the average light input of the surrounding square to produce a band limited signal. Second, for every pixel report the value at the corresponding point in the band limited image. The first step is filtering (in this case with a box filter), the second is point sampling.