3 ms·

Objective metrics have many weaknesses and pitfalls, but keep in mind that just eyeballing images is not necessarily better. Non-blind n=1 human test can be ju

by pornel 3y ago

Objective metrics have many weaknesses and pitfalls, but keep in mind that just eyeballing images is not necessarily better. Non-blind n=1 human test can be just as flawed. What you like in a particular scenario is not representative of codecs' overall performance. Testing with humans requires proper setup and a large sample size (which BTW JPEG XL has done!) The problem is that these codecs are close enough in performance to be below "noise" level of human judgement. You will not be able to reliably distinguish q=80 vs q=81 of the same codec, even though there is objectively a difference between them. And you can't lower quality to potato level to make your job easier — that changes the nature of the benchmark. People also just differ in opinion whether blurry without detail is better than detailed but with blocking or ringing. If you ask people to rate images, stddev of their scores will be pretty large, so wide that scores of objective metrics can fit in 1-sigma. People also tend to pick "nicer" image, rather than the one that is closer to the original. That's a bit different task than codecs aim for. Codecs can allocate more or less of the file to color channels, so you can get different conclusions based on e.g. amount of bright red in the image. So testing is hard. Plenty of pitfalls. Showing a smaller file that "looks the same" is easy, but deceptive.

bscphil 3y ago
Agreed that large double-blinded surveys with trained participants is pretty much the gold standard, and that there's no true objectivity about e.g. blurring vs blocking. > Showing a smaller file that "looks the same" is easy, but deceptive. It's much better to show same-sized files and let the viewer assess their quality (this is what the comparison I linked does), but there are deceptive ways to do even this. See also this great old blog post about doing comparisons for video codecs: https://web.archive.org/web/20141103202912/http://x264dev.multimedia.cx/archives/472 https://web.archive.org/web/20141103202912/http://x264dev.mu...
jorvi 3y ago
> People also tend to pick "nicer" image, rather than the one that is closer to the original. That's a bit different task than codecs aim for. Samsung turns their screens and phone cameras to “vivid” processing by default (they do provide a “natural” toggle, to their credit). I think over 90% of people don’t realize what they are seeing is not close to reality, and instead ask people with other phones, especially iPhones, why their screen or photo looks so washed out. There’s many faults to BFDLs, but I love Apple providing “vivid” processing as an option and “natural” as the default. Sometimes the masses just don’t know what’s actually good for them.
- ygra 3y ago
  The question is photos is whether people want something close to reality or would they rather have an exaggerated, arguably more beautiful version of reality to remember and share? I guess unless you're doing journalism, arguing for less processed photos may be a moot point, at least in phone cameras, as those are likely overwhelmingly often used for snapshots.
class4behavior 3y ago
Hm, if you zoom in you can clearly see the difference between jxl and avif. The problem is rather the inconsistency of the results. For instance, at medium quality Jxl seems to be better at preserving fine details and structure like the mark on the door, the traces on the lower part of the bridge, but avif appears to be better at preserving clarity of complex details, like far-away windows, cars, a tennis racket.
- YetAnotherNick 3y ago
  To me, AVIF seems to be better for preserving sharp edges, JPEG XL seems better for preserving textures.
  - FullstakBlogger 3y ago
    If you used nothing but nails to build various styles of beds, you wouldn't have the same experience sleeping on them as the originals; Unless, of course, the original was a bed of nails. Sharp edges are just one texture in an infinite range of textures, and AVIF looks like it constructs everything out of sharp edges in a way that's really obvious to the eye at all compression levels. With JPEG XL I can at least tell what's missing, or too artifacted to make out. With AVIF you have no idea what has been completely erased.
- pornel 3y ago
  Lossy codecs intentionally allow distortions that are too small to see with the naked eye (without zooming in). They're designed to operate at some "normal" viewing distance. If you zoom in, you defeat that technique. In case you actually wanted to compress images specifically for viewing when zoomed in, you should use different codecs, or higher quality, or configure codecs differently (e.g. in classic JPEG make quantization table preserve high frequences more). But for a benchmark that claims to compare codecs in general, you should only use normal viewing distance. Currently it's controversial whether the norm is still ~100dpi or whether it should be the "Retina"/2x resolution, but definitely it's not some zoomed-in 5dpi.

by pornel 3y ago

bscphil 3y ago

Agreed that large double-blinded surveys with trained participants is pretty much the gold standard, and that there's no true objectivity about e.g. blurring vs blocking. > Showing a smaller file that "looks the same" is easy, but deceptive. It's much better to show same-sized files and let the viewer assess their quality (this is what the comparison I linked does), but there are deceptive ways to do even this. See also this great old blog post about doing comparisons for video codecs: https://web.archive.org/web/20141103202912/http://x264dev.multimedia.cx/archives/472 https://web.archive.org/web/20141103202912/http://x264dev.mu...

jorvi 3y ago

> People also tend to pick "nicer" image, rather than the one that is closer to the original. That's a bit different task than codecs aim for. Samsung turns their screens and phone cameras to “vivid” processing by default (they do provide a “natural” toggle, to their credit). I think over 90% of people don’t realize what they are seeing is not close to reality, and instead ask people with other phones, especially iPhones, why their screen or photo looks so washed out. There’s many faults to BFDLs, but I love Apple providing “vivid” processing as an option and “natural” as the default. Sometimes the masses just don’t know what’s actually good for them.

ygra 3y ago
The question is photos is whether people want something close to reality or would they rather have an exaggerated, arguably more beautiful version of reality to remember and share? I guess unless you're doing journalism, arguing for less processed photos may be a moot point, at least in phone cameras, as those are likely overwhelmingly often used for snapshots.

class4behavior 3y ago

Hm, if you zoom in you can clearly see the difference between jxl and avif. The problem is rather the inconsistency of the results. For instance, at medium quality Jxl seems to be better at preserving fine details and structure like the mark on the door, the traces on the lower part of the bridge, but avif appears to be better at preserving clarity of complex details, like far-away windows, cars, a tennis racket.

YetAnotherNick 3y ago
To me, AVIF seems to be better for preserving sharp edges, JPEG XL seems better for preserving textures.
- FullstakBlogger 3y ago
  If you used nothing but nails to build various styles of beds, you wouldn't have the same experience sleeping on them as the originals; Unless, of course, the original was a bed of nails. Sharp edges are just one texture in an infinite range of textures, and AVIF looks like it constructs everything out of sharp edges in a way that's really obvious to the eye at all compression levels. With JPEG XL I can at least tell what's missing, or too artifacted to make out. With AVIF you have no idea what has been completely erased.
pornel 3y ago
Lossy codecs intentionally allow distortions that are too small to see with the naked eye (without zooming in). They're designed to operate at some "normal" viewing distance. If you zoom in, you defeat that technique. In case you actually wanted to compress images specifically for viewing when zoomed in, you should use different codecs, or higher quality, or configure codecs differently (e.g. in classic JPEG make quantization table preserve high frequences more). But for a benchmark that claims to compare codecs in general, you should only use normal viewing distance. Currently it's controversial whether the norm is still ~100dpi or whether it should be the "Retina"/2x resolution, but definitely it's not some zoomed-in 5dpi.