3 ms·

I suspect you're misreading "realistic quality" as "good quality". I mean "realistic" as opposite of "unrealistic", i.e. a quality level that you'd never want t

by pornel 5y ago

I suspect you're misreading "realistic quality" as "good quality". I mean "realistic" as opposite of "unrealistic", i.e. a quality level that you'd never want to use in practice. Don't benchmark with a quality level that is stupid/useless/obviously inappropriate for your usage of images that you actually want. It's almost a tautology, so I'm really surprised it's controversial.

zamadatix 5y ago
My disagreement is that there is such a thing as a realistic or unrealistic quality one could use for images in the first place as if there is some very limited SSIM range one isn't supposed to want to use image codecs outside of that would result in irrelevant benchmark data. The blame being attributed to this actually belongs to extrapolating from a single test case and assuming it implies something about all the test cases as opposed to having run an invalid test case. Particularly when referring to the cases of people assuming how a codec performs at lowest quality implies something about the higher quality comparisons. The issue here was never about the lowest quality options being unrealistic it's about assuming results from one test were directly correlated to what another would show If I were to replace the section with where my thoughts on it are it'd probably be titled something along the lines of "compare across a full spectrum of quality" and detail why you want to avoid extrapolation from a small range of test scenarios instead of avoiding some range completely.
- pornel 5y ago
  So I think your argument is that a single quality level is a single datapoint, and therefore not enough to say "codec A is better than codec B on average". That is technically true, but I'm not even suggesting to attempt to draw such conclusion, because (apart from an easy case of a Pareto improvement) such total order may not even exist. It can be both true that codec A>B and codec B>A at the same time. Therefore no matter how much data you throw at such comprehensive comparison, and how meticulously you aggregate it, it's going to be false/misleading in some way. There can be a "best codec for smushed thumbnails" and a "best codec for sharp icons" and a "best codec for archiving scanned film", but if you average over these categories, the "overall best codec" may be inferior at all of these. So while comparing images at one quality level doesn't generalize to "codec A is overall better than codec B", it does give you a more accurate "codec A is better than codec B for the quality I'm planning to use".

by pornel 5y ago

zamadatix 5y ago

My disagreement is that there is such a thing as a realistic or unrealistic quality one could use for images in the first place as if there is some very limited SSIM range one isn't supposed to want to use image codecs outside of that would result in irrelevant benchmark data. The blame being attributed to this actually belongs to extrapolating from a single test case and assuming it implies something about all the test cases as opposed to having run an invalid test case. Particularly when referring to the cases of people assuming how a codec performs at lowest quality implies something about the higher quality comparisons. The issue here was never about the lowest quality options being unrealistic it's about assuming results from one test were directly correlated to what another would show If I were to replace the section with where my thoughts on it are it'd probably be titled something along the lines of "compare across a full spectrum of quality" and detail why you want to avoid extrapolation from a small range of test scenarios instead of avoiding some range completely.

pornel 5y ago
So I think your argument is that a single quality level is a single datapoint, and therefore not enough to say "codec A is better than codec B on average". That is technically true, but I'm not even suggesting to attempt to draw such conclusion, because (apart from an easy case of a Pareto improvement) such total order may not even exist. It can be both true that codec A>B and codec B>A at the same time. Therefore no matter how much data you throw at such comprehensive comparison, and how meticulously you aggregate it, it's going to be false/misleading in some way. There can be a "best codec for smushed thumbnails" and a "best codec for sharp icons" and a "best codec for archiving scanned film", but if you average over these categories, the "overall best codec" may be inferior at all of these. So while comparing images at one quality level doesn't generalize to "codec A is overall better than codec B", it does give you a more accurate "codec A is better than codec B for the quality I'm planning to use".