So I think your argument is that a single quality level is a single datapoint, and therefore not enough to say "codec A is better than codec B on average". Tha

So I think your argument is that a single quality level is a single datapoint, and therefore not enough to say "codec A is better than codec B on average". That is technically true, but I'm not even suggesting to attempt to draw such conclusion, because (apart from an easy case of a Pareto improvement) such total order may not even exist. It can be both true that codec A>B and codec B>A at the same time. Therefore no matter how much data you throw at such comprehensive comparison, and how meticulously you aggregate it, it's going to be false/misleading in some way. There can be a "best codec for smushed thumbnails" and a "best codec for sharp icons" and a "best codec for archiving scanned film", but if you average over these categories, the "overall best codec" may be inferior at all of these. So while comparing images at one quality level doesn't generalize to "codec A is overall better than codec B", it does give you a more accurate "codec A is better than codec B for the quality I'm planning to use".