4 ms·

1. How else would you construct an objective test? Creating images of the same quality and then comparing their file sizes seems prohibitively difficult if no

by jfager 16y ago

1. How else would you construct an objective test? Creating images of the same quality and then comparing their file sizes seems prohibitively difficult if not impossible. This test, otoh, implies the same information (minus the exact difference in file size), but is pretty easy to set up. 2. Those were the reported numbers, but it's not like that was the only data point discussed in the article. Every single photo had deeper analysis and included subjective evaluation. He even directly made the point that you did about the last photo. 3. Loading megabytes of photo data into folks browsers by default doesn't seem worth the benefit (the page is already sluggish). If you're interested, the originals are available for download.

eru 16y ago
About 1: This test looks at the dual claim, and that's more or less justified. (Look up duality in linear optimization for the definition. It's useful.)
sp332 16y ago
There's more than one "objective" test for image quality, and some of them are closer to human perception than PSNR. The trouble with comparisons like this is that the JPG encoder might be tuned to minimize PSNR, but if the WebP encoder is tuned to optimize for something else, of course it won't perform as well when you compare the PSNR! More info on the issues with this type of benchmark from Jason Garrett-Glaser (a.k.a. Dark Shikari), an x264 developer: http://x264dev.multimedia.cx/?p=458 http://x264dev.multimedia.cx/?p=458
- jfager 16y ago
  That's fine, I agree. My point was not "His metric is the most awesome, shut up", it was "Despite only giving one kind of number, the author still included other analysis and subjective opinion in the writeup for each photo, so it's not as bad as you're trying to imply".
  - sp332 16y ago
    You asked How else would you construct an objective test? And my answer is, just make sure you're being fair in measuring the same thing the encoders are optimising for.
    - jfager 16y ago
      That was in response to the complaint that the post looked at images of the same size and compared their quality, rather than looking at images of the same quality and comparing their size. If you know of a practical way to objectively do the latter, please share.
    - scott_s 16y ago
      If I only test the optimal circumstances for a new thing, then I have done a terrible job at trying to understand the consequences of adopting it.
      - sp332 16y ago
        On two occasions I have been asked,—"Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" .... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. --Babbage (1864), Passages from the Life of a Philosopher, ch. 5 "Difference Engine No. 1" You can't reasonably use an encoder optimized for PSNR and then "ding" it for producing output with bad SSIM, or vice-versa.
        scott_s 16y ago
        I'm not familiar enough with the image encoding algorithms (and the tradeoffs they are faced with) to comment on that, specifically. But I've done enough research with systems in general to disagree with your statement. No matter what I optimize a system for, it's always fair for someone to test it under adverse conditions. If we're going to adopt something, we need to know its limitations. Further, my understanding of Google's intent with webp is that it is being offered as a replacement to jpeg. In that case, even if it's optimized for one thing, it's not just fair but necessary to see how it works under all other things.
lysium 16y ago
Well, if you test an algorithm that claims to have same image quality with smaller file size, you should compare images of the same quality and look at the files size, not the other way round. Besides, I don't see why you need megabytes of photo data, regular internet photo data would have sufficed (for which the authors say they have developed webp anyways). I agree with your point that he included subjective evaluation, though.

by jfager 16y ago

eru 16y ago

About 1: This test looks at the dual claim, and that's more or less justified. (Look up duality in linear optimization for the definition. It's useful.)

sp332 16y ago

There's more than one "objective" test for image quality, and some of them are closer to human perception than PSNR. The trouble with comparisons like this is that the JPG encoder might be tuned to minimize PSNR, but if the WebP encoder is tuned to optimize for something else, of course it won't perform as well when you compare the PSNR! More info on the issues with this type of benchmark from Jason Garrett-Glaser (a.k.a. Dark Shikari), an x264 developer: http://x264dev.multimedia.cx/?p=458 http://x264dev.multimedia.cx/?p=458

jfager 16y ago
That's fine, I agree. My point was not "His metric is the most awesome, shut up", it was "Despite only giving one kind of number, the author still included other analysis and subjective opinion in the writeup for each photo, so it's not as bad as you're trying to imply".
- sp332 16y ago
  You asked How else would you construct an objective test? And my answer is, just make sure you're being fair in measuring the same thing the encoders are optimising for.
  - jfager 16y ago
    That was in response to the complaint that the post looked at images of the same size and compared their quality, rather than looking at images of the same quality and comparing their size. If you know of a practical way to objectively do the latter, please share.
  - scott_s 16y ago
    If I only test the optimal circumstances for a new thing, then I have done a terrible job at trying to understand the consequences of adopting it.
    - sp332 16y ago
      On two occasions I have been asked,—"Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" .... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. --Babbage (1864), Passages from the Life of a Philosopher, ch. 5 "Difference Engine No. 1" You can't reasonably use an encoder optimized for PSNR and then "ding" it for producing output with bad SSIM, or vice-versa.
      - scott_s 16y ago
        I'm not familiar enough with the image encoding algorithms (and the tradeoffs they are faced with) to comment on that, specifically. But I've done enough research with systems in general to disagree with your statement. No matter what I optimize a system for, it's always fair for someone to test it under adverse conditions. If we're going to adopt something, we need to know its limitations. Further, my understanding of Google's intent with webp is that it is being offered as a replacement to jpeg. In that case, even if it's optimized for one thing, it's not just fair but necessary to see how it works under all other things.

lysium 16y ago

Well, if you test an algorithm that claims to have same image quality with smaller file size, you should compare images of the same quality and look at the files size, not the other way round. Besides, I don't see why you need megabytes of photo data, regular internet photo data would have sufficed (for which the authors say they have developed webp anyways). I agree with your point that he included subjective evaluation, though.