3 ms·

> In order to quantify Jpegli's image quality improvement we enlisted the help of crowdsourcing raters to compare pairs of images from Cloudinary Image Dataset

by bArray 2y ago

> In order to quantify Jpegli's image quality improvement we enlisted the help of crowdsourcing raters to compare pairs of images from Cloudinary Image Dataset '22, encoded using three codecs: Jpegli, libjpeg-turbo and MozJPEG, at several bitrates. Looking further [1]: > It consists in requiring a choice between two different distortions of the same image, and computes an Elo ranking (an estimate of the probability of each method being considered higher quality by the raters) of distortions based on that. Compared to traditional Opinion Score methods, it avoids requiring test subjects to calibrate their scores. This seems like a bad way to evaluate image quality. Humans can tend towards liking more highly saturated colours, which would be a distortion of the original image. If it was just a simple kernel that turned any image into a GIF cartoon, and then I had it rated by cartoon enthusiasts, I'm sure I could prove GIF is better than JPEG. I think that to produce something more fair, it would need to be "Given the following raw image, which of the following two images appears to better represent the above image?" The allowed answers should be "A", "B" and "unsure". ELO would likely be less appropriate. I would also like to see an analysis regarding which images were most influential in deciding which approach is better and why. Is it colour related, artefact related, information frequency related? I'm sure they could gain some deeper insight into why one method is favoured over the other. [1] https://github.com/google-research/google-research/blob/master/mucped23/README.md https://github.com/google-research/google-research/blob/mast...

Permik 2y ago
One other thing to control for is the subpixel layout of their display which is almost always forgotten in these studies.
- bArray 2y ago
  I did think about this - but then I thought the variation in displays/monitors and people would enhance the experiment.
zond 2y ago
The next sentence says "The test subject is able to flip between the two distortions, and has the original image available on the side for comparison at all times.", which indicates that the subjects weren't shown only the distortions.
- bArray 2y ago
  The change was literally just made: https://github.com/google-research/google-research/commit/4a841ea379145a5482517ad679a31fa08d794909 https://github.com/google-research/google-research/commit/4a... It appears this was in response to Hacker News comments.
geraldhh 2y ago
> Humans can tend towards liking more highly saturated colours, which would be a distortion of the original image. android with google photos did/does this whereas apple went with enhanced contrast. as far as i can tell, they're both wrong but one mostly notices the 'distortion' if used to the other.

by bArray 2y ago

Permik 2y ago

One other thing to control for is the subpixel layout of their display which is almost always forgotten in these studies.

bArray 2y ago
I did think about this - but then I thought the variation in displays/monitors and people would enhance the experiment.

zond 2y ago

The next sentence says "The test subject is able to flip between the two distortions, and has the original image available on the side for comparison at all times.", which indicates that the subjects weren't shown only the distortions.

bArray 2y ago
The change was literally just made: https://github.com/google-research/google-research/commit/4a841ea379145a5482517ad679a31fa08d794909 https://github.com/google-research/google-research/commit/4a... It appears this was in response to Hacker News comments.

geraldhh 2y ago

> Humans can tend towards liking more highly saturated colours, which would be a distortion of the original image. android with google photos did/does this whereas apple went with enhanced contrast. as far as i can tell, they're both wrong but one mostly notices the 'distortion' if used to the other.