5 ms·

Studying Lossy Image Compression Efficiency

66 points by dmmalam 13y ago

ralfd 13y ago
Discussion on reddit: http://www.reddit.com/r/programming/comments/1ottdn/mozilla_releases_study_on_lossy_image_compression http://www.reddit.com/r/programming/comments/1ottdn/mozilla_... I sometimes wonder why something gains traction on reddit and not on hacker news or vice versa.
joosters 13y ago
When comparing against JPEG, it's always worth pointing out that most JPEG compressed images aren't optimal and you can reduce their size further with no quality loss - http://www.kokkonen.net/tjko/projects.html http://www.kokkonen.net/tjko/projects.html jpegoptim will improve the compression in almost all cases. This highlights a big problem with work on better image compression - we can already do better, but no-one bothers. If you're trying to introduce a whole new codec, you've got a big hurdle to jump.
- gillianseed 13y ago
  >If you're trying to introduce a whole new codec, you've got a big hurdle to jump. Agreed, I think the big hurdle is simply that jpeg is 'good enough' for web use, and that a transition to a new standard lossy image format just isn't 'worth the hassle' so to speak. As for webp, I personally use it for lossless encoding instead of PNG on personal files these days, but I don't really use it for lossy as I don't think it makes a big enough difference versus jpg and jpg is supported everywhere. That said the current webp implementation is based upon VP8, so I suppose the next implementation will be based upon VP9 which would likely make it competitive with the HEVC based image codec, and unlike the HEVC based one it isn't patent/royalty encumbered.
  - ZeroGravitas 13y ago
    The WebP team said they wouldn't revise to match VP9. The improvements mostly didn't apply to still images so it wasn't worth it.
    - brigade 13y ago
      Bigger transforms and more intra modes certainly apply to still images. JPEG's 8x8 transform is pretty small given how large images are nowadays, and VP8 only has a 4x4 transform.
      - ZeroGravitas 13y ago
        Yes, they mentioned that as one clear win: https://groups.google.com/a/webmproject.org/d/topic/webp-discuss/qmASWczhS0c/discussion https://groups.google.com/a/webmproject.org/d/topic/webp-dis...
- mistercow 13y ago
  Another potential problem is that JPEG encoding and decoding is very mature. I remember this being an issue for color spaces JPEG 2000 in earlier versions of OS X (no idea about more recent versions). No matter what you did as a developer, it never seemed to be possible to get colors to look the same way once they'd been passed through the system-provided JPEG 2000 encoder. If you get all modern browsers to agree to support a new codec, but they all render them slightly differently, nobody is going to use it.
kibwen 13y ago
(Warning, unsourced hearsay ahead!) Backstory here, gleaned from eavesdropping on Mozilla employees and Gecko engineers: Google and Facebook are leaning on Mozilla to implement WebP in Firefox, while Mozilla's been resisting due to (in their eyes) the lack of measurable improvement over JPEG. Furthermore, as joosters alludes to elsewhere in this thread, most JPEG libraries set the default compression level way too conservatively, and most users of the libraries don't bother to ever deviate from that default. Merely increasing the compression level from the default can yield a 10% decrease in image size without degrading quality whatsoever as measured by the standard metrics mentioned in the OP. But there's the rub: the standard metrics kinda suck! Nobody's yet come up with a really solid algorithm to compare the quality of different image formats and compression ratios. So all this research we're doing may very well be useless and misguided. :P (I'd love it if someone who actually knows what they're talking about can pitch in. Corrections welcomed!)
- RexRollman 13y ago
  I personally keep expecting to see a schism develop between Mozilla and Google, considering they compete on two fronts: web browsing and mobile OS. Hopefully, nothing more than just healthy competition will happen.
anon1385 13y ago
I was a bit surprised to see Mozilla admitting that HEVC exists without smearing it. If only they could be this impartial in their video comparisons. Instead we had to endure the embarrassing spectacle of them trying to argue that Theora was competitive with H.264[1]. [1] http://weblogs.mozillazine.org/asa/archives/2009/06/theora_video_vs.html http://weblogs.mozillazine.org/asa/archives/2009/06/theora_v...
- gsnedders 13y ago
  Mozilla has never made any statement that Theora is competitive with H.264; various people (both who work for Mozilla and around the community) have made such statements, but they don't speak for Mozilla.
tangue 13y ago
Mozilla misses one important of WebP : Alpha channel. Today PNG 24 is the only way to achieve transparency with a 24 bits palette, resulting in unnecessary huge files. As a webdesigner, WebP would replace PNG 24, not jpeg. Anyway, I have this strange impression that Mozilla is more akin to spent ressources dismissing the format rather than implementing it or finding an alternative.
- ZeroGravitas 13y ago
  They're aware of that as a startup they fund asked them to look at WebP again because it worked so well for their use-case of icons with alpha. I believe that is why they reopened the bug. It's not covered by this study though.
- jws 13y ago
  Years ago I wrote a little proof of concept burying an alpha channel in the JPEG application data area and pulling it back out in the client browser to render JPEG with alpha. It worked well, for some browsers. Now I expect it would work well on most of them, though I haven't looked at it in years. It was received with zero interest. That may mean no one needs JPEG with alpha. You can read more at http://jim.studt.net/jpeg-alpha/ http://jim.studt.net/jpeg-alpha/
  - ralfd 13y ago
    That is pretty nifty! I also liked your graphic explantation with the asrtronaut.
- HeyImAlex 13y ago
  >Today PNG 24 is the only way to achieve transparency with a 24 bits palette, resulting in unnecessary huge files. That's not necessarily true. If you only have completely-transparent and completely-opaque pixels color type 2 will work, and if your total unique colors (RGBA) number less than 256 you can still do plaetted.
- pornel 13y ago
  I've developed lossy PNG encoder that creates 3-4 times smaller PNG files: http://pngmini.com/lossypng.html http://pngmini.com/lossypng.html
clarry 13y ago
I think Mozilla is right to tread carefully and not commit to yet another format for a minor improvement. In my opinion our web clients are bloated enough as is, and they're only getting bigger. Once you add support for a format and it gains mainstream traction on the web, it'll be really hard to remove, and we'll be stuck with software that has to support so many things for reasons that do not make sense in the near future. As for the study, I have to admit I did not read it. But the first thing to note is that computed metrics (SSIM, PSNR) are not reliable approximations of perceived quality. Optimizing for either will produce a codec that pleases the computer, not the eye. This is why the tuning of image and video codecs for psychovisual effects is such a laborious, manual process -- you have to have a human viewer to assess the quality. In the long term I think it would be good if a new still-image format was derived from the intra portion of some reasonably advanced video codec, as they all beat JPEG handily. This is essentially what WebP is, but if I recall correctly, Google didn't really give VP8 the love it needed when they acquired On2. Instead they rushed with a "release early, release often" mentality, presumably hoping the community would jump the boat first and do the grunt work for them later (this was during html5 video wars when some thought Theora sucks and others thought h.264 is too encumbered -- VP8 was Google's resolution). The details are shady but the curious should search for the debate around early VP8 (I think the x264 folk did some extensive notes). Needless to say I think, VP8 (and WebP) wasn't and isn't the game changer Google would've hoped. To avoid patents, a reasonable choice would IMHO be to look at Xiph.Org's upcoming video codec Daala. Curiously Mozilla has the right man in house since Monty moved from Red Hat to Mozilla like, almost like yesterday... Daala however isn't quite ready yet as the Xiph guys are busy researching out new ideas and unicorns. But the results so far are very interesting to say the least; anyone interested should check out Monty's demo pages.[1] As others have said, JPEG is "good enough" for most of us, and even JPEG is used rather inefficiently, often with too high quality. So there's no compelling reason to push for a new format here-and-now. Give it some time and commit once there's something significant I say. In the future we'll thank ourselves for not having deployed all that junk over minor increments. [1] http://people.xiph.org/~xiphmont/demo/ http://people.xiph.org/~xiphmont/demo/
- pornel 13y ago
  Mozilla hasn't used PSNR. They've used PSNR-HVS-M, which is more sophisticated. In test of objective metrics SSIM was one of the best (0.8 spearman correlation with human scores) http://www.ponomarenko.info/tid2008.htm http://www.ponomarenko.info/tid2008.htm IHMO the study has been done very well. They've even eliminated potential differences coming from colorspace conversion and chroma subsampling. If anything, this study favors WebP that can't do full-resolution color.
darkstalker 13y ago
What happened to JPEG 2000?
- pornel 13y ago
  Patents, and weak results of wavelets.
pornel 13y ago
"JPEG-killers" we have are disappointingly unambitious. That's skating to where the puck was 20 years ago. They're not aiming to be RAW killers and OpenEXR killers. They're not making anything new possible, they're just polishing the same old thing. Photographers are increasingly avoiding JPEG because of limited dynamic range it has. WebP (due to its video heritage) has even lower dynamic range than JPEG and can't preserve full resolution of chroma channels. JPEG-killers add simple alpha channel that 17-year-old PNG has, but nobody even thought about adding alpha with blending modes (e.g additive, to be able to create lights, or per-channel to be able to represent subpixel AA). After all these years there still isn't a format that can save a Photoshop layer and represent non-trivially blended transparency correctly (closest you can get is images wrapped in SVG with filters, uhh).
- mistercow 13y ago
  >After all these years there still isn't a format that can save a Photoshop layer and represent non-trivially blended transparency correctly (closest you can get is images wrapped in SVG with filters, uhh). There are other ways you can hack around it. One is to save the JPEG at double width and save the alpha channel as a gray scale image on the right, and then use css masks for webkit, canvas for firefox, and CSS filters for IE to add the alpha channel when the page renders. A similar technique can be used to do transparent video: http://jakearchibald.com/scratch/alphavid/ http://jakearchibald.com/scratch/alphavid/ For better results, you actually want to pad the right side of the non-alpha image to a multiple of 8 so that you hit JPEG block boundaries. Then you repeat the edge pixels so that you don't get artifacts at the edges. Handling artifacts from transparent pixels turning non-transparent is a little trickier, but one strategy is to apply a gaussian blur (radius 8) to the original image, discard the alpha channel, and then fill all completely transparent pixels in the original with the blurred version.
mappu 13y ago
A few years back i compared SSIM on Lenna for a few image codecs: http://code.ivysaur.me/srv/ssim_0.png http://code.ivysaur.me/srv/ssim_0.png comparing Photoshop's JPEG encoder, imagemagick's jp2k encoder, WebP, and a single frame from x264 (which came out ahead at all file sizes, as expected). You have to be significantly better than JPEG to really make a compelling argument as the successor, though. IPv6 is undoubtedly better than IPv4 and the upgrade started over a decade ago.

66 points by dmmalam 13y ago

ralfd 13y ago

Discussion on reddit: http://www.reddit.com/r/programming/comments/1ottdn/mozilla_releases_study_on_lossy_image_compression http://www.reddit.com/r/programming/comments/1ottdn/mozilla_... I sometimes wonder why something gains traction on reddit and not on hacker news or vice versa.

joosters 13y ago

When comparing against JPEG, it's always worth pointing out that most JPEG compressed images aren't optimal and you can reduce their size further with no quality loss - http://www.kokkonen.net/tjko/projects.html http://www.kokkonen.net/tjko/projects.html jpegoptim will improve the compression in almost all cases. This highlights a big problem with work on better image compression - we can already do better, but no-one bothers. If you're trying to introduce a whole new codec, you've got a big hurdle to jump.

gillianseed 13y ago
>If you're trying to introduce a whole new codec, you've got a big hurdle to jump. Agreed, I think the big hurdle is simply that jpeg is 'good enough' for web use, and that a transition to a new standard lossy image format just isn't 'worth the hassle' so to speak. As for webp, I personally use it for lossless encoding instead of PNG on personal files these days, but I don't really use it for lossy as I don't think it makes a big enough difference versus jpg and jpg is supported everywhere. That said the current webp implementation is based upon VP8, so I suppose the next implementation will be based upon VP9 which would likely make it competitive with the HEVC based image codec, and unlike the HEVC based one it isn't patent/royalty encumbered.
- ZeroGravitas 13y ago
  The WebP team said they wouldn't revise to match VP9. The improvements mostly didn't apply to still images so it wasn't worth it.
  - brigade 13y ago
    Bigger transforms and more intra modes certainly apply to still images. JPEG's 8x8 transform is pretty small given how large images are nowadays, and VP8 only has a 4x4 transform.
    - ZeroGravitas 13y ago
      Yes, they mentioned that as one clear win: https://groups.google.com/a/webmproject.org/d/topic/webp-discuss/qmASWczhS0c/discussion https://groups.google.com/a/webmproject.org/d/topic/webp-dis...
mistercow 13y ago
Another potential problem is that JPEG encoding and decoding is very mature. I remember this being an issue for color spaces JPEG 2000 in earlier versions of OS X (no idea about more recent versions). No matter what you did as a developer, it never seemed to be possible to get colors to look the same way once they'd been passed through the system-provided JPEG 2000 encoder. If you get all modern browsers to agree to support a new codec, but they all render them slightly differently, nobody is going to use it.

kibwen 13y ago

(Warning, unsourced hearsay ahead!) Backstory here, gleaned from eavesdropping on Mozilla employees and Gecko engineers: Google and Facebook are leaning on Mozilla to implement WebP in Firefox, while Mozilla's been resisting due to (in their eyes) the lack of measurable improvement over JPEG. Furthermore, as joosters alludes to elsewhere in this thread, most JPEG libraries set the default compression level way too conservatively, and most users of the libraries don't bother to ever deviate from that default. Merely increasing the compression level from the default can yield a 10% decrease in image size without degrading quality whatsoever as measured by the standard metrics mentioned in the OP. But there's the rub: the standard metrics kinda suck! Nobody's yet come up with a really solid algorithm to compare the quality of different image formats and compression ratios. So all this research we're doing may very well be useless and misguided. :P (I'd love it if someone who actually knows what they're talking about can pitch in. Corrections welcomed!)

RexRollman 13y ago
I personally keep expecting to see a schism develop between Mozilla and Google, considering they compete on two fronts: web browsing and mobile OS. Hopefully, nothing more than just healthy competition will happen.

anon1385 13y ago

I was a bit surprised to see Mozilla admitting that HEVC exists without smearing it. If only they could be this impartial in their video comparisons. Instead we had to endure the embarrassing spectacle of them trying to argue that Theora was competitive with H.264[1]. [1] http://weblogs.mozillazine.org/asa/archives/2009/06/theora_video_vs.html http://weblogs.mozillazine.org/asa/archives/2009/06/theora_v...

gsnedders 13y ago
Mozilla has never made any statement that Theora is competitive with H.264; various people (both who work for Mozilla and around the community) have made such statements, but they don't speak for Mozilla.

tangue 13y ago

Mozilla misses one important of WebP : Alpha channel. Today PNG 24 is the only way to achieve transparency with a 24 bits palette, resulting in unnecessary huge files. As a webdesigner, WebP would replace PNG 24, not jpeg. Anyway, I have this strange impression that Mozilla is more akin to spent ressources dismissing the format rather than implementing it or finding an alternative.

ZeroGravitas 13y ago
They're aware of that as a startup they fund asked them to look at WebP again because it worked so well for their use-case of icons with alpha. I believe that is why they reopened the bug. It's not covered by this study though.
jws 13y ago
Years ago I wrote a little proof of concept burying an alpha channel in the JPEG application data area and pulling it back out in the client browser to render JPEG with alpha. It worked well, for some browsers. Now I expect it would work well on most of them, though I haven't looked at it in years. It was received with zero interest. That may mean no one needs JPEG with alpha. You can read more at http://jim.studt.net/jpeg-alpha/ http://jim.studt.net/jpeg-alpha/
- ralfd 13y ago
  That is pretty nifty! I also liked your graphic explantation with the asrtronaut.
HeyImAlex 13y ago
>Today PNG 24 is the only way to achieve transparency with a 24 bits palette, resulting in unnecessary huge files. That's not necessarily true. If you only have completely-transparent and completely-opaque pixels color type 2 will work, and if your total unique colors (RGBA) number less than 256 you can still do plaetted.
pornel 13y ago
I've developed lossy PNG encoder that creates 3-4 times smaller PNG files: http://pngmini.com/lossypng.html http://pngmini.com/lossypng.html

clarry 13y ago

I think Mozilla is right to tread carefully and not commit to yet another format for a minor improvement. In my opinion our web clients are bloated enough as is, and they're only getting bigger. Once you add support for a format and it gains mainstream traction on the web, it'll be really hard to remove, and we'll be stuck with software that has to support so many things for reasons that do not make sense in the near future. As for the study, I have to admit I did not read it. But the first thing to note is that computed metrics (SSIM, PSNR) are not reliable approximations of perceived quality. Optimizing for either will produce a codec that pleases the computer, not the eye. This is why the tuning of image and video codecs for psychovisual effects is such a laborious, manual process -- you have to have a human viewer to assess the quality. In the long term I think it would be good if a new still-image format was derived from the intra portion of some reasonably advanced video codec, as they all beat JPEG handily. This is essentially what WebP is, but if I recall correctly, Google didn't really give VP8 the love it needed when they acquired On2. Instead they rushed with a "release early, release often" mentality, presumably hoping the community would jump the boat first and do the grunt work for them later (this was during html5 video wars when some thought Theora sucks and others thought h.264 is too encumbered -- VP8 was Google's resolution). The details are shady but the curious should search for the debate around early VP8 (I think the x264 folk did some extensive notes). Needless to say I think, VP8 (and WebP) wasn't and isn't the game changer Google would've hoped. To avoid patents, a reasonable choice would IMHO be to look at Xiph.Org's upcoming video codec Daala. Curiously Mozilla has the right man in house since Monty moved from Red Hat to Mozilla like, almost like yesterday... Daala however isn't quite ready yet as the Xiph guys are busy researching out new ideas and unicorns. But the results so far are very interesting to say the least; anyone interested should check out Monty's demo pages.[1] As others have said, JPEG is "good enough" for most of us, and even JPEG is used rather inefficiently, often with too high quality. So there's no compelling reason to push for a new format here-and-now. Give it some time and commit once there's something significant I say. In the future we'll thank ourselves for not having deployed all that junk over minor increments. [1] http://people.xiph.org/~xiphmont/demo/ http://people.xiph.org/~xiphmont/demo/

pornel 13y ago
Mozilla hasn't used PSNR. They've used PSNR-HVS-M, which is more sophisticated. In test of objective metrics SSIM was one of the best (0.8 spearman correlation with human scores) http://www.ponomarenko.info/tid2008.htm http://www.ponomarenko.info/tid2008.htm IHMO the study has been done very well. They've even eliminated potential differences coming from colorspace conversion and chroma subsampling. If anything, this study favors WebP that can't do full-resolution color.

darkstalker 13y ago

What happened to JPEG 2000?

pornel 13y ago
Patents, and weak results of wavelets.

pornel 13y ago

"JPEG-killers" we have are disappointingly unambitious. That's skating to where the puck was 20 years ago. They're not aiming to be RAW killers and OpenEXR killers. They're not making anything new possible, they're just polishing the same old thing. Photographers are increasingly avoiding JPEG because of limited dynamic range it has. WebP (due to its video heritage) has even lower dynamic range than JPEG and can't preserve full resolution of chroma channels. JPEG-killers add simple alpha channel that 17-year-old PNG has, but nobody even thought about adding alpha with blending modes (e.g additive, to be able to create lights, or per-channel to be able to represent subpixel AA). After all these years there still isn't a format that can save a Photoshop layer and represent non-trivially blended transparency correctly (closest you can get is images wrapped in SVG with filters, uhh).

mistercow 13y ago
>After all these years there still isn't a format that can save a Photoshop layer and represent non-trivially blended transparency correctly (closest you can get is images wrapped in SVG with filters, uhh). There are other ways you can hack around it. One is to save the JPEG at double width and save the alpha channel as a gray scale image on the right, and then use css masks for webkit, canvas for firefox, and CSS filters for IE to add the alpha channel when the page renders. A similar technique can be used to do transparent video: http://jakearchibald.com/scratch/alphavid/ http://jakearchibald.com/scratch/alphavid/ For better results, you actually want to pad the right side of the non-alpha image to a multiple of 8 so that you hit JPEG block boundaries. Then you repeat the edge pixels so that you don't get artifacts at the edges. Handling artifacts from transparent pixels turning non-transparent is a little trickier, but one strategy is to apply a gaussian blur (radius 8) to the original image, discard the alpha channel, and then fill all completely transparent pixels in the original with the blurred version.

mappu 13y ago

A few years back i compared SSIM on Lenna for a few image codecs: http://code.ivysaur.me/srv/ssim_0.png http://code.ivysaur.me/srv/ssim_0.png comparing Photoshop's JPEG encoder, imagemagick's jp2k encoder, WebP, and a single frame from x264 (which came out ahead at all file sizes, as expected). You have to be significantly better than JPEG to really make a compelling argument as the successor, though. IPv6 is undoubtedly better than IPv4 and the upgrade started over a decade ago.