6 ms·

JPEG2000: Choices and Tradeoffs for Encoders (2004) [pdf]

30 points by nkurz 11y ago

jsingleton 11y ago
I did my final University project on an algorithm based on JPEG2000. It's a pretty interesting way of compressing images. One of the best things about using the Wavelet Transform over the Discrete Cosine Transform is it allows you to structure the file or stream from low to high frequency. This lets you get an image of any size out of a single file just by taking a subset of it. No more resizing thumbnails. It's kind of like a super version of progressive JPEG. Sadly it never took off as storage / bandwidth are cheap and JPEG is so entrenched (especially in hardware). It still has some niche applications in constrained environments though. Edit: I dug out the paper if anyone wants to read it: https://unop.uk/misc/university-final-year-project-cbir https://unop.uk/misc/university-final-year-project-cbir It's funny looking back at something you wrote over 8 years ago any seeing how you've improved. In other words, please don't use this to judge me! Abstract This project investigated the effects of scaling images with JPEG2000 emulation software on a CBIR system based on MPEG-7. A CBIR system was built that consisted of indexing and retrieval. The performance of the system was evaluated with standard metrics. Both JPEG2000 and MPEG-7 never really took off. Most people moved on to more advanced algorithms.
- varjag 11y ago
  My understanding is patent encumbering was what killed JPEG2000. There was an effort later to waive the patents, but not sure how comprehensive and perhaps a little too late. So it's the same reason noone uses theoretically superior arithmetic coding option in baseline JPEGs.
  - jsingleton 11y ago
    To some extent, yes. However, JPEG was also patented and that did pretty well. Someone made a lot of money on licensing. I think WebP stands a pretty good chance of supplanting JPEG on the web but it will still be used in lots of places. https://en.wikipedia.org/wiki/WebP https://en.wikipedia.org/wiki/WebP
    - ZeroGravitas 11y ago
      I don't think anyone really used the bit of JPEG that was flagged up initially as not being royalty-free (I think IBM owned that part?) and later on I think it was only patent trolls that tried (unsuccesfully) to licence JPEG. Which licencing on JPEG are you referring to?
      - jsingleton 11y ago
        I don't think it applied to the version used on the web. However, this is just from memory based on conversations I had with MPEG and JPEG board members a long time ago. I can't find any sources so I could be mistaken.
    - gillianseed 11y ago
      >However, JPEG was also patented and that did pretty well. I was under the impression that the free implementations (like libjpeg) did not use and of the patented techniques (such as arithmetic coding which patent recently expired) ? >I think WebP stands a pretty good chance of supplanting JPEG on the web but it will still be used in lots of places. I don't think anything will supplant JPEG as the de facto image format on the 'web', it's simply 'good enough' and of course free and supported everywhere. Also as for lossy compression I'm not sure WebP is a real improvement over JPEG, I'd love to see it used instead of PNG for lossless compression though, as here it is clearly superior.
    - varjag 11y ago
      > However, JPEG was also patented and that did pretty well. This is exactly the bit I was referring to (the arithmetic coding method). The rest of CCITT/ITU baseline spec was unencumbered, that's what everyone ended up using. Source: me, who implemented a baseline JPEG codec in late '90s.
    - acomjean 11y ago
      Did you mean GIF? Patents and compuserve/unisys attempt to make money off them basically pushed the development of PNG to replace it. https://en.wikipedia.org/wiki/GIF https://en.wikipedia.org/wiki/GIF
  - IshKebab 11y ago
    It was more that it didn't offer enough benefits. It was "the files are a bit smaller". Image files don't take up that much space anyway so that is less compelling than for video compression. What they needed to do is add more features, like WebP has now: * Single format for photos and diagrams * Transparency * Lossy and lossless in the same format * HDR * Tiling of large pictures Hopefully those features (especially lossy transparency) will help WebP do better than JPEG2000, though I'm not holding my breath. If IE and Firefox ever add support for it then I think web developers will start using it a lot.
    - varjag 11y ago
      > It was more that it didn't offer enough benefits. It was "the files are a bit smaller". Well it was more than that: it offered unlimited color depth for example, a big progress from 8 bit per channel. It was technically superior format in nearly every respect.
    - JupiterMoon 11y ago
      Anecdotes are not data but... It's taken over 10 years before I can safely send people png images rather than tiffs in my line of work. I'm not switching again until the benefits are blindingly huge and obvious. Basically I think you are correct as to why JPEG2000 did not take off.
    - PaulKeeble 11y ago
      Performance was also a big problem, they had a lot of issues in the beginning justifying the amount of time it took to encode and especially decode the images. IIRC it was of the 100x the time of JPEG and that meant that at speeds of the time the bandwidth saved was overshadowed by the decode time. I did a lot of working with it and had quite a lot of problems with performance of the sample software.
      - yoklov 11y ago
        Did you feel that this was an intrinsic property of the format, or just that less/not enough time had been spent on developing efficient implementations?
- acdha 11y ago
  > This lets you get an image of any size out of a single file just by taking a subset of it. No more resizing thumbnails. I always think of this during responsive image discussions – if JPEG-2000 had caught on, we could have solved that entire problem simply by adding an attribute to <img> telling the client which HTTP byte ranges it needs to request for various sizes. Single-file, cache-friendly, degrades well, and … will never happen. > Sadly it never took off as storage / bandwidth are cheap and JPEG is so entrenched (especially in hardware) JPEG proving to be impressively durable certainly was a key factor but I also think a lot of it comes down to naive moves by the community. In addition to the patent situation, there were just basic things like having a huge spec which for a long time was neither freely available nor accompanied by a public test suite. I work with librarians who have large scanned digital collections and it was definitely interesting to compare the reactions: the users disliked JPEG 2000 because they'd been conditioned to expect it to be quite noticeably slower, unsupported by some of their tools, and knew that they couldn't rely on any two programs actually working with a given set of files until they tested it (http://jpylyzer.openpreservation.org/ http://jpylyzer.openpreservation.org/ has helped catch that earlier). Meanwhile, the JPEG-2000 standards people & other boosters just tended to assume inevitability, where everyone would drop other formats once they saw how inferior they were. Things like Photoshop removing JPEG-2000 support (first discussed publicly in 2007, dropped in Elements in 2011) had this odd effect where some people expressed considerable concern (http://wiki.opf-labs.org/display/TR/JPEG+2000+support+discontinued+in+Photoshop+Elements http://wiki.opf-labs.org/display/TR/JPEG+2000+support+discon...) but others just assumed it was some sort of unrepresentative quirk. I cannot help but think that the situation would be considerably different if something like OpenJPEG.org had gotten significant support much earlier. Jasper, JJ2000, etc. were the most popular way to add support at all and they're slow, buggy and generally stagnant so many people had a bad first experience which encouraged avoiding the format.
  - saurik 11y ago
    > we could have solved that entire problem simply by adding an attribute to <img> telling the client which HTTP byte ranges it needs to request for various sizes You would never have a range start anywhere but the top of the file, as the whole point is that the later octaves are refinements of earlier ones. What makes JPEG2000 so amazing for this use case is precisely that you don't need to care about "ranges" (if you needed ranges, we might as well just have multiple files..). Instead, browsers just keep downloading the file until they either achieve the level of quality they need to render at the scale they currently care about (being able to continue later, if required) or run out of file. At worst, only as a "hint", I could see a benefit to having a table of sizes, but I feel like the cool solution for at would be to have an encoded version of the octave offsets, sort of like a zindex but for JPEG.
TorKlingberg 11y ago
It's interesting how JPEG2000 use has not really taken off outside of a few niche areas. Web browser vendors have been reluctant to implement it because of increases attack surface, and of course web sites will not use it without browser support. Patents may also be an issue. JPEG works mostly on 8x8 blocks of pixels, and in high resolution pictures each block is almost a single color. JPEG does a poor job taking advantage of similarity between the blocks. JPEG2000 does at least see some use in archival formats.
- anon1385 11y ago
  >Web browser vendors have been reluctant to implement it because of increases attack surface Which isn't a very convincing argument when they are quite happy to implement dozens of other things that vastly increase the attack surface of browsers but have far lower practical impact on the average web page. Imagine how much bandwidth has been wasted over the last 10 years because the web didn't have a lossy image format that supported transparency.
- captainmuon 11y ago
  > Web browser vendors have been reluctant to implement it because of increases attack surface, and of course web sites will not use it without browser support. I believe codecs should actually be part of the OS. There should be one good, well-tested implementation per OS that browsers and other apps just link to. It might actually be possible to sandbox the codec components for increased security. Ideally, a web browser would just be a HTML+stuff parser, a layout engine, and a bit of gui, and shell out to the OS for everything else. Unfortunately, it seems the monolithic kitchen sink model has won.
  - vetinari 11y ago
    The monolithic kitchen sink has won, because the OS bundled libraries have different APIs (not only among different OSes, but also among different versions - and OS vendors are interested in pushing the latest), so app vendors just said to themselves "screw it" and bundled library (like jpeglib or libpng) that has same API on all systems and versioning is under their control. That's the short version ;).
pornel 11y ago
JPEG2000 just isn't that great, and DCT-based compression is very hard to beat. DCT with varying block sizes gets most of the efficiency of wavelets with fewer complications. A poorly encoded block only makes a limited area of the image look bad. In wavelets errors in lower frequencies have ripple (wink) effects. It's mathematically easier to reason about DCT, so encoders can be really good. DCT can easily create noise and sharp edges (even if they're fake, they still can create realistic texture), but any sort of noise and sharpness is expensive with wavelets. In practice it turned out it's easier to do DCT and then blur unwanted edges than to express edges to wavelets.
therealmarv 11y ago
RIP. The wars today are WebP (Google) vs. JPEG-XR (Microsoft) vs. BPG ?! Maybe they will all also die... I only heard that WebP is/was used by Facebook and is used by Telegram.
- ashmud 11y ago
  BPG may have patent issues. http://bellard.org/bpg/ http://bellard.org/bpg/ "Some of the HEVC algorithms may be protected by patents in some countries (read the FFmpeg Patent Mini-FAQ for more information). Most devices already include or will include hardware HEVC support, so we suggest to use it if patents are an issue."
fnordfnordfnord 11y ago
I'd like to see this get more attention. http://bellard.org/bpg/ http://bellard.org/bpg/

30 points by nkurz 11y ago

I did my final University project on an algorithm based on JPEG2000. It's a pretty interesting way of compressing images. One of the best things about using the Wavelet Transform over the Discrete Cosine Transform is it allows you to structure the file or stream from low to high frequency. This lets you get an image of any size out of a single file just by taking a subset of it. No more resizing thumbnails. It's kind of like a super version of progressive JPEG. Sadly it never took off as storage / bandwidth are cheap and JPEG is so entrenched (especially in hardware). It still has some niche applications in constrained environments though. Edit: I dug out the paper if anyone wants to read it: https://unop.uk/misc/university-final-year-project-cbir https://unop.uk/misc/university-final-year-project-cbir It's funny looking back at something you wrote over 8 years ago any seeing how you've improved. In other words, please don't use this to judge me! Abstract This project investigated the effects of scaling images with JPEG2000 emulation software on a CBIR system based on MPEG-7. A CBIR system was built that consisted of indexing and retrieval. The performance of the system was evaluated with standard metrics. Both JPEG2000 and MPEG-7 never really took off. Most people moved on to more advanced algorithms.

varjag 11y ago
My understanding is patent encumbering was what killed JPEG2000. There was an effort later to waive the patents, but not sure how comprehensive and perhaps a little too late. So it's the same reason noone uses theoretically superior arithmetic coding option in baseline JPEGs.
- jsingleton 11y ago
  To some extent, yes. However, JPEG was also patented and that did pretty well. Someone made a lot of money on licensing. I think WebP stands a pretty good chance of supplanting JPEG on the web but it will still be used in lots of places. https://en.wikipedia.org/wiki/WebP https://en.wikipedia.org/wiki/WebP
  - ZeroGravitas 11y ago
    I don't think anyone really used the bit of JPEG that was flagged up initially as not being royalty-free (I think IBM owned that part?) and later on I think it was only patent trolls that tried (unsuccesfully) to licence JPEG. Which licencing on JPEG are you referring to?
    - jsingleton 11y ago
      I don't think it applied to the version used on the web. However, this is just from memory based on conversations I had with MPEG and JPEG board members a long time ago. I can't find any sources so I could be mistaken.
  - gillianseed 11y ago
    >However, JPEG was also patented and that did pretty well. I was under the impression that the free implementations (like libjpeg) did not use and of the patented techniques (such as arithmetic coding which patent recently expired) ? >I think WebP stands a pretty good chance of supplanting JPEG on the web but it will still be used in lots of places. I don't think anything will supplant JPEG as the de facto image format on the 'web', it's simply 'good enough' and of course free and supported everywhere. Also as for lossy compression I'm not sure WebP is a real improvement over JPEG, I'd love to see it used instead of PNG for lossless compression though, as here it is clearly superior.
  - varjag 11y ago
    > However, JPEG was also patented and that did pretty well. This is exactly the bit I was referring to (the arithmetic coding method). The rest of CCITT/ITU baseline spec was unencumbered, that's what everyone ended up using. Source: me, who implemented a baseline JPEG codec in late '90s.
  - acomjean 11y ago
    Did you mean GIF? Patents and compuserve/unisys attempt to make money off them basically pushed the development of PNG to replace it. https://en.wikipedia.org/wiki/GIF https://en.wikipedia.org/wiki/GIF
- IshKebab 11y ago
  It was more that it didn't offer enough benefits. It was "the files are a bit smaller". Image files don't take up that much space anyway so that is less compelling than for video compression. What they needed to do is add more features, like WebP has now: * Single format for photos and diagrams * Transparency * Lossy and lossless in the same format * HDR * Tiling of large pictures Hopefully those features (especially lossy transparency) will help WebP do better than JPEG2000, though I'm not holding my breath. If IE and Firefox ever add support for it then I think web developers will start using it a lot.
  - varjag 11y ago
    > It was more that it didn't offer enough benefits. It was "the files are a bit smaller". Well it was more than that: it offered unlimited color depth for example, a big progress from 8 bit per channel. It was technically superior format in nearly every respect.
  - JupiterMoon 11y ago
    Anecdotes are not data but... It's taken over 10 years before I can safely send people png images rather than tiffs in my line of work. I'm not switching again until the benefits are blindingly huge and obvious. Basically I think you are correct as to why JPEG2000 did not take off.
  - PaulKeeble 11y ago
    Performance was also a big problem, they had a lot of issues in the beginning justifying the amount of time it took to encode and especially decode the images. IIRC it was of the 100x the time of JPEG and that meant that at speeds of the time the bandwidth saved was overshadowed by the decode time. I did a lot of working with it and had quite a lot of problems with performance of the sample software.
    - yoklov 11y ago
      Did you feel that this was an intrinsic property of the format, or just that less/not enough time had been spent on developing efficient implementations?
acdha 11y ago
> This lets you get an image of any size out of a single file just by taking a subset of it. No more resizing thumbnails. I always think of this during responsive image discussions – if JPEG-2000 had caught on, we could have solved that entire problem simply by adding an attribute to <img> telling the client which HTTP byte ranges it needs to request for various sizes. Single-file, cache-friendly, degrades well, and … will never happen. > Sadly it never took off as storage / bandwidth are cheap and JPEG is so entrenched (especially in hardware) JPEG proving to be impressively durable certainly was a key factor but I also think a lot of it comes down to naive moves by the community. In addition to the patent situation, there were just basic things like having a huge spec which for a long time was neither freely available nor accompanied by a public test suite. I work with librarians who have large scanned digital collections and it was definitely interesting to compare the reactions: the users disliked JPEG 2000 because they'd been conditioned to expect it to be quite noticeably slower, unsupported by some of their tools, and knew that they couldn't rely on any two programs actually working with a given set of files until they tested it (http://jpylyzer.openpreservation.org/ http://jpylyzer.openpreservation.org/ has helped catch that earlier). Meanwhile, the JPEG-2000 standards people & other boosters just tended to assume inevitability, where everyone would drop other formats once they saw how inferior they were. Things like Photoshop removing JPEG-2000 support (first discussed publicly in 2007, dropped in Elements in 2011) had this odd effect where some people expressed considerable concern (http://wiki.opf-labs.org/display/TR/JPEG+2000+support+discontinued+in+Photoshop+Elements http://wiki.opf-labs.org/display/TR/JPEG+2000+support+discon...) but others just assumed it was some sort of unrepresentative quirk. I cannot help but think that the situation would be considerably different if something like OpenJPEG.org had gotten significant support much earlier. Jasper, JJ2000, etc. were the most popular way to add support at all and they're slow, buggy and generally stagnant so many people had a bad first experience which encouraged avoiding the format.
- saurik 11y ago
  > we could have solved that entire problem simply by adding an attribute to <img> telling the client which HTTP byte ranges it needs to request for various sizes You would never have a range start anywhere but the top of the file, as the whole point is that the later octaves are refinements of earlier ones. What makes JPEG2000 so amazing for this use case is precisely that you don't need to care about "ranges" (if you needed ranges, we might as well just have multiple files..). Instead, browsers just keep downloading the file until they either achieve the level of quality they need to render at the scale they currently care about (being able to continue later, if required) or run out of file. At worst, only as a "hint", I could see a benefit to having a table of sizes, but I feel like the cool solution for at would be to have an encoded version of the octave offsets, sort of like a zindex but for JPEG.

TorKlingberg 11y ago

It's interesting how JPEG2000 use has not really taken off outside of a few niche areas. Web browser vendors have been reluctant to implement it because of increases attack surface, and of course web sites will not use it without browser support. Patents may also be an issue. JPEG works mostly on 8x8 blocks of pixels, and in high resolution pictures each block is almost a single color. JPEG does a poor job taking advantage of similarity between the blocks. JPEG2000 does at least see some use in archival formats.

anon1385 11y ago
>Web browser vendors have been reluctant to implement it because of increases attack surface Which isn't a very convincing argument when they are quite happy to implement dozens of other things that vastly increase the attack surface of browsers but have far lower practical impact on the average web page. Imagine how much bandwidth has been wasted over the last 10 years because the web didn't have a lossy image format that supported transparency.
captainmuon 11y ago
> Web browser vendors have been reluctant to implement it because of increases attack surface, and of course web sites will not use it without browser support. I believe codecs should actually be part of the OS. There should be one good, well-tested implementation per OS that browsers and other apps just link to. It might actually be possible to sandbox the codec components for increased security. Ideally, a web browser would just be a HTML+stuff parser, a layout engine, and a bit of gui, and shell out to the OS for everything else. Unfortunately, it seems the monolithic kitchen sink model has won.
- vetinari 11y ago
  The monolithic kitchen sink has won, because the OS bundled libraries have different APIs (not only among different OSes, but also among different versions - and OS vendors are interested in pushing the latest), so app vendors just said to themselves "screw it" and bundled library (like jpeglib or libpng) that has same API on all systems and versioning is under their control. That's the short version ;).

pornel 11y ago

JPEG2000 just isn't that great, and DCT-based compression is very hard to beat. DCT with varying block sizes gets most of the efficiency of wavelets with fewer complications. A poorly encoded block only makes a limited area of the image look bad. In wavelets errors in lower frequencies have ripple (wink) effects. It's mathematically easier to reason about DCT, so encoders can be really good. DCT can easily create noise and sharp edges (even if they're fake, they still can create realistic texture), but any sort of noise and sharpness is expensive with wavelets. In practice it turned out it's easier to do DCT and then blur unwanted edges than to express edges to wavelets.

therealmarv 11y ago

RIP. The wars today are WebP (Google) vs. JPEG-XR (Microsoft) vs. BPG ?! Maybe they will all also die... I only heard that WebP is/was used by Facebook and is used by Telegram.

ashmud 11y ago
BPG may have patent issues. http://bellard.org/bpg/ http://bellard.org/bpg/ "Some of the HEVC algorithms may be protected by patents in some countries (read the FFmpeg Patent Mini-FAQ for more information). Most devices already include or will include hardware HEVC support, so we suggest to use it if patents are an issue."

fnordfnordfnord 11y ago

I'd like to see this get more attention. http://bellard.org/bpg/ http://bellard.org/bpg/