6 ms·

Using Saliency in progressive JPEG XL images

89 points by signa11 5y ago

ksec 5y ago
Can I use JPEG XL Yet [1] ? Again Image / Video / Audio Codec Support on Safari requires OS update. Unless this is on their agenda for iOS 16 and next macOS, which I doubt, it will likely be a few years before support of it becomes widely available. [1] https://caniuse.com/jpegxl https://caniuse.com/jpegxl
- Dylan16807 5y ago
  I'm surprised that there seem to be no polyfills.
  - chrismorgan 5y ago
    Polyfills for image and video formats are pretty much never going to be worth it: they’ll be a fairly hefty chunk of code (so you’ll need a lot of shrunk-by-using-the-polyfilled-format images to balance it out), and they’ll load much more slowly than the native (so that lot of images is now a liability), while not being able to smooth loading the way stuff native to the browser does. I don’t think a compelling reason to do it that way has ever or is ever likely to exist; it’s an all-round bad idea in practice. Just fall back to a different format, that’s what <picture> is designed for.
    - Dylan16807 5y ago
      I'm not sure how big a jxl decoder has to be, but there's a wasm webp decoder that zips to about 50KB. That kind of cost pays for itself very quickly. And whether it loads faster or slower depends on your connection. But yes it hurts to not get the smoothness benefits.
      - chrismorgan 5y ago
        Hmm… https://webmproject.github.io/libwebp-demo/webp_wasm/index.html https://webmproject.github.io/libwebp-demo/webp_wasm/index.h... ships 141KB/54KB (bytes/transferred) of WASM and 116KB/34KB of JavaScript, including lots of bogus SDL stuff. https://kenchris.github.io/wasm-webp/ https://kenchris.github.io/wasm-webp/ is 120KB/44KB of WASM and 30KB/9KB of JavaScript, much more reasonable. (Incidentally, I never like the JavaScript such tools produce; it’s always hideously unoptimised. Experience suggests to me that I could reduce that 30KB to under 10KB with a little effort, and possibly to under 5KB; and if I could touch the WASM as well, still further.) Definitely smaller than I thought I recalled, and WASM bytes are no dearer than image bytes, though JS bytes are much more expensive, so it’s closer to just the more expensive decoding of the actual images that’ll bite you. Plus the not-integrated loading experience, of course.
- zebracanevra 5y ago
  It doesn't matter too much (for the web) as you can use <picture> and <source> and specify multiple jxl/avif/webp/jpeg sources, and the browser will choose the first type it understands. These elements are widely supported. Hopefully JXL doesn't become yet another image format (927)... some features of it are incredible. For example, you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image.
  - est31 5y ago
    > you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image. What's the advantage of that? If I'm dissatisfied with the compression quality in some areas, I can tweak the quality parameter already.
    - zebracanevra 5y ago
      See https://github.com/libjxl/libjxl/pull/466 https://github.com/libjxl/libjxl/pull/466 for more details. As far as I understand it, the synthetic mode compresses better for (as you can guess) synthetic content. So if you're encoding a mixed content screenshot, you'll get better quality and size by encoding the photos and UI separately.
      - est31 5y ago
        Oh I see, that makes sense.
beingflo 5y ago
Reminds me of the mechanism twitter used to crop images based on gaze prediction [1]. If their model is anything to go by we know which parts of a bikini picture we will see in high-res first [2] [1] https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images https://blog.twitter.com/engineering/en_us/topics/infrastruc... [2] https://www.reportr.world/news/twitter-drops-image-cropping-algorithm-after-finding-bias-a4677-20210520 https://www.reportr.world/news/twitter-drops-image-cropping-...
miav 5y ago
It's very surprising to me that progressive loading techniques have been around for decades, but have barely been used on the web. Images often make for most of a websites content, size wise, and are frequently far larger than they need to be. Even those with generally good internet experience waiting for images to load for a noticeable amount of time. A 95% compressed version of an image is almost always good enough for a first glance and then the rest of the image can be seamlessly, progressively loaded. It's apparently been a ubiquitous problem for decades, with a known solution, yet it has not been applied. I'm curious as to why.
- p0nce 5y ago
  Decoding a progressive JPEG when you have the whole stream is more CPU intensive and the decoder itself is more complex.
  - joosters 5y ago
    Decoding JPEG images is not very CPU-intensive at all. Back in 1995, an ARM chip was fast enough to render JPEGs on the fly - i.e. apps didn't need to buffer the decoded versions, it was fast enough to draw them straight from the decoder when redrawing windows. (This was on RISC OS 3.6, IIRC)
    - p0nce 5y ago
      It is more CPU intensive than a baseline JPEG.
      - sk65535 5y ago
        rule of the thumb: progressive JPEG costs ~3x more resources (CPU, memory) than baseline JPEG. With baseline jpeg, you can blit the decoded blocks to screen directly and forget about it. For progressive, you have to buffer the whole image (in DCT format!) and do the idct / color-space conversion on all passes.
        5y ago
lnyan 5y ago
Related: https://jpegxl.info/ https://jpegxl.info/ https://news.ycombinator.com/item?id=27577328 https://news.ycombinator.com/item?id=27577328 Chrome status: behind a flag `enable-jxl` FireFox status: only in Nightly and behind a flag `image.jxl.enabled` Edge status: behind a runtime flag `--enable-features=JXL`
- Santosh83 5y ago
  Safari status?
  - Sesse__ 5y ago
    Safari is the new IE6; you can basically not expect anything to happen in a long time.
    - londons_explore 5y ago
      The webkit core of safari is opensource right? Do they accept outside contributions for stuff like this?
      - zebracanevra 5y ago
        See https://lists.webkit.org/pipermail/webkit-dev/2021-May/031846.html https://lists.webkit.org/pipermail/webkit-dev/2021-May/03184... and https://lists.webkit.org/pipermail/webkit-dev/2021-May/031851.html https://lists.webkit.org/pipermail/webkit-dev/2021-May/03185... > image format support depends on OS library support, rather than anything in WebKit. [...] to be clear, Apple’s OSes do not currently support JPEG XL.
- yboris 5y ago
  I even got `jxl` to work in Electron! https://github.com/whyboris/Video-Hub-App/compare/jxl https://github.com/whyboris/Video-Hub-App/compare/jxl
mchusma 5y ago
I was honestly skeptical of another format beyond AVIF for images. I thought the next generation coded would be based on neural net encoding/decoding for lossy images (as we should be able to get "easy" 2x wins). However, I think progressive loading in JPEG XL is an amazing feature, and a significant update, and is perhaps more important that raw compression ability. My understanding is also that you can also use the progressive nature to have 1 "high res" version, and potentially render different sizes for different connections, and do some interesting things. For example, I believe you could (in principal) do something like stop loading after 1s so long as a minimum size level was achieved) - to make things fast on all devices, but only the quality would vary. So if you are on a 4k monitor and gigabit download? You can get massive images. On a 3g phone? Get the basics. It's a really great feature, and does seem like it would likely have the most real world impact on actual website perceived performance.
- ByTheBook 5y ago
  Another strong feature is the great upgrade path from existing jpeg files, which can be losslessly recompressed into jpeg xl with an additional ~20% compression. At this point I don't really see a reason for AVIF, hardware accelerated decoding could be a benefit, but AFAIK there's no device supporting it.
- floatboth 5y ago
  AVIF should be completely cancelled in favor of JXL. A native designed-for-images codec is so much better than "use video codec for one frame" in countless ways. Indeed progressive being probably the biggest one. Oh, and Apple might be less hostile to it, because it's not based on an "anti-MPEG" video codec.
  - jonsneyers 5y ago
    I think AVIF is a great replacement for GIF (the acronyms even rhyme and all!). For short-video-without-audio, it's as good as it gets. For still images though, I don't really see any major advantage of AVIF over JXL except AVIF being a bit older and already enabled by default in Chrome. I do see several advantages of JXL over AVIF though. So the way I see it, the future is: AVIF instead of GIF (and APNG and animated WebP) JPEG XL instead of JPEG (and PNG and still WebP) Disclaimer: I am probably biased since I am a jxl dev. Please do your own independent evaluation (and feel free to share your opinions/conclusions with me).
hkmaxpro 5y ago
Nice discussion of progressive loading and other features in JPEG XL (albeit not relying on saliency) https://www.slideshare.net/cloudinarymarketing/imagecon-2019-jon-sneyer https://www.slideshare.net/cloudinarymarketing/imagecon-2019... Previous HN discussion https://news.ycombinator.com/item?id=21612708 https://news.ycombinator.com/item?id=21612708
cosmotic 5y ago
I'd be worried about a user experience where the image looked good enough that a user thought it was done downloading but then didn't realize it was getting better or still downloading and moved on. A nice indicator would be helpful.

89 points by signa11 5y ago

ksec 5y ago

Can I use JPEG XL Yet [1] ? Again Image / Video / Audio Codec Support on Safari requires OS update. Unless this is on their agenda for iOS 16 and next macOS, which I doubt, it will likely be a few years before support of it becomes widely available. [1] https://caniuse.com/jpegxl https://caniuse.com/jpegxl

Dylan16807 5y ago
I'm surprised that there seem to be no polyfills.
- chrismorgan 5y ago
  Polyfills for image and video formats are pretty much never going to be worth it: they’ll be a fairly hefty chunk of code (so you’ll need a lot of shrunk-by-using-the-polyfilled-format images to balance it out), and they’ll load much more slowly than the native (so that lot of images is now a liability), while not being able to smooth loading the way stuff native to the browser does. I don’t think a compelling reason to do it that way has ever or is ever likely to exist; it’s an all-round bad idea in practice. Just fall back to a different format, that’s what <picture> is designed for.
  - Dylan16807 5y ago
    I'm not sure how big a jxl decoder has to be, but there's a wasm webp decoder that zips to about 50KB. That kind of cost pays for itself very quickly. And whether it loads faster or slower depends on your connection. But yes it hurts to not get the smoothness benefits.
    - chrismorgan 5y ago
      Hmm… https://webmproject.github.io/libwebp-demo/webp_wasm/index.html https://webmproject.github.io/libwebp-demo/webp_wasm/index.h... ships 141KB/54KB (bytes/transferred) of WASM and 116KB/34KB of JavaScript, including lots of bogus SDL stuff. https://kenchris.github.io/wasm-webp/ https://kenchris.github.io/wasm-webp/ is 120KB/44KB of WASM and 30KB/9KB of JavaScript, much more reasonable. (Incidentally, I never like the JavaScript such tools produce; it’s always hideously unoptimised. Experience suggests to me that I could reduce that 30KB to under 10KB with a little effort, and possibly to under 5KB; and if I could touch the WASM as well, still further.) Definitely smaller than I thought I recalled, and WASM bytes are no dearer than image bytes, though JS bytes are much more expensive, so it’s closer to just the more expensive decoding of the actual images that’ll bite you. Plus the not-integrated loading experience, of course.
zebracanevra 5y ago
It doesn't matter too much (for the web) as you can use <picture> and <source> and specify multiple jxl/avif/webp/jpeg sources, and the browser will choose the first type it understands. These elements are widely supported. Hopefully JXL doesn't become yet another image format (927)... some features of it are incredible. For example, you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image.
- est31 5y ago
  > you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image. What's the advantage of that? If I'm dissatisfied with the compression quality in some areas, I can tweak the quality parameter already.
  - zebracanevra 5y ago
    See https://github.com/libjxl/libjxl/pull/466 https://github.com/libjxl/libjxl/pull/466 for more details. As far as I understand it, the synthetic mode compresses better for (as you can guess) synthetic content. So if you're encoding a mixed content screenshot, you'll get better quality and size by encoding the photos and UI separately.
    - est31 5y ago
      Oh I see, that makes sense.

beingflo 5y ago

Reminds me of the mechanism twitter used to crop images based on gaze prediction [1]. If their model is anything to go by we know which parts of a bikini picture we will see in high-res first [2] [1] https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images https://blog.twitter.com/engineering/en_us/topics/infrastruc... [2] https://www.reportr.world/news/twitter-drops-image-cropping-algorithm-after-finding-bias-a4677-20210520 https://www.reportr.world/news/twitter-drops-image-cropping-...

miav 5y ago

It's very surprising to me that progressive loading techniques have been around for decades, but have barely been used on the web. Images often make for most of a websites content, size wise, and are frequently far larger than they need to be. Even those with generally good internet experience waiting for images to load for a noticeable amount of time. A 95% compressed version of an image is almost always good enough for a first glance and then the rest of the image can be seamlessly, progressively loaded. It's apparently been a ubiquitous problem for decades, with a known solution, yet it has not been applied. I'm curious as to why.

p0nce 5y ago
Decoding a progressive JPEG when you have the whole stream is more CPU intensive and the decoder itself is more complex.
- joosters 5y ago
  Decoding JPEG images is not very CPU-intensive at all. Back in 1995, an ARM chip was fast enough to render JPEGs on the fly - i.e. apps didn't need to buffer the decoded versions, it was fast enough to draw them straight from the decoder when redrawing windows. (This was on RISC OS 3.6, IIRC)
  - p0nce 5y ago
    It is more CPU intensive than a baseline JPEG.
    - sk65535 5y ago
      rule of the thumb: progressive JPEG costs ~3x more resources (CPU, memory) than baseline JPEG. With baseline jpeg, you can blit the decoded blocks to screen directly and forget about it. For progressive, you have to buffer the whole image (in DCT format!) and do the idct / color-space conversion on all passes.
      - 5y ago

lnyan 5y ago

Related: https://jpegxl.info/ https://jpegxl.info/ https://news.ycombinator.com/item?id=27577328 https://news.ycombinator.com/item?id=27577328 Chrome status: behind a flag `enable-jxl` FireFox status: only in Nightly and behind a flag `image.jxl.enabled` Edge status: behind a runtime flag `--enable-features=JXL`

Santosh83 5y ago
Safari status?
- Sesse__ 5y ago
  Safari is the new IE6; you can basically not expect anything to happen in a long time.
  - londons_explore 5y ago
    The webkit core of safari is opensource right? Do they accept outside contributions for stuff like this?
    - zebracanevra 5y ago
      See https://lists.webkit.org/pipermail/webkit-dev/2021-May/031846.html https://lists.webkit.org/pipermail/webkit-dev/2021-May/03184... and https://lists.webkit.org/pipermail/webkit-dev/2021-May/031851.html https://lists.webkit.org/pipermail/webkit-dev/2021-May/03185... > image format support depends on OS library support, rather than anything in WebKit. [...] to be clear, Apple’s OSes do not currently support JPEG XL.
yboris 5y ago
I even got `jxl` to work in Electron! https://github.com/whyboris/Video-Hub-App/compare/jxl https://github.com/whyboris/Video-Hub-App/compare/jxl

mchusma 5y ago

I was honestly skeptical of another format beyond AVIF for images. I thought the next generation coded would be based on neural net encoding/decoding for lossy images (as we should be able to get "easy" 2x wins). However, I think progressive loading in JPEG XL is an amazing feature, and a significant update, and is perhaps more important that raw compression ability. My understanding is also that you can also use the progressive nature to have 1 "high res" version, and potentially render different sizes for different connections, and do some interesting things. For example, I believe you could (in principal) do something like stop loading after 1s so long as a minimum size level was achieved) - to make things fast on all devices, but only the quality would vary. So if you are on a 4k monitor and gigabit download? You can get massive images. On a 3g phone? Get the basics. It's a really great feature, and does seem like it would likely have the most real world impact on actual website perceived performance.

ByTheBook 5y ago
Another strong feature is the great upgrade path from existing jpeg files, which can be losslessly recompressed into jpeg xl with an additional ~20% compression. At this point I don't really see a reason for AVIF, hardware accelerated decoding could be a benefit, but AFAIK there's no device supporting it.
floatboth 5y ago
AVIF should be completely cancelled in favor of JXL. A native designed-for-images codec is so much better than "use video codec for one frame" in countless ways. Indeed progressive being probably the biggest one. Oh, and Apple might be less hostile to it, because it's not based on an "anti-MPEG" video codec.
- jonsneyers 5y ago
  I think AVIF is a great replacement for GIF (the acronyms even rhyme and all!). For short-video-without-audio, it's as good as it gets. For still images though, I don't really see any major advantage of AVIF over JXL except AVIF being a bit older and already enabled by default in Chrome. I do see several advantages of JXL over AVIF though. So the way I see it, the future is: AVIF instead of GIF (and APNG and animated WebP) JPEG XL instead of JPEG (and PNG and still WebP) Disclaimer: I am probably biased since I am a jxl dev. Please do your own independent evaluation (and feel free to share your opinions/conclusions with me).

hkmaxpro 5y ago

Nice discussion of progressive loading and other features in JPEG XL (albeit not relying on saliency) https://www.slideshare.net/cloudinarymarketing/imagecon-2019-jon-sneyer https://www.slideshare.net/cloudinarymarketing/imagecon-2019... Previous HN discussion https://news.ycombinator.com/item?id=21612708 https://news.ycombinator.com/item?id=21612708

cosmotic 5y ago

I'd be worried about a user experience where the image looked good enough that a user thought it was done downloading but then didn't realize it was getting better or still downloading and moved on. A nice indicator would be helpful.