3 ms·

My comment was apparently too concise to give you a sense of complications. I can think of those concrete problems: - Rendering images from a different origin

My comment was apparently too concise to give you a sense of complications. I can think of those concrete problems: - Rendering images from a different origin without CORS. This is a fundamental limitation of any JS or WebAssembly solution and can't be fixed. Thankfully this use case is relatively rare. - Not all approaches can provide a seamless upgrade. For example if you replace all `<img src="foo.jxl">` with a canvas DOM will be changed and anything expecting the element to be HTMLImageElement will break. Likewise, CSS Painting API [1] (a relevant part of CSS Houdini) requires you to explicitly write `paint(foo)` everywhere. The only seamless solution will be therefore a service worker, but it can't introduce any new image format; it can only convert to natively supported formats. And browsers currently don't have a "raw" image format for this purpose. JXL.js [2] for example had to use JPEG as a delivery format because other formats were too slow, as I've been told. - It is very hard to check if a certain image is visible or not, and react accordingly. This is what I intended to imply by saying that canvas has to unconditionally retain all pixels, because if implementations can't decide if it's safe to unload images, they can't do so and memory will contain invisible images in the form of canvases. Browsers do have a ground truth and so can safely unload currently invisible images from memory when the memory pressure is high. [1] https://developer.mozilla.org/en-US/docs/Web/API/CSS_Painting_API https://developer.mozilla.org/en-US/docs/Web/API/CSS_Paintin... [2] https://github.com/niutech/jxl.js https://github.com/niutech/jxl.js

Sephr 4y ago
> Not all approaches can provide a seamless upgrade. For example if you replace all `<img src="foo.jxl">` with a canvas DOM will be changed and anything expecting the element to be HTMLImageElement will break. Likewise, CSS Painting API [1] (a relevant part of CSS Houdini) requires you to explicitly write `paint(foo)` everywhere. The only seamless solution will be therefore a service worker, but it can't introduce any new image format; it can only convert to natively supported formats. And browsers currently don't have a "raw" image format for this purpose. JXL.js [2] for example had to use JPEG as a delivery format because other formats were too slow, as I've been told. You can get around many these compatibility issues by creating a custom element that inherits from HTMLImageElement. This provides API compatibility. For CSS compatibility, the elements you would replace in a MutationObserver would be the same tag name but a different namespace for CSS compatibility. For the CSS compatibility trick, see https://eligrey.com/demos/hotlink.js/ https://eligrey.com/demos/hotlink.js/ which replaces images with CSS-compatible (not HTMLImageElement-compatible) iframes. > - It is very hard to check if a certain image is visible or not, and react accordingly. You can use Element.checkVisibility()¹ and the contentvisibilityautostatechanged event²𝄒³ to do this. Browser support is currently limited to Chromium-based browsers. 1. https://drafts.csswg.org/cssom-view/#dom-element-checkvisibility https://drafts.csswg.org/cssom-view/#dom-element-checkvisibi... 2. https://github.com/vmpstr/web-proposals/blob/main/explainers/cv-auto-event.md#example https://github.com/vmpstr/web-proposals/blob/main/explainers... 3. https://caniuse.com/mdn-api_element_contentvisibilityautostatechanged_event https://caniuse.com/mdn-api_element_contentvisibilityautosta...
- lifthrasiir 4y ago
  Thank you for pointing out contentvisibilityautostatechanged, I was aware of `content-visibility` but didn't know that it has an associated event. I'm less sure about CSS compatibility, hotlink.js for example used an iframe which opens a whole can of worms.
niutech 4y ago
I've just added a config parameter to JXL.js for choosing the target image type: JPG/PNG/WebP. Keep in mind that PNGs can take a lot memory!
supermatt 4y ago
You dont need to replace the img with canvas DOM - you can capture the canvas output as a dataURL.
- lifthrasiir 4y ago
  You can, but that is hugely inefficient. Any additional draw to the canvas has to generate a data URL for the image to be progressively decoded.
- rezonant 4y ago
  Can't reply to the reply to your reply but... Blobs can be used here and avoid the inefficiency of data URL conversions
  - niutech 4y ago
    This is exactly what I use in my JXL.js, as well as Web Workers and OfflineCanvas.
  - supermatt 4y ago
    Thanks for the tip! (Not for jpegXL specifically, but ill definitely check that out and update some code where i use dataURLs instead of objectURLs accordingly)
charcircuit 4y ago
>It is very hard to check if a certain image is visible or not Canvases are rectangles. The viewport is a rectangle. Checking if rectangles overlap is easy.
- lifthrasiir 4y ago
  The image is visible != the canvas and viewport overlaps, and this is not even a good enough approximation (the image can be obscured by other layers for example). Intersection Observer v2 takes us a bit closer but the visibility in this definition (not obscured at all) doesn't fully agree with what we want (has some pixels visible, some false positives allowed).

Sephr 4y ago

> Not all approaches can provide a seamless upgrade. For example if you replace all `<img src="foo.jxl">` with a canvas DOM will be changed and anything expecting the element to be HTMLImageElement will break. Likewise, CSS Painting API [1] (a relevant part of CSS Houdini) requires you to explicitly write `paint(foo)` everywhere. The only seamless solution will be therefore a service worker, but it can't introduce any new image format; it can only convert to natively supported formats. And browsers currently don't have a "raw" image format for this purpose. JXL.js [2] for example had to use JPEG as a delivery format because other formats were too slow, as I've been told. You can get around many these compatibility issues by creating a custom element that inherits from HTMLImageElement. This provides API compatibility. For CSS compatibility, the elements you would replace in a MutationObserver would be the same tag name but a different namespace for CSS compatibility. For the CSS compatibility trick, see https://eligrey.com/demos/hotlink.js/ https://eligrey.com/demos/hotlink.js/ which replaces images with CSS-compatible (not HTMLImageElement-compatible) iframes. > - It is very hard to check if a certain image is visible or not, and react accordingly. You can use Element.checkVisibility()¹ and the contentvisibilityautostatechanged event²𝄒³ to do this. Browser support is currently limited to Chromium-based browsers. 1. https://drafts.csswg.org/cssom-view/#dom-element-checkvisibility https://drafts.csswg.org/cssom-view/#dom-element-checkvisibi... 2. https://github.com/vmpstr/web-proposals/blob/main/explainers/cv-auto-event.md#example https://github.com/vmpstr/web-proposals/blob/main/explainers... 3. https://caniuse.com/mdn-api_element_contentvisibilityautostatechanged_event https://caniuse.com/mdn-api_element_contentvisibilityautosta...

lifthrasiir 4y ago
Thank you for pointing out contentvisibilityautostatechanged, I was aware of `content-visibility` but didn't know that it has an associated event. I'm less sure about CSS compatibility, hotlink.js for example used an iframe which opens a whole can of worms.

niutech 4y ago

I've just added a config parameter to JXL.js for choosing the target image type: JPG/PNG/WebP. Keep in mind that PNGs can take a lot memory!

supermatt 4y ago

You dont need to replace the img with canvas DOM - you can capture the canvas output as a dataURL.

lifthrasiir 4y ago
You can, but that is hugely inefficient. Any additional draw to the canvas has to generate a data URL for the image to be progressively decoded.
rezonant 4y ago
Can't reply to the reply to your reply but... Blobs can be used here and avoid the inefficiency of data URL conversions
- niutech 4y ago
  This is exactly what I use in my JXL.js, as well as Web Workers and OfflineCanvas.
- supermatt 4y ago
  Thanks for the tip! (Not for jpegXL specifically, but ill definitely check that out and update some code where i use dataURLs instead of objectURLs accordingly)

charcircuit 4y ago

>It is very hard to check if a certain image is visible or not Canvases are rectangles. The viewport is a rectangle. Checking if rectangles overlap is easy.

lifthrasiir 4y ago
The image is visible != the canvas and viewport overlaps, and this is not even a good enough approximation (the image can be obscured by other layers for example). Intersection Observer v2 takes us a bit closer but the visibility in this definition (not obscured at all) doesn't fully agree with what we want (has some pixels visible, some false positives allowed).