3 ms·

On iOS, any screenshot taken while iOS background blur is active will balloon from 0.15MB to 15.0MB, because iOS uses PNG for screenshots and blurred background

On iOS, any screenshot taken while iOS background blur is active will balloon from 0.15MB to 15.0MB, because iOS uses PNG for screenshots and blurred backgrounds are apparently irreducible by PNG. Does the WebP format permit bounded areas of an image to be represented at lower fidelity with a smooth blur, so that the blurred-background effect can be stored and retrieved using fewer pixels and a blur algorithm rather than more pixels and no blur algorithm? Do PNG or HEIF (h265) support this? Does any image format support bounded areas of lower fidelity? The tricky part is that this implies that some areas of the image should intentionally be reproduced as ‘lossy’ and ‘lower fidelity’ and so forth, which is precisely correct - but goes against the grain of image encoding in the past.

floatingatoll 6y ago
I did some reading of the PNG specification and it may be possible to take advantage of Adam7 interlacing (8 passes total) and scanline filtering (several methods) to write out an image where the scanlines of 'known to be blurry' areas contain only 1 pass of image data and 7 passes of highly-compressible scanlines with filters that generate blur at decode-time. Doing this formally at scale would require the compositor and the encoder to cooperate, as the encoder would benefit greatly from having access to both the 'blurred' and 'unblurred' areas without the blur filter having been applied to the former, as it could then construct low-fidelity, low-bandwidth, visually pleasing blur for the 'blurred' segments. This exceeds my ability to write PNGs by hand and it certainly exceeds the bounds of what most people think 'an encoder' should be capable of doing, but at least it presents a path forward. I'll post to HN someday if I ever somehow manage to do this.
felixfbecker 6y ago
The only image format I can think of where you can encode a blur applied to some part of the image is SVG
kccqzy 6y ago
At that point you might as well use something like PDF, where you can choose exactly what image format you need for each region. Put a JPG as the blurred background, then put a transparent PNG as the foreground for text and buttons in your screenshot.
- floatingatoll 6y ago
  macOS screenshots appeared to be PDF natively for the first few years, which made sense since their window compositor was operating what appeared to be a PDF canvas in-memory. They're still clearly using a PDF-like canvas, since there are certain windows that aren't included in screenshots — but what's underneath them is — which is not possible without a layered canvas somewhere. I think after the first decade they stopped offering .PDF screenshots and switched to .PNG now, and based on other replies alongside yours, it's likely they'll be replaced by HEIF soon, but it bodes well for improvements here that their screenshot engine has access to the original layers. (This isn't just relevant to Apple users — imagine if Photoshop's PNG/HEIF encoder could export blurred segments with lower byte density than unblurred segments, for example. Folks generally are not used to thinking about fidelity-sensitive image encoding, and that's why this is so interesting to me.)
m0dest 6y ago
HEIC (HEVC-based image) and AVIF (AV1-based image) both support a lossless mode that does efficient compression of blurs and gradients, without losing any visual fidelity. They're good candidates for screenshots in the future. Lossless HEIC is often 50% of the size of the equivalent PNG.
jonsneyers 6y ago
HEIF, AVIF and JPEG XL all support multi-layer images, so you could e.g. encode the background in a lossy layer (which works well on blurry stuff) and foreground in a lossless layer (which works well on screenshots).

floatingatoll 6y ago

I did some reading of the PNG specification and it may be possible to take advantage of Adam7 interlacing (8 passes total) and scanline filtering (several methods) to write out an image where the scanlines of 'known to be blurry' areas contain only 1 pass of image data and 7 passes of highly-compressible scanlines with filters that generate blur at decode-time. Doing this formally at scale would require the compositor and the encoder to cooperate, as the encoder would benefit greatly from having access to both the 'blurred' and 'unblurred' areas without the blur filter having been applied to the former, as it could then construct low-fidelity, low-bandwidth, visually pleasing blur for the 'blurred' segments. This exceeds my ability to write PNGs by hand and it certainly exceeds the bounds of what most people think 'an encoder' should be capable of doing, but at least it presents a path forward. I'll post to HN someday if I ever somehow manage to do this.

felixfbecker 6y ago

The only image format I can think of where you can encode a blur applied to some part of the image is SVG

kccqzy 6y ago

At that point you might as well use something like PDF, where you can choose exactly what image format you need for each region. Put a JPG as the blurred background, then put a transparent PNG as the foreground for text and buttons in your screenshot.

floatingatoll 6y ago
macOS screenshots appeared to be PDF natively for the first few years, which made sense since their window compositor was operating what appeared to be a PDF canvas in-memory. They're still clearly using a PDF-like canvas, since there are certain windows that aren't included in screenshots — but what's underneath them is — which is not possible without a layered canvas somewhere. I think after the first decade they stopped offering .PDF screenshots and switched to .PNG now, and based on other replies alongside yours, it's likely they'll be replaced by HEIF soon, but it bodes well for improvements here that their screenshot engine has access to the original layers. (This isn't just relevant to Apple users — imagine if Photoshop's PNG/HEIF encoder could export blurred segments with lower byte density than unblurred segments, for example. Folks generally are not used to thinking about fidelity-sensitive image encoding, and that's why this is so interesting to me.)

m0dest 6y ago

HEIC (HEVC-based image) and AVIF (AV1-based image) both support a lossless mode that does efficient compression of blurs and gradients, without losing any visual fidelity. They're good candidates for screenshots in the future. Lossless HEIC is often 50% of the size of the equivalent PNG.

jonsneyers 6y ago

HEIF, AVIF and JPEG XL all support multi-layer images, so you could e.g. encode the background in a lossy layer (which works well on blurry stuff) and foreground in a lossless layer (which works well on screenshots).