6 ms·

This seems conceptually most similar to Fabrice Bellard's proposal 'BPG' [1], which was essentially a lightweight wrapper around an HEVC I-frame. The format got

This seems conceptually most similar to Fabrice Bellard's proposal 'BPG' [1], which was essentially a lightweight wrapper around an HEVC I-frame. The format got small amounts of attention in encoding circles, but didn't pick up mainstream traction. The image encoding layer of HEIF is the same HEVC, but the container is MPEG-4 Part 12 (the Quicktime-descendant ISO Base Media Format; the core behind .mp4, .3gp, etc.), which theoretically gives it wide support. In this structure, HEIF supports image sequences, which will help it compete with animated GIF, GIFV (which is really just a short MPEG-4 Part 14 file containing usually an H.264 video track), animated WebP (who uses these?), and WebM. This format doesn't really blaze new ground (EDIT: it does in the sense that it defines a container format to express image-y constructs like still images and sequences of images in an ISO media container, but see my other comment that asks how this is similar but different to video [3]), but if this repackaging and the resulting code donation and political clout helps it gain traction, we still would gain a lot. The problem, of course, is always with backwards-compatibility. WebP was aggressively promoted by Google the same way Microsoft used to promote its quirky Windows Media formats back in the early 2000s, but the WebP and non-WebP camps are still largely separate. This is unfortunate, because WebP in my opinion really isn't very good, and a backwards-compatible compressor on top of JPEG, such as Dropbox's Lepton [2] achieves similar results. Any HEVC-based image compressor is necessarily incompatible with JPEG; so broad buy-in would be required from makers of software and hardware products, from operating systems, file managers, image viewers, image editors, cameras, and the like, to really enjoy its improved compression rates, vs. being just another incompatible format that only functions inside controlled ecosystems. The video codec AV1 currently in development was supposed to be a grand alliance of disparate companies to agree on a common format for the future, and rectify the WebM vs. non-WebM split, but continuing to promulgate sophisticated formats based on work outside of this scope (such as MPEG- or ITU-derived custom formats) just muddies this further. [1] https://bellard.org/bpg/ https://bellard.org/bpg/ [2] https://news.ycombinator.com/item?id=13230805 https://news.ycombinator.com/item?id=13230805 [3] https://news.ycombinator.com/item?id=14490734 https://news.ycombinator.com/item?id=14490734

zerocrates 9y ago
It mentions that it supports not only sequences but inter-frame prediction (so P- and possibly B- frames I assume), which is sort of surprising to me but of course makes sense especially given Apple's "image burst" usage. I'm also suprised to learn that WebP apparently supports P-frames in its animations. What's really different between this and the existing usages of the MP4 container for images, though? Is it just the packaging as an "image" format, or specification of more image-y metadata?
- jacobolus 9y ago
  https://nokiatech.github.io/heif/technical.html https://nokiatech.github.io/heif/technical.html
- jpap 9y ago
  HEIF is actually very closely related to MP4 (or really, the ISO Base Media File Format (BMFF); and historically, QuickTime which is what they are both based on). You are right -- the main difference is that BMFF was originally designed for time-based media (video, audio) but isn't in itself well suited towards media that isn't time based, like images and their associated metadata (e.g. Exif). HEIF extends the ISO BMFF so that you can have untimed media -- one or more photos, a collection -- or even mixed media comprising both untimed and timed media. Apple's live photo is a good example of mixed media, comprising a photo and a video. But the HEIF container format goes so much further. You could have image sequences -- for example a higher-quality version of Animated GIF (using I-frames only, or P/B for more compression) with looping -- tiling, auxiliary images like depth maps and alpha channels, or even stereo images with a left and right channel. Though originally HEIF came out of the HEVC standards track, hence the words "High Efficiency" in the name, it was later extended to include other codecs like JPEG, AVC. There's no reason it couldn't be extended to include VP9, PNG, or any other codec. Think of HEIF as a versatile, extensible, standardized container format. The media coding scheme is separable. This is a big deal for the future of image/media coding because we're no-longer locked into "yet another format" that's tied to the codec. With the ISO BMFF box model, HEIF can grow to adapt new constructs and codecs for some generations to come.
  - 9y ago
wereHamster 9y ago
At last for the web (which is what WebP targets mostly, right?) you don't need broad support. We have the new <picture> element where the UA selects the most appropriate image from a list of options. So Chrome will prefer webp, other browsers will pick jpeg, png or whatever else they support at the moment.
- deleted 9y ago
  [deleted]
- jacobolus 9y ago
  If possible, I’d much prefer having a single format that is widely supported, hardware accelerated, and with a state-of-the-art codec. Making content authors create multiple different files in different formats—some with deficient feature support—sounds like a pain in the butt for everyone. If we can get everyone to agree on a format that handles layers, exposure bracketing, transparency, animation, high bit depth, different color models, both lossy and lossless compression, every common type of metadata, etc., that would be great.
  - nkkollaw 9y ago
    There are services like Imgix that will do the conversion automatically on the fly, you don't have to convert images manually.
  - deleted 9y ago
    [deleted]
  - astrange 9y ago
    If your format handles everything it isn't a format, it just is everything. (The one you want is TIFF though.)
    - jacobolus 9y ago
      The point is that you want support for those features (certainly the core codecs, etc.) in the spec, so if you send someone a file you can be sure they can read it. TIFF is not useful if you want to use a state-of-the-art lossy codec and get hardware-accelerated decoding on an arbitrary mobile device, or if you want to display an animated image with transparency on a web page.
zokier 9y ago
I think the significance of compatibility is increasingly diminishing due the effect of "clouds" and other walled gardens that handle your data in a more integrated manner. With the advent of wasm etc it is not completely unreasonable to ship your own image decoder with the application, and optionally a JPEG encoder for on-demand client side re-encoding for exporting images. There is just less expectations to be able to interoperate via "dumb files" in the "app ecosystem" where the apps run in their own isolated sandboxes and import/export are explicit actions. Not saying that this is the best thing, but the situation is different enough from the time when WebP/WMP were introduced that HEIF actually might have a fighting chance. The fact that HEIF appears to be far bigger improvement over JPEG than previous efforts of course also helps.
- deleted 9y ago
  [deleted]
- TheCoreh 9y ago
  Shipping a JPEG encoder is not necessary: all browsers support encoding images via toDataURL('image/jpeg') and toBlob(...)
  - nomel 9y ago
    I think he meant non-standard, and if not, his point still stands: with webasm, you might not have to worry about silly client side things like codecs.
Veratyr 9y ago
> The video codec AV1 currently in development was supposed to be a grand alliance of disparate companies to agree on a common format for the future, and rectify the WebM vs. non-WebM split, but continuing to promulgate sophisticated formats based on work outside of this scope (such as MPEG- or ITU-derived custom formats) just muddies this further. Apple is just blatantly not interested in cooperating with the rest of the market. Mozilla, Google and Microsoft are all part of AoM, along with a bunch of hardware companies and distributors yet Apple is absent and apparently pursuing HEVC instead. This isn't new either, Apple pushed HLS when others were pushing DASH and MPEG-TS when others were pushing fragmented MP4. They seem really determined to go their own way and ignore everyone else for some reason.
- threeseed 9y ago
  You're talking nonsense. H.265 is an ITU-T standard and was adopted as the standard for broadcast television ATSC. This isn't an Apple developed codec. And many more companies support it over AoM: http://www.mpegla.com/main/programs/HEVC/Pages/Licensors.aspx http://www.mpegla.com/main/programs/HEVC/Pages/Licensors.asp... http://www.mpegla.com/main/programs/HEVC/Pages/Licensees.aspx http://www.mpegla.com/main/programs/HEVC/Pages/Licensees.asp... http://www.hevcadvance.com/pdfnew/LicensorList.pdf http://www.hevcadvance.com/pdfnew/LicensorList.pdf http://www.hevcadvance.com/pdfnew/LicenseeList.pdf http://www.hevcadvance.com/pdfnew/LicenseeList.pdf Including the likes of: BBC, Samsung, Dolby, GE, MediaTek, Philips, Mitsubishi, Warner Bros, Sony etc. And as for hardware vendors well AMD, Intel, Nvidia etc all support H.265.
  - Veratyr 9y ago
    > This isn't an Apple developed codec. I didn't say it was, I said Apple was pursuing it. > And many more companies support it over AoM You can't claim that yet as AV1 isn't yet finalized.
    - abritinthebay 9y ago
      > You can't claim that yet as AV1 isn't yet finalized. Do more companies support it? Yes. Failure to ship is also a competitive disadvantage - AV1 may be great (we don't know) but if this one gets traction first that may not matter.
jpap 9y ago
> This format doesn't really blaze new ground (EDIT: it does in the sense that it defines a container format to express image-y constructs like still images and sequences of images in an ISO media container, but see my other comment that asks how this is similar but different to video [3]), but if this repackaging and the resulting code donation and political clout helps it gain traction, we still would gain a lot. You need to think about HEIF as the container and disconnected from the codec used to compress the media content. In that sense, HEIF is a massive trailblazer. It gives us an ISO standard to describe image and mixed media consistent with existing methods (i.e. MP4) while allowing for new constructs (i.e. useful features of the future in addition to tiling, depth map, stereo, etc.) and new codecs as they are developed and adopted. HEIF is a universal forward-looking format. Until now, almost all of the major imaging formats were tied to the codec and not terribly extensible to new use cases. While BPG is a great format in the short term (it gave us HEVC coded images around 2.5 years ago), it isn't an ideal choice for the long term when viewed as above.
- niutech 9y ago
  TIFF and Exif (which JPEG is based on) are not tied to any particular codec and are extensible, so HEIF is not a game changer. BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif?
  - jpap 9y ago
    > TIFF and Exif (which JPEG is based on) are not tied to any particular codec and are extensible, so HEIF is not a game changer. TIFF is a great format but its ability for extensibility is limited. You can't readily contain in a video track in addition to a photo, or a burst sequence that utilizes intra-frame prediction. > BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif? Yes; I once owned point-and-shoot cameras that did this. The audio was pretty poor quality because they didn't employ compression and wanted to keep the file sizes small. However you're limited to the 64k maximum JPEG marker segment size for non-image metadata; or ugly hacks like chaining segments (e.g. ICC profiles). Exif is strictly limited to being contained within 64k. How big is your audio track again? ;-) JPEG has had its time as a wildly successful format but has also held back the imaging world from adopting a standardized way to cater for new applications; burst above, mixed media (photo + video + audio), depth maps, stereo images, alpha (that's not a hack), lossless, etc., etc. HEIF has all the key ingredients, including extensibility, to support these modern applications and grow as the universal container format for decades to come.

zerocrates 9y ago

It mentions that it supports not only sequences but inter-frame prediction (so P- and possibly B- frames I assume), which is sort of surprising to me but of course makes sense especially given Apple's "image burst" usage. I'm also suprised to learn that WebP apparently supports P-frames in its animations. What's really different between this and the existing usages of the MP4 container for images, though? Is it just the packaging as an "image" format, or specification of more image-y metadata?

jacobolus 9y ago
https://nokiatech.github.io/heif/technical.html https://nokiatech.github.io/heif/technical.html
jpap 9y ago
HEIF is actually very closely related to MP4 (or really, the ISO Base Media File Format (BMFF); and historically, QuickTime which is what they are both based on). You are right -- the main difference is that BMFF was originally designed for time-based media (video, audio) but isn't in itself well suited towards media that isn't time based, like images and their associated metadata (e.g. Exif). HEIF extends the ISO BMFF so that you can have untimed media -- one or more photos, a collection -- or even mixed media comprising both untimed and timed media. Apple's live photo is a good example of mixed media, comprising a photo and a video. But the HEIF container format goes so much further. You could have image sequences -- for example a higher-quality version of Animated GIF (using I-frames only, or P/B for more compression) with looping -- tiling, auxiliary images like depth maps and alpha channels, or even stereo images with a left and right channel. Though originally HEIF came out of the HEVC standards track, hence the words "High Efficiency" in the name, it was later extended to include other codecs like JPEG, AVC. There's no reason it couldn't be extended to include VP9, PNG, or any other codec. Think of HEIF as a versatile, extensible, standardized container format. The media coding scheme is separable. This is a big deal for the future of image/media coding because we're no-longer locked into "yet another format" that's tied to the codec. With the ISO BMFF box model, HEIF can grow to adapt new constructs and codecs for some generations to come.
- 9y ago

wereHamster 9y ago

At last for the web (which is what WebP targets mostly, right?) you don't need broad support. We have the new <picture> element where the UA selects the most appropriate image from a list of options. So Chrome will prefer webp, other browsers will pick jpeg, png or whatever else they support at the moment.

deleted 9y ago
[deleted]
jacobolus 9y ago
If possible, I’d much prefer having a single format that is widely supported, hardware accelerated, and with a state-of-the-art codec. Making content authors create multiple different files in different formats—some with deficient feature support—sounds like a pain in the butt for everyone. If we can get everyone to agree on a format that handles layers, exposure bracketing, transparency, animation, high bit depth, different color models, both lossy and lossless compression, every common type of metadata, etc., that would be great.
- nkkollaw 9y ago
  There are services like Imgix that will do the conversion automatically on the fly, you don't have to convert images manually.
- deleted 9y ago
  [deleted]
- astrange 9y ago
  If your format handles everything it isn't a format, it just is everything. (The one you want is TIFF though.)
  - jacobolus 9y ago
    The point is that you want support for those features (certainly the core codecs, etc.) in the spec, so if you send someone a file you can be sure they can read it. TIFF is not useful if you want to use a state-of-the-art lossy codec and get hardware-accelerated decoding on an arbitrary mobile device, or if you want to display an animated image with transparency on a web page.

zokier 9y ago

I think the significance of compatibility is increasingly diminishing due the effect of "clouds" and other walled gardens that handle your data in a more integrated manner. With the advent of wasm etc it is not completely unreasonable to ship your own image decoder with the application, and optionally a JPEG encoder for on-demand client side re-encoding for exporting images. There is just less expectations to be able to interoperate via "dumb files" in the "app ecosystem" where the apps run in their own isolated sandboxes and import/export are explicit actions. Not saying that this is the best thing, but the situation is different enough from the time when WebP/WMP were introduced that HEIF actually might have a fighting chance. The fact that HEIF appears to be far bigger improvement over JPEG than previous efforts of course also helps.

deleted 9y ago
[deleted]
TheCoreh 9y ago
Shipping a JPEG encoder is not necessary: all browsers support encoding images via toDataURL('image/jpeg') and toBlob(...)
- nomel 9y ago
  I think he meant non-standard, and if not, his point still stands: with webasm, you might not have to worry about silly client side things like codecs.

Veratyr 9y ago

> The video codec AV1 currently in development was supposed to be a grand alliance of disparate companies to agree on a common format for the future, and rectify the WebM vs. non-WebM split, but continuing to promulgate sophisticated formats based on work outside of this scope (such as MPEG- or ITU-derived custom formats) just muddies this further. Apple is just blatantly not interested in cooperating with the rest of the market. Mozilla, Google and Microsoft are all part of AoM, along with a bunch of hardware companies and distributors yet Apple is absent and apparently pursuing HEVC instead. This isn't new either, Apple pushed HLS when others were pushing DASH and MPEG-TS when others were pushing fragmented MP4. They seem really determined to go their own way and ignore everyone else for some reason.

threeseed 9y ago
You're talking nonsense. H.265 is an ITU-T standard and was adopted as the standard for broadcast television ATSC. This isn't an Apple developed codec. And many more companies support it over AoM: http://www.mpegla.com/main/programs/HEVC/Pages/Licensors.aspx http://www.mpegla.com/main/programs/HEVC/Pages/Licensors.asp... http://www.mpegla.com/main/programs/HEVC/Pages/Licensees.aspx http://www.mpegla.com/main/programs/HEVC/Pages/Licensees.asp... http://www.hevcadvance.com/pdfnew/LicensorList.pdf http://www.hevcadvance.com/pdfnew/LicensorList.pdf http://www.hevcadvance.com/pdfnew/LicenseeList.pdf http://www.hevcadvance.com/pdfnew/LicenseeList.pdf Including the likes of: BBC, Samsung, Dolby, GE, MediaTek, Philips, Mitsubishi, Warner Bros, Sony etc. And as for hardware vendors well AMD, Intel, Nvidia etc all support H.265.
- Veratyr 9y ago
  > This isn't an Apple developed codec. I didn't say it was, I said Apple was pursuing it. > And many more companies support it over AoM You can't claim that yet as AV1 isn't yet finalized.
  - abritinthebay 9y ago
    > You can't claim that yet as AV1 isn't yet finalized. Do more companies support it? Yes. Failure to ship is also a competitive disadvantage - AV1 may be great (we don't know) but if this one gets traction first that may not matter.

jpap 9y ago

> This format doesn't really blaze new ground (EDIT: it does in the sense that it defines a container format to express image-y constructs like still images and sequences of images in an ISO media container, but see my other comment that asks how this is similar but different to video [3]), but if this repackaging and the resulting code donation and political clout helps it gain traction, we still would gain a lot. You need to think about HEIF as the container and disconnected from the codec used to compress the media content. In that sense, HEIF is a massive trailblazer. It gives us an ISO standard to describe image and mixed media consistent with existing methods (i.e. MP4) while allowing for new constructs (i.e. useful features of the future in addition to tiling, depth map, stereo, etc.) and new codecs as they are developed and adopted. HEIF is a universal forward-looking format. Until now, almost all of the major imaging formats were tied to the codec and not terribly extensible to new use cases. While BPG is a great format in the short term (it gave us HEVC coded images around 2.5 years ago), it isn't an ideal choice for the long term when viewed as above.

niutech 9y ago
TIFF and Exif (which JPEG is based on) are not tied to any particular codec and are extensible, so HEIF is not a game changer. BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif?
- jpap 9y ago
  > TIFF and Exif (which JPEG is based on) are not tied to any particular codec and are extensible, so HEIF is not a game changer. TIFF is a great format but its ability for extensibility is limited. You can't readily contain in a video track in addition to a photo, or a burst sequence that utilizes intra-frame prediction. > BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif? Yes; I once owned point-and-shoot cameras that did this. The audio was pretty poor quality because they didn't employ compression and wanted to keep the file sizes small. However you're limited to the 64k maximum JPEG marker segment size for non-image metadata; or ugly hacks like chaining segments (e.g. ICC profiles). Exif is strictly limited to being contained within 64k. How big is your audio track again? ;-) JPEG has had its time as a wildly successful format but has also held back the imaging world from adopting a standardized way to cater for new applications; burst above, mixed media (photo + video + audio), depth maps, stereo images, alpha (that's not a hack), lossless, etc., etc. HEIF has all the key ingredients, including extensibility, to support these modern applications and grow as the universal container format for decades to come.