7 ms·

Question: why do we see stable video and audio "container formats" like MKV that persist as encodings come and go (where you might not be able to play a new .mk

by derefr 3y ago

Question: why do we see stable video and audio "container formats" like MKV that persist as encodings come and go (where you might not be able to play a new .mkv file on an old player, but the expected answer to that is to upgrade your player to a new version, with universal support for pretty much any encoding being an inevitability on at least all software players); but every new image encoding seemingly necessitates its own new container format and file extension, and a minor format war to decide who will support it? Is this because almost all AV decoders use libffmpeg or a fork thereof; where libffmpeg is basically an "uber-library" that supports all interesting AV formats and codecs; and therefore you can expect ~everything to get support for a new codec whenever libffmpeg includes it (rather than some programs just never ending up supporting the codec)? If so — is there a reason that there isn't a libffmpeg-like uber-library for image formats+codecs?

DaleCurtis 3y ago
We're starting to see a move towards this with HEIF / AVIF containers, however in cases where "every bit must be saved" the general purpose containers like ISO-BMFF introduce some wastage that is unappealing.
- derefr 3y ago
  > however in cases where "every bit must be saved" the general purpose containers like ISO-BMFF introduce some wastage that is unappealing. Sure, but I don't mean general-purpose mulimedia containers (that put a lot of work into making multiple streams seekable with shared timing info.) I mean bit-efficient, image-oriented, but image-encoding-neutral container formats. There are at least two already-existing extensible image file formats that could be used for this: PNG and TIFF. In fact, TIFF was designed for this purpose — and even has several different encodings it supports! But in practice, you don't see the people who create new image codecs these days thinking of themselves as creating image codecs — they think of themselves as creating vertically-integrated image formats-plus-codecs. You don't see the authors of these new image specifications thinking "maybe I should be neutral on container format for this codec, and instead just specify what the bitstream for the image data looks like and what metadata would need to be stored about said bitstream to decode it in the abstract; and leave containerizing it to someone else." Let alone do you ever see someone think "hey, maybe I should invent a codec... and then create multiple reference implementations for how it would be stored inside a TIFF container, a PNG container, an MKV container..."
  - zokier 3y ago
    But HEIC/AVIF did exactly that, defined image format on top of standard container (isobmff/heif). JPEG-XL is the odd one out because it doesn't have standardized HEIF format, but for example JPEG-XS and JPEG-XR are supported in HEIF.
    - jonsneyers 3y ago
      JPEG XL uses the ISOBMFF container, with an option to skip the container completely and just use a raw codestream. HEIF is also ISOBMFF based but adds more mandatory stuff so you end up with more header overhead, and it adds some functionality at the container level (like layers, or using one codestream for the color image and another codestream for the alpha channel) that is useful for codecs that don't have that functionality at the codestream level — like video codecs which typically only support yuv, so if you want to do alpha you have to do it with one yuv frame and one yuv400 frame, and use a container like HEIF to indicate that the second frame represents the alpha channel. So if you want to use a video codec like HEVC or AV1 for still images and have functionality like alpha channels, ICC profiles, or orientation, then you need such a container since these codecs do not natively support those things. But for JPEG XL this is not needed since JPEG XL already does have native support for all of these things — it was designed to be a still image codec after all. It's also more effective for compression to support these things at the codec level, e.g. in JPEG XL you can have an RGBA palette which can be useful for lossless compression of certain images, while in HEIC/AVIF this is impossible since the RGB and A are in two different codestreams which are independent from one another and only combined at the container level. It would be possible to define a JPEG XL payload for the HEIF container but it would not really bring anything except a few hundred bytes of extra header overhead and possibly some risk of patent infringement since the IP situation of HEIF is not super clear (Nokia claims it has relevant patents on it, and those are not expired yet).
      - zokier 3y ago
        > JPEG XL uses the ISOBMFF container, with an option to skip the container completely and just use a raw codestream Hey, thanks for the clarification! I was basing my info on Wikipedia (my bad), ISO BMFF page doesn't mention JXL at all, and even JPEG XL page has only small print in infobox saying that its "based on" ISO BMFF but the main article text doesn't mention that at all. > But for JPEG XL this is not needed since JPEG XL already does have native support for all of these things — it was designed to be a still image codec after all I suppose that is bit the thing grand-parent comment was complaining about, format not designed for general-purpose containers but rather as an standalone thing. I suppose it could be fun thought experiment to imagine what JXL would look like if it were specifically designed to be used in HEIF. Of course it is well understandable that making tailored purpose-built format ends up better in many ways vs trying to fit into some existing generic thing. > It would be possible to define a JPEG XL payload for the HEIF container but it would not really bring anything except a few hundred bytes of extra header overhead and possibly some risk of patent infringement since the IP situation of HEIF is not super clear (Nokia claims it has relevant patents on it, and those are not expired yet). I suppose JXL-in-HEIF would allow some image management tools to have common code path for handling JXL and HEIC/AVIF files, grabbing metadata etc, and possibly would not need any specific JXL support. But that is probably not a practical concern in reality.
- netol 3y ago
  And at the same time, we are likely going to use codec-specific extensions for all AOM video codecs (.av1, .av2) as well as for images (.webp2, not sure if .avif2 will ever exist but I guess so), even when the same container is used, as we did with .webm (which was a subset of .mkv)
capableweb 3y ago
I'm guessing it's mostly up to mostly tradition/momentum on how the formats where initially created and maintained. Videos has (most of the time at least) at least two tracks at the same time that has to be syncronized, and most of the time it's one video track and one audio track. With that in mind, it makes sense to wrap those in a "container" and allow the video and audio to be different formats. You also can have multiple audios/video tracks in one file, but I digress. With images, it didn't make sense at least in the beginning, to have one container because you just have one image (or many, in the case of .gif).
killerstorm 3y ago
Video container formats do something useful: they let you to package several streams together (audio, video, subtitles), and they can take of some important aspects of av streaming, letting codec part to focus on being a codec. They let you to use existing audio codecs with a new video codec. OTOH a still image container would do nothing useful. If an image is all that needs to be contained, there's no need for a wrapper.
- yread 3y ago
  That's a very basic view, take a look at TIFF or DICOM specs. It can be useful to have multiple images, resolutions, channels, z or t dimensions, metadata, ... all in a single container as it's all one "image"
  - Groxx 3y ago
    captions / alt-text could also very reasonably be part of the image, as well as descriptions of regions and other metadata. there are LOTS of uses for "image containers" that go beyond just pixels. heck, look at EXIF, which is extremely widespread - it's often stripped to save space on the web, but it's definitely useful and used.
- derefr 3y ago
  > a still image container would do nothing useful It would, at least, create a codec-neutral location and format for image metadata, with codec-neutral (and ideally extensible + vendor-namespaced) fields. EXIF is just a JPEG thing. There is a reason that TIFF is still to this day used in medical imaging — it allows embedding of standardized medical-namespace metadata fields. Also, presuming the container format itself is extensible, it would also allow the PNG approach to ancillary data embedding ("allow optional chunks with vendor-specific meanings, for data that can be useful to clients, but which image processors can know it's safe to strip without understanding because 'is optional' is a syntactic part of the chunk name") to be used with arbitrary images — in a way where those chunks can even survive the image being transcoded! (If you're unaware, when you transcode a video file between video codecs using e.g. Handbrake, ancillary data like thumbnail and subtitle tracks will be ported as-is to the new file, as long as the new container format also supports those tracks.) Also, speaking of subtitle tracks, here's something most people may have never considered: you know how video containers can embed "soft" subtitle tracks? Why shouldn't images embed "soft" subtitle tracks, in multiple languages? Why shouldn't you expect your OS screen-reader feature to be able to read you your accessibility-enabled comic books in your native language — and in the right order (an order that, for comic books, a simple OCR-driven text extraction could never figure out)? (There are community image-curation services that allow images to be user-annotated with soft subtitles; but they do it by storing the subtitle data outside of the image file, in a database; sending the subtitle data separately as an XHR response after the image-display view loads; and then overlaying the soft-subtitle interaction-regions onto the image using client-side Javascript. Which makes sense in a world where users are able to freely edit the subtitles... but in a world where the subtitles are burned into the image at publication time by the author or publisher, it should be the browser [or other image viewer] doing this overlaying! Saving the image file should save the soft subtitles along with it! Just like when right-click-Save-ing a <video> element!)
dundarious 3y ago
Container formats for video often need to: - contain multiple streams of synced video, audio, and subtitles - contain alternate streams of audio - contain chapter information - contain metadata such as artist information For web distribution of static images, you want almost none of those things, especially regarding alternate streams. You just want to download the one stream you want. Easiest way to do that is to just serve each stream as a separate file, and not mux different streams into a single container in the first place. Also, I could be wrong on this part, but my understanding is that for web streaming video, you don't really want those mkv* features either. You typically serve individual and separate streams of video, audio, and text, sourced from separate files, and your player/browser syncs them. The alternative would be unnecessary demux on the server side, or the client unnecessarily downloads irrelevant streams. The metadata is the only case where I see the potential benefit of a single container format. * Not specific to mkv, other containers have them of course
- oefrha 3y ago
  > You typically serve individual and separate streams of video, audio, and text, sourced from separate files, and your player/browser syncs them. That's one school of thought. Some of the biggest streaming providers simply serve a single muxed video+audio HLS stream based on bandwidth detection. Doesn't work very well for multi-language prerecorded content of course, but that's just one use case.
  - dundarious 3y ago
    That's true, but my understanding is they serve a specific mux for a specific bandwidth profile, and serve it by just transmitting bytes, no demux required. I didn't mean to imply that wasn't a common option. I only meant to say I don't think a common option is to have a giant mux of all possible bandwidth profiles into one container file, that has to be demuxed at serve time. My understanding is that YouTube supports both the "separate streams" and "specific mux per-bandwidth profile" methods, and picks one based on the codec support/preferences of the client.
- pixl97 3y ago
  Container formats increase size. Now for video that doesn't matter much because it doesn't move the needle. For images a container format could be a significant percentage of the total image size.
brucethemoose2 3y ago
> with universal support for pretty much any encoding being an inevitability on at least all software players This is not a good assumption. MKV supports a loooot of things which many video players will not support at all. And IIRC some browsers do not support MKV.
- CharlesW 3y ago
  They all support ISOBMFF, though. https://en.wikipedia.org/wiki/ISO_base_media_file_format https://en.wikipedia.org/wiki/ISO_base_media_file_format
  - vanderZwan 3y ago
    But do they all support all of ISOBMFF?
Conscat 3y ago
There is ImageMagick, which last I checked (a year ago?) didn't support KTX, but it did have DDS and a lot of other niche image formats.
mananaysiempre 3y ago
The original entrant in this competition is TIFF, and—like Matroska or QuickTime add indexing to raw MP3 or MPEG-TS—it does provide useful functionality over raw codec stream non-formats like JPEG (properly JIF/JFIF/EXIF), in the form of striping or tiling and ready-made downscaled versions for the same image. But where unindexed video is essentially unworkable, an untiled image is in most cases OK, except for a couple of narrow application areas that need to deal with humongous amounts of pixel data. So you’re absolutely going to see TIFF containers with JPEG or JPEG2000 tiles used for geospatial, medical, or hi-res scanned images, but given the sad state of open tooling for all of these, there’s little to no compatibility between their various subsets of the TIFF spec, especially across vendors, and more or less no FOSS beyond libtiff. (Not even viewers for larger-than-RAM images!) Some other people have used TIFF but in places where’s very little to be gained from compatibility (e.g. Canon’s CR2 raw images are TIFF-based, but nobody cares). LogLuv TIFF is a viable HDR format, but it’s in an awkward place between the hobby-renderer-friendly Radiance HDR, the Pixar-backed OpenEXR, and whatever consumer photo thing each of the major vendors is pushing this month; it also doesn’t have a bit-level spec so much as a couple of journal articles and some code in libtiff. Why did this happen? Aside from the niche character of very large images, Adobe has abandoned the TIFF spec fairly quickly after it acquired it as part of Aldus, but IIUC for the first decade or so of that neglect Adobe legal was nevertheless fairly proactive about shutting up anyone who used the trademarked name for an incompatible extension (like TIFF64—and nowadays if you need TIFF you likely have >2G of data). Admittedly TIFF is also an overly flexible mess, but then so are Matroska (thus the need for the WebM profile of it) and QuickTime/BMFF (thus 3GPP, MOV, MP4, ..., which are vaguely speaking all subsets of the same thing). One way or another, TIFF is to some extent what you want, but it doesn’t get a lot of use these days. No browser support either, which is likely important. Maybe the HEIF container (yet another QuickTime/BMFF profile) is better from a technical standpoint, but the transitive closure of the relevant ISO specs likely comes at $10k or more. So it’s a bit sad all around.
- mcpackieh 3y ago
  > Admittedly TIFF is also an overly flexible mess, but then so are Matroska (thus the need for the WebM profile of it) Webm went way too far when they stripped out support for subtitles. The engineers who made that decision should be ashamed.
amelius 3y ago
Question: why don't we: 1. put a reference to the decoder into the header of the compressed file 2. download the decoder only when needed, and cache it if required 3. run the decoder inside a sandbox 4. allow multiple implementations, based on hardware, but at least one reference implementation that runs everywhere Then we never need any new formats. The system will just support any format. When you're not online, make sure you cached the decoders for whatever files you installed on your system.
- tinus_hn 3y ago
  We used to have formats like this, and then the attacker points to his decoder/malware package. Apart from that of course the decoder has to be fast and thus native and interface with the OS, so the decoder is X86 on the today version of Windows, until the company hosting it dies and the patented, copyrighted decoder disappears from the internet.
  - Dylan16807 3y ago
    Formats like what? You're not skipping the "sandbox" part are you? A decoder can be extremely isolated. It's much easier to sandbox a decoder than to sandbox javascript, for example.
    - tinus_hn 3y ago
      If you assume you just say the magic word ‘sandbox’ and everything is safe, then yes, security is a solved problem. This is however a prime example of the saying that in theory, there is no difference between theory and practice but in practice there is.
      - Dylan16807 3y ago
        You're treating sandboxing like it's all the same, but it's not. Multimedia decoders are one of the absolute easiest things to sandbox. Webassembly was designed for sandboxing and even that's very overcomplicated for what a decoder needs. If we had standard headers, then reading metadata wouldn't be part of the decoder. The decoder would only need to take in bytes and output a bitmap, or take in bytes and output PCM audio. It doesn't need to be able to call any functions, or run any system calls, and the data it outputs can safely contain any bytes because nothing will interpret it. It's like taking the very core of webassembly and then not attaching it to anything. The attack surface is astoundingly small. You just need to give it a big array of memory and let it run arithmetic within that array, plus some control flow instructions. Easy to interpret, easy to safely JIT compile.
jonsneyers 3y ago
Containers are just containers — you still need a decoder for their payload codec. This is the same for video and images. For video, containers are more important because you typically have several different codecs being used together (in particular video and audio) and the different bitstreams need to be interleaved. The ISOBMFF format is used as a container for MP4, JPEG 2000, JPEG XL, HEIF, AVIF, etc. And yes, there are ffmpeg-like "uber-libraries" for images: ImageMagick, GraphicsMagic, libvips, imlib2 and gdk-pixbuf are examples of those. They support basically all image formats, and applications based on one of these will 'automatically' get JPEG XL support. Apple also has such an "uber-library" called CoreMedia, which means any application that uses this library will also get JPEG XL support automatically.
quackdoc 3y ago
videos can have their own "containers" too, for instance in AV1 the stream is stored inside of an obu, which is wrapped in a external container. (such as matroska) if you really wanted to you could (and can) put images into containers too, PNGs in a matroska are actually pretty useful way for transfering PNG sequences. you can also with a simple mod on a older commit of ffmpeg (the commit that added animated jxl broke this method and I haven't gotten around to fixing it) by simply adding jxl 4cc to mux JXL sequences into MKV.
GuB-42 3y ago
I think that's because video is a much more active and complex topic than still images. We are still using image formats from the 90s, and their matching containers, and they are good enough, so there is not much work for going beyond that. There is no real incentive for making a more flexible format. By comparison, video is the biggest bandwidth hog and people care a lot. And mkv supports video, multiple sound tracks, subtitles,... All using different codecs made by different people (ex: h265+opus or vp9+vorbis, or any other combination). An image container usually only has the image and a few metadata.

by derefr 3y ago

DaleCurtis 3y ago

We're starting to see a move towards this with HEIF / AVIF containers, however in cases where "every bit must be saved" the general purpose containers like ISO-BMFF introduce some wastage that is unappealing.

derefr 3y ago
> however in cases where "every bit must be saved" the general purpose containers like ISO-BMFF introduce some wastage that is unappealing. Sure, but I don't mean general-purpose mulimedia containers (that put a lot of work into making multiple streams seekable with shared timing info.) I mean bit-efficient, image-oriented, but image-encoding-neutral container formats. There are at least two already-existing extensible image file formats that could be used for this: PNG and TIFF. In fact, TIFF was designed for this purpose — and even has several different encodings it supports! But in practice, you don't see the people who create new image codecs these days thinking of themselves as creating image codecs — they think of themselves as creating vertically-integrated image formats-plus-codecs. You don't see the authors of these new image specifications thinking "maybe I should be neutral on container format for this codec, and instead just specify what the bitstream for the image data looks like and what metadata would need to be stored about said bitstream to decode it in the abstract; and leave containerizing it to someone else." Let alone do you ever see someone think "hey, maybe I should invent a codec... and then create multiple reference implementations for how it would be stored inside a TIFF container, a PNG container, an MKV container..."
- zokier 3y ago
  But HEIC/AVIF did exactly that, defined image format on top of standard container (isobmff/heif). JPEG-XL is the odd one out because it doesn't have standardized HEIF format, but for example JPEG-XS and JPEG-XR are supported in HEIF.
  - jonsneyers 3y ago
    JPEG XL uses the ISOBMFF container, with an option to skip the container completely and just use a raw codestream. HEIF is also ISOBMFF based but adds more mandatory stuff so you end up with more header overhead, and it adds some functionality at the container level (like layers, or using one codestream for the color image and another codestream for the alpha channel) that is useful for codecs that don't have that functionality at the codestream level — like video codecs which typically only support yuv, so if you want to do alpha you have to do it with one yuv frame and one yuv400 frame, and use a container like HEIF to indicate that the second frame represents the alpha channel. So if you want to use a video codec like HEVC or AV1 for still images and have functionality like alpha channels, ICC profiles, or orientation, then you need such a container since these codecs do not natively support those things. But for JPEG XL this is not needed since JPEG XL already does have native support for all of these things — it was designed to be a still image codec after all. It's also more effective for compression to support these things at the codec level, e.g. in JPEG XL you can have an RGBA palette which can be useful for lossless compression of certain images, while in HEIC/AVIF this is impossible since the RGB and A are in two different codestreams which are independent from one another and only combined at the container level. It would be possible to define a JPEG XL payload for the HEIF container but it would not really bring anything except a few hundred bytes of extra header overhead and possibly some risk of patent infringement since the IP situation of HEIF is not super clear (Nokia claims it has relevant patents on it, and those are not expired yet).
    - zokier 3y ago
      > JPEG XL uses the ISOBMFF container, with an option to skip the container completely and just use a raw codestream Hey, thanks for the clarification! I was basing my info on Wikipedia (my bad), ISO BMFF page doesn't mention JXL at all, and even JPEG XL page has only small print in infobox saying that its "based on" ISO BMFF but the main article text doesn't mention that at all. > But for JPEG XL this is not needed since JPEG XL already does have native support for all of these things — it was designed to be a still image codec after all I suppose that is bit the thing grand-parent comment was complaining about, format not designed for general-purpose containers but rather as an standalone thing. I suppose it could be fun thought experiment to imagine what JXL would look like if it were specifically designed to be used in HEIF. Of course it is well understandable that making tailored purpose-built format ends up better in many ways vs trying to fit into some existing generic thing. > It would be possible to define a JPEG XL payload for the HEIF container but it would not really bring anything except a few hundred bytes of extra header overhead and possibly some risk of patent infringement since the IP situation of HEIF is not super clear (Nokia claims it has relevant patents on it, and those are not expired yet). I suppose JXL-in-HEIF would allow some image management tools to have common code path for handling JXL and HEIC/AVIF files, grabbing metadata etc, and possibly would not need any specific JXL support. But that is probably not a practical concern in reality.
netol 3y ago
And at the same time, we are likely going to use codec-specific extensions for all AOM video codecs (.av1, .av2) as well as for images (.webp2, not sure if .avif2 will ever exist but I guess so), even when the same container is used, as we did with .webm (which was a subset of .mkv)

capableweb 3y ago

I'm guessing it's mostly up to mostly tradition/momentum on how the formats where initially created and maintained. Videos has (most of the time at least) at least two tracks at the same time that has to be syncronized, and most of the time it's one video track and one audio track. With that in mind, it makes sense to wrap those in a "container" and allow the video and audio to be different formats. You also can have multiple audios/video tracks in one file, but I digress. With images, it didn't make sense at least in the beginning, to have one container because you just have one image (or many, in the case of .gif).

killerstorm 3y ago

Video container formats do something useful: they let you to package several streams together (audio, video, subtitles), and they can take of some important aspects of av streaming, letting codec part to focus on being a codec. They let you to use existing audio codecs with a new video codec. OTOH a still image container would do nothing useful. If an image is all that needs to be contained, there's no need for a wrapper.

yread 3y ago
That's a very basic view, take a look at TIFF or DICOM specs. It can be useful to have multiple images, resolutions, channels, z or t dimensions, metadata, ... all in a single container as it's all one "image"
- Groxx 3y ago
  captions / alt-text could also very reasonably be part of the image, as well as descriptions of regions and other metadata. there are LOTS of uses for "image containers" that go beyond just pixels. heck, look at EXIF, which is extremely widespread - it's often stripped to save space on the web, but it's definitely useful and used.
derefr 3y ago
> a still image container would do nothing useful It would, at least, create a codec-neutral location and format for image metadata, with codec-neutral (and ideally extensible + vendor-namespaced) fields. EXIF is just a JPEG thing. There is a reason that TIFF is still to this day used in medical imaging — it allows embedding of standardized medical-namespace metadata fields. Also, presuming the container format itself is extensible, it would also allow the PNG approach to ancillary data embedding ("allow optional chunks with vendor-specific meanings, for data that can be useful to clients, but which image processors can know it's safe to strip without understanding because 'is optional' is a syntactic part of the chunk name") to be used with arbitrary images — in a way where those chunks can even survive the image being transcoded! (If you're unaware, when you transcode a video file between video codecs using e.g. Handbrake, ancillary data like thumbnail and subtitle tracks will be ported as-is to the new file, as long as the new container format also supports those tracks.) Also, speaking of subtitle tracks, here's something most people may have never considered: you know how video containers can embed "soft" subtitle tracks? Why shouldn't images embed "soft" subtitle tracks, in multiple languages? Why shouldn't you expect your OS screen-reader feature to be able to read you your accessibility-enabled comic books in your native language — and in the right order (an order that, for comic books, a simple OCR-driven text extraction could never figure out)? (There are community image-curation services that allow images to be user-annotated with soft subtitles; but they do it by storing the subtitle data outside of the image file, in a database; sending the subtitle data separately as an XHR response after the image-display view loads; and then overlaying the soft-subtitle interaction-regions onto the image using client-side Javascript. Which makes sense in a world where users are able to freely edit the subtitles... but in a world where the subtitles are burned into the image at publication time by the author or publisher, it should be the browser [or other image viewer] doing this overlaying! Saving the image file should save the soft subtitles along with it! Just like when right-click-Save-ing a <video> element!)

dundarious 3y ago

Container formats for video often need to: - contain multiple streams of synced video, audio, and subtitles - contain alternate streams of audio - contain chapter information - contain metadata such as artist information For web distribution of static images, you want almost none of those things, especially regarding alternate streams. You just want to download the one stream you want. Easiest way to do that is to just serve each stream as a separate file, and not mux different streams into a single container in the first place. Also, I could be wrong on this part, but my understanding is that for web streaming video, you don't really want those mkv* features either. You typically serve individual and separate streams of video, audio, and text, sourced from separate files, and your player/browser syncs them. The alternative would be unnecessary demux on the server side, or the client unnecessarily downloads irrelevant streams. The metadata is the only case where I see the potential benefit of a single container format. * Not specific to mkv, other containers have them of course

oefrha 3y ago
> You typically serve individual and separate streams of video, audio, and text, sourced from separate files, and your player/browser syncs them. That's one school of thought. Some of the biggest streaming providers simply serve a single muxed video+audio HLS stream based on bandwidth detection. Doesn't work very well for multi-language prerecorded content of course, but that's just one use case.
- dundarious 3y ago
  That's true, but my understanding is they serve a specific mux for a specific bandwidth profile, and serve it by just transmitting bytes, no demux required. I didn't mean to imply that wasn't a common option. I only meant to say I don't think a common option is to have a giant mux of all possible bandwidth profiles into one container file, that has to be demuxed at serve time. My understanding is that YouTube supports both the "separate streams" and "specific mux per-bandwidth profile" methods, and picks one based on the codec support/preferences of the client.
pixl97 3y ago
Container formats increase size. Now for video that doesn't matter much because it doesn't move the needle. For images a container format could be a significant percentage of the total image size.

brucethemoose2 3y ago

> with universal support for pretty much any encoding being an inevitability on at least all software players This is not a good assumption. MKV supports a loooot of things which many video players will not support at all. And IIRC some browsers do not support MKV.

CharlesW 3y ago
They all support ISOBMFF, though. https://en.wikipedia.org/wiki/ISO_base_media_file_format https://en.wikipedia.org/wiki/ISO_base_media_file_format
- vanderZwan 3y ago
  But do they all support all of ISOBMFF?

Conscat 3y ago

There is ImageMagick, which last I checked (a year ago?) didn't support KTX, but it did have DDS and a lot of other niche image formats.

mananaysiempre 3y ago

The original entrant in this competition is TIFF, and—like Matroska or QuickTime add indexing to raw MP3 or MPEG-TS—it does provide useful functionality over raw codec stream non-formats like JPEG (properly JIF/JFIF/EXIF), in the form of striping or tiling and ready-made downscaled versions for the same image. But where unindexed video is essentially unworkable, an untiled image is in most cases OK, except for a couple of narrow application areas that need to deal with humongous amounts of pixel data. So you’re absolutely going to see TIFF containers with JPEG or JPEG2000 tiles used for geospatial, medical, or hi-res scanned images, but given the sad state of open tooling for all of these, there’s little to no compatibility between their various subsets of the TIFF spec, especially across vendors, and more or less no FOSS beyond libtiff. (Not even viewers for larger-than-RAM images!) Some other people have used TIFF but in places where’s very little to be gained from compatibility (e.g. Canon’s CR2 raw images are TIFF-based, but nobody cares). LogLuv TIFF is a viable HDR format, but it’s in an awkward place between the hobby-renderer-friendly Radiance HDR, the Pixar-backed OpenEXR, and whatever consumer photo thing each of the major vendors is pushing this month; it also doesn’t have a bit-level spec so much as a couple of journal articles and some code in libtiff. Why did this happen? Aside from the niche character of very large images, Adobe has abandoned the TIFF spec fairly quickly after it acquired it as part of Aldus, but IIUC for the first decade or so of that neglect Adobe legal was nevertheless fairly proactive about shutting up anyone who used the trademarked name for an incompatible extension (like TIFF64—and nowadays if you need TIFF you likely have >2G of data). Admittedly TIFF is also an overly flexible mess, but then so are Matroska (thus the need for the WebM profile of it) and QuickTime/BMFF (thus 3GPP, MOV, MP4, ..., which are vaguely speaking all subsets of the same thing). One way or another, TIFF is to some extent what you want, but it doesn’t get a lot of use these days. No browser support either, which is likely important. Maybe the HEIF container (yet another QuickTime/BMFF profile) is better from a technical standpoint, but the transitive closure of the relevant ISO specs likely comes at $10k or more. So it’s a bit sad all around.

mcpackieh 3y ago
> Admittedly TIFF is also an overly flexible mess, but then so are Matroska (thus the need for the WebM profile of it) Webm went way too far when they stripped out support for subtitles. The engineers who made that decision should be ashamed.

amelius 3y ago

Question: why don't we: 1. put a reference to the decoder into the header of the compressed file 2. download the decoder only when needed, and cache it if required 3. run the decoder inside a sandbox 4. allow multiple implementations, based on hardware, but at least one reference implementation that runs everywhere Then we never need any new formats. The system will just support any format. When you're not online, make sure you cached the decoders for whatever files you installed on your system.

tinus_hn 3y ago
We used to have formats like this, and then the attacker points to his decoder/malware package. Apart from that of course the decoder has to be fast and thus native and interface with the OS, so the decoder is X86 on the today version of Windows, until the company hosting it dies and the patented, copyrighted decoder disappears from the internet.
- Dylan16807 3y ago
  Formats like what? You're not skipping the "sandbox" part are you? A decoder can be extremely isolated. It's much easier to sandbox a decoder than to sandbox javascript, for example.
  - tinus_hn 3y ago
    If you assume you just say the magic word ‘sandbox’ and everything is safe, then yes, security is a solved problem. This is however a prime example of the saying that in theory, there is no difference between theory and practice but in practice there is.
    - Dylan16807 3y ago
      You're treating sandboxing like it's all the same, but it's not. Multimedia decoders are one of the absolute easiest things to sandbox. Webassembly was designed for sandboxing and even that's very overcomplicated for what a decoder needs. If we had standard headers, then reading metadata wouldn't be part of the decoder. The decoder would only need to take in bytes and output a bitmap, or take in bytes and output PCM audio. It doesn't need to be able to call any functions, or run any system calls, and the data it outputs can safely contain any bytes because nothing will interpret it. It's like taking the very core of webassembly and then not attaching it to anything. The attack surface is astoundingly small. You just need to give it a big array of memory and let it run arithmetic within that array, plus some control flow instructions. Easy to interpret, easy to safely JIT compile.

jonsneyers 3y ago

Containers are just containers — you still need a decoder for their payload codec. This is the same for video and images. For video, containers are more important because you typically have several different codecs being used together (in particular video and audio) and the different bitstreams need to be interleaved. The ISOBMFF format is used as a container for MP4, JPEG 2000, JPEG XL, HEIF, AVIF, etc. And yes, there are ffmpeg-like "uber-libraries" for images: ImageMagick, GraphicsMagic, libvips, imlib2 and gdk-pixbuf are examples of those. They support basically all image formats, and applications based on one of these will 'automatically' get JPEG XL support. Apple also has such an "uber-library" called CoreMedia, which means any application that uses this library will also get JPEG XL support automatically.

quackdoc 3y ago

videos can have their own "containers" too, for instance in AV1 the stream is stored inside of an obu, which is wrapped in a external container. (such as matroska) if you really wanted to you could (and can) put images into containers too, PNGs in a matroska are actually pretty useful way for transfering PNG sequences. you can also with a simple mod on a older commit of ffmpeg (the commit that added animated jxl broke this method and I haven't gotten around to fixing it) by simply adding jxl 4cc to mux JXL sequences into MKV.

GuB-42 3y ago

I think that's because video is a much more active and complex topic than still images. We are still using image formats from the 90s, and their matching containers, and they are good enough, so there is not much work for going beyond that. There is no real incentive for making a more flexible format. By comparison, video is the biggest bandwidth hog and people care a lot. And mkv supports video, multiple sound tracks, subtitles,... All using different codecs made by different people (ex: h265+opus or vp9+vorbis, or any other combination). An image container usually only has the image and a few metadata.