Anyone else feel let down by all the existing container formats out there, and feel this itch to like, write their own? I know, I know, that's like the worst po

Anyone else feel let down by all the existing container formats out there, and feel this itch to like, write their own? I know, I know, that's like the worst possible idea, xkcd #927 "now there are 15 standards", but like, what if? What I'm looking for is basically something that ticks all the boxes of LOC's 7 Sustainability Factors. Is able to store arbitrary binary data, along with metadata, in a single file. Easy to pack/unpack, particularly in Python. I'm pretty well covered for image/audio/video formats, but none of those lend themselves to n-dim arrays. Ogg, matroska, and HDF5 are all container formats which can ostensibly do this, possibly even well. Ogg and matroska are supposed to be able to support arbitrary data steams and metadata, but actually finding tooling to do the operations I want to do has yet to bear fruit. HDF5 is insanely complex, and I have some concerns about performance and data corruption risk. Tar is tempting but it does some really goofy things, like ending an archive with 1kB of zeros. What actually looks most promising as an existing "format" is the RIFF family of chunked encoding. Dead simple, python stdlib even has a reader. Only real downside is 2*32 ~= 4 gigabyte size limit, though this can be overcome by using a "stream of chunks" model. Alternatively, a stream based on Msgpack, with some slight constraints, is very appealing. Sqlite3, also very tempting. If I did write such a "format", it would definitely be built on the primitives of either RIFF, Msgpack, or Sqlite-as-application-file, so it'd be less of a "file format" and more of "existing format with some opinions about schema". Protip for anyone also thinking about making their own file format: make sure the API/format supports some sort of arbitrary metadata model! Apache Feather and Parquet are pretty recent, but don't really support writing metadata, you kind of have to hack around it. IMHO, this is a huge oversight, especially considering the internal metadata is literally just JSON-encoded fields. Edit: actually just reread the follow-up to "moving away from hdf5" and it mentions ASDF format. This looks really promising. YAML metadata header with binary blocks. Normally it's not a good idea to start with UTF8 and later go binary, but if you know when to stop reading text, it's not the worst. Also, it's a name collision with the asdf version manager (and also anything else named after rolling the left hand home keys) https://cyrille.rossant.net/should-you-use-hdf5/ https://cyrille.rossant.net/should-you-use-hdf5/ https://towardsdatascience.com/saving-metadata-with-dataframes-71f51f558d8e https://towardsdatascience.com/saving-metadata-with-datafram... https://en.m.wikipedia.org/wiki/Resource_Interchange_File_Format https://en.m.wikipedia.org/wiki/Resource_Interchange_File_Fo... https://msgpack.org/index.html https://msgpack.org/index.html https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html https://asdf-standard.readthedocs.io/en/latest/ https://asdf-standard.readthedocs.io/en/latest/