4 ms·
What is a good example of a well-designed file format. It seems like file formats are often designed according to subjective opinions and tastes. Is there "un
by 1vuio0pswjnm7 5y ago
What is a good example of a well-designed file format. It seems like file formats are often designed according to subjective opinions and tastes. Is there "universally-accepted" guidance to designing them.
- LeoPanthera 5y agoLzip, which is a better version of xz, is designed to be well designed. If you see what I mean. https://www.nongnu.org/lzip/lzip_talk_ghm_2019.html https://www.nongnu.org/lzip/lzip_talk_ghm_2019.html
- tyingq 5y agoI'd mostly settle for "documented well". Like the Sqlite file format: https://www.sqlite.org/fileformat.html https://www.sqlite.org/fileformat.html Also nice that it's platform independent. Same file works on big/small endian machines.
- nayuki 5y agoFittingly, the author of SQLite proposed an alternative to Zip files based on SQLite database files: https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html ; https://sqlite.org/sqlar/doc/trunk/README.md https://sqlite.org/sqlar/doc/trunk/README.md
- liuliu 5y agoAt this point, just don't design file format. Use flatbuffers. 1. It has good forward / backward compatibility stories; 2. Schema is your file format documentation; 3. Don't need to decode every parts; 4. Can skip for unknowns (if you use union type); 5. Many language bindings. The only downside, and this is a big one: validate a flatbuffers payload method doesn't have nearly as much language coverage as the encode / decode methods. (i.e. C++ has it, but a lot of other languages don't have validate method implemented). Case study - Apache Arrow uses flatbuffers to persist its metadata: https://github.com/apache/arrow/tree/master/format https://github.com/apache/arrow/tree/master/format
- sliken 5y agoHeh, well that's the point of the article. Zip as an example of what not to do. So to improve things a file format should have a complete specification, conformance tests, have a fingerprint, a version number, variable length records should have lengths (to allow skipping records you don't understand), should allow for 64 bit lengths, and there should be index (and not a partial or incorrect index which is legal in zip). On top of that, anything intent on allow archives should provide for checksums and error correction (Reed-Solomon or similar).
- mjevans 5y agoAt a glance the, developed with years of hindsight later, 7zip specification (the official version of which might be in the SDK?) appears to be a conformant counter-example to a zip file format. This is a more easily accessed derivitive of that specification. https://py7zr.readthedocs.io/en/latest/archive_format.html https://py7zr.readthedocs.io/en/latest/archive_format.html
- jiggawatts 5y agoDespite using the flawed ZIP structure, I was moderately impressed by the "System.IO.Packaging" library in .NET, which is an implementation of the Open Packaging Conventions (OPC) that underpins the various MS Office document formats. It has a nice central directory, an extensibility mechanism, and even supports incremental saving of large documents in a reasonably sane way. Another fancy feature only typically seen in video container formats is the ability to interleave chunks to enable streaming loads or efficient storage on mechanical drives. Unfortunately, layering XML on top of ZIP is hideously inefficient for all sorts of reasons. IMHO, an ideal generic format should have some of the high-level API features of OPC, but implemented with a low-level storage encoding that is: 1) Linear in the sense that no O(n^2) or even O(n log n) algorithms are never needed (such as XML parsing). Reading the data should be a single-pass, with all buffer sizes known in advance. It's okay to sacrifice write speed, because writing is less common than reading, and often scales 1:many. E.g.: a file written once is often read many times. It's almost never the case that a file is repeatedly overwritten and read only once. Similarly, writes can be cached using write-back trivially, but reads are often uncachable and are on the critical path for end-user perceived latency. 2) Inherently paralleliseable. A mistake inherited from the 1990s era of file formats is strictly sequential formats that can never be decoded faster than whatever a single CPU core can do. At a very small compression efficiency loss of about 1-5%, huge wins can be achieved by simply restarting the compressor every 64KB or so. Similarly, instead of a naive hash function, use a Merkle tree, which can be verified in parallel. 3) Safe in the sense that overlapping block references are never allowed, and the decoder code should be generated from the schema in such as way that this is strictly enforced. This would allow languages like Rust or C# to safely return "span" references to the file data buffer without having to break it up into a million tiny allocations. If aliasing is allowed, this breaks the memory model of several languages.
- cesarb 5y agoI always liked the PNG file format (http://www.libpng.org/pub/png/spec/1.2/png-1.2-pdg.html http://www.libpng.org/pub/png/spec/1.2/png-1.2-pdg.html) as a good example of a well-designed file format. It starts with a fixed eight-byte signature, followed by a series of chunks, each chunk having exactly the same four fields: the chunk length, the chunk type, a variable-sized data, and a CRC over the type and data. The chunk type includes a flag telling the decoder whether the chunk can be skipped if the type is unknown (since all chunks have a length in the same place, every decoder will know how to skip the chunk). The specification also has a rationale section explaining the reason for several of its choices.
- a_e_k 5y agoRIFF [0] (based on IFF) is a generic chunked format that's not too different. It's the basis for WAV, AVI, and WebP, among others [1]. [0] https://johnloomis.org/cpe102/asgn/asgn1/riff.html https://johnloomis.org/cpe102/asgn/asgn1/riff.html [1] https://en.wikipedia.org/wiki/Resource_Interchange_File_Format https://en.wikipedia.org/wiki/Resource_Interchange_File_Form...
- d110af5ccf 5y agoJpeg2000 is also a really neat format. Among other things, it supports progressive decoding and tiling. (https://en.wikipedia.org/wiki/JPEG_2000 https://en.wikipedia.org/wiki/JPEG_2000)
- DaiPlusPlus 5y agoI've seen PNG as derivative of RIFF: in that it's a file comprised of atomic "chunks" (though PNG's chunks are much larger and not time-based), RIFF itself is basically the same as EA's IFF format from 1985 - so it's not like good ideas like a well-reasoned file-format are a recent innovation. It does seem a bit weird to think that something as highly technical (and thus: subject to becoming stale quickly) as a multimedia file format has withstood the test of time so well - while "simpler" files like Office documents from 15 years ago are almost unreadable today.
- 5y ago
- asah 5y agoMS Word 97 binary doc format with OLE. /s
- genmud 5y agoThat hit me right in the feels, lol. > Microsoft engineer looks at crack pipe... “Yea, let’s put an entire filesystem inside this new document format”
- unilynx 5y agoYou mean like ODF and DOCX still are zip files, basically a file system? It was there for a reason - you could embed documents/excelsheets/external apps that supported OLE into each other (think adding an editable excel graph to your document). And the FAT-like OLE file format allowed you to do partial updates to a file in place, which was pretty useful given the slow write speeds of hard disks or even floppy drives. Fully rewriting the whole ZIP/DOCX file on ever Save like we can do now wasn't an option then. I'm not sure if, given the requirements driven by that embedding use case, the engineers could have come up with something better in the 90s.