4 ms·

Heh, well that's the point of the article. Zip as an example of what not to do. So to improve things a file format should have a complete specification, confo

by sliken 5y ago

Heh, well that's the point of the article. Zip as an example of what not to do. So to improve things a file format should have a complete specification, conformance tests, have a fingerprint, a version number, variable length records should have lengths (to allow skipping records you don't understand), should allow for 64 bit lengths, and there should be index (and not a partial or incorrect index which is legal in zip). On top of that, anything intent on allow archives should provide for checksums and error correction (Reed-Solomon or similar).

mjevans 5y ago
At a glance the, developed with years of hindsight later, 7zip specification (the official version of which might be in the SDK?) appears to be a conformant counter-example to a zip file format. This is a more easily accessed derivitive of that specification. https://py7zr.readthedocs.io/en/latest/archive_format.html https://py7zr.readthedocs.io/en/latest/archive_format.html
jiggawatts 5y ago
Despite using the flawed ZIP structure, I was moderately impressed by the "System.IO.Packaging" library in .NET, which is an implementation of the Open Packaging Conventions (OPC) that underpins the various MS Office document formats. It has a nice central directory, an extensibility mechanism, and even supports incremental saving of large documents in a reasonably sane way. Another fancy feature only typically seen in video container formats is the ability to interleave chunks to enable streaming loads or efficient storage on mechanical drives. Unfortunately, layering XML on top of ZIP is hideously inefficient for all sorts of reasons. IMHO, an ideal generic format should have some of the high-level API features of OPC, but implemented with a low-level storage encoding that is: 1) Linear in the sense that no O(n^2) or even O(n log n) algorithms are never needed (such as XML parsing). Reading the data should be a single-pass, with all buffer sizes known in advance. It's okay to sacrifice write speed, because writing is less common than reading, and often scales 1:many. E.g.: a file written once is often read many times. It's almost never the case that a file is repeatedly overwritten and read only once. Similarly, writes can be cached using write-back trivially, but reads are often uncachable and are on the critical path for end-user perceived latency. 2) Inherently paralleliseable. A mistake inherited from the 1990s era of file formats is strictly sequential formats that can never be decoded faster than whatever a single CPU core can do. At a very small compression efficiency loss of about 1-5%, huge wins can be achieved by simply restarting the compressor every 64KB or so. Similarly, instead of a naive hash function, use a Merkle tree, which can be verified in parallel. 3) Safe in the sense that overlapping block references are never allowed, and the decoder code should be generated from the schema in such as way that this is strictly enforced. This would allow languages like Rust or C# to safely return "span" references to the file data buffer without having to break it up into a million tiny allocations. If aliasing is allowed, this breaks the memory model of several languages.
- bumblebritches5 5y ago
  Interesting ideas

by sliken 5y ago

mjevans 5y ago

At a glance the, developed with years of hindsight later, 7zip specification (the official version of which might be in the SDK?) appears to be a conformant counter-example to a zip file format. This is a more easily accessed derivitive of that specification. https://py7zr.readthedocs.io/en/latest/archive_format.html https://py7zr.readthedocs.io/en/latest/archive_format.html

jiggawatts 5y ago

Despite using the flawed ZIP structure, I was moderately impressed by the "System.IO.Packaging" library in .NET, which is an implementation of the Open Packaging Conventions (OPC) that underpins the various MS Office document formats. It has a nice central directory, an extensibility mechanism, and even supports incremental saving of large documents in a reasonably sane way. Another fancy feature only typically seen in video container formats is the ability to interleave chunks to enable streaming loads or efficient storage on mechanical drives. Unfortunately, layering XML on top of ZIP is hideously inefficient for all sorts of reasons. IMHO, an ideal generic format should have some of the high-level API features of OPC, but implemented with a low-level storage encoding that is: 1) Linear in the sense that no O(n^2) or even O(n log n) algorithms are never needed (such as XML parsing). Reading the data should be a single-pass, with all buffer sizes known in advance. It's okay to sacrifice write speed, because writing is less common than reading, and often scales 1:many. E.g.: a file written once is often read many times. It's almost never the case that a file is repeatedly overwritten and read only once. Similarly, writes can be cached using write-back trivially, but reads are often uncachable and are on the critical path for end-user perceived latency. 2) Inherently paralleliseable. A mistake inherited from the 1990s era of file formats is strictly sequential formats that can never be decoded faster than whatever a single CPU core can do. At a very small compression efficiency loss of about 1-5%, huge wins can be achieved by simply restarting the compressor every 64KB or so. Similarly, instead of a naive hash function, use a Merkle tree, which can be verified in parallel. 3) Safe in the sense that overlapping block references are never allowed, and the decoder code should be generated from the schema in such as way that this is strictly enforced. This would allow languages like Rust or C# to safely return "span" references to the file data buffer without having to break it up into a million tiny allocations. If aliasing is allowed, this breaks the memory model of several languages.

bumblebritches5 5y ago
Interesting ideas