3 ms·
Despite using the flawed ZIP structure, I was moderately impressed by the "System.IO.Packaging" library in .NET, which is an implementation of the Open Packagin
by jiggawatts 5y ago
Despite using the flawed ZIP structure, I was moderately impressed by the "System.IO.Packaging" library in .NET, which is an implementation of the Open Packaging Conventions (OPC) that underpins the various MS Office document formats.
It has a nice central directory, an extensibility mechanism, and even supports incremental saving of large documents in a reasonably sane way. Another fancy feature only typically seen in video container formats is the ability to interleave chunks to enable streaming loads or efficient storage on mechanical drives.
Unfortunately, layering XML on top of ZIP is hideously inefficient for all sorts of reasons.
IMHO, an ideal generic format should have some of the high-level API features of OPC, but implemented with a low-level storage encoding that is:
1) Linear in the sense that no O(n^2) or even O(n log n) algorithms are never needed (such as XML parsing). Reading the data should be a single-pass, with all buffer sizes known in advance. It's okay to sacrifice write speed, because writing is less common than reading, and often scales 1:many. E.g.: a file written once is often read many times. It's almost never the case that a file is repeatedly overwritten and read only once. Similarly, writes can be cached using write-back trivially, but reads are often uncachable and are on the critical path for end-user perceived latency.
2) Inherently paralleliseable. A mistake inherited from the 1990s era of file formats is strictly sequential formats that can never be decoded faster than whatever a single CPU core can do. At a very small compression efficiency loss of about 1-5%, huge wins can be achieved by simply restarting the compressor every 64KB or so. Similarly, instead of a naive hash function, use a Merkle tree, which can be verified in parallel.
3) Safe in the sense that overlapping block references are never allowed, and the decoder code should be generated from the schema in such as way that this is strictly enforced. This would allow languages like Rust or C# to safely return "span" references to the file data buffer without having to break it up into a million tiny allocations. If aliasing is allowed, this breaks the memory model of several languages.
- bumblebritches5 5y agoInteresting ideas