4 ms·
I was looking at the details of the ZIP format just last week when I discovered that a major performance issue was being caused by quirks of a ZIP implementatio
by EvanAnderson 5y ago
I was looking at the details of the ZIP format just last week when I discovered that a major performance issue was being caused by quirks of a ZIP implementation. Some knowledge of the ZIP format on the developer's part would have saved massive I/O loads (and WAN traffic).
There were various "archive" ZIP files that had grown to several gigabytes each. The developer was using Info-ZIP "zip.exe" to "append" files to the archives multiple times per hour (>100). Like virtually every ZIP implementation, Info-ZIP just re-writes the entire file when appending. As a result, I was seeing multiple terabytes of I/O churn each day as new ZIP files were being created to replace the old ones (which was killing VM replication across a 500Mbps pipe, and is what drew my attention to begin with).
While I would have loved to find a ZIP implementation that just appended the file and a new central directory I found it easier to just rename the ZIP files every day to minimize the churn. Still, I can't believe there isn't a ZIP implementation out there that can do this.
- bradleyjg 5y agoGzip files can just be straight concatenated together. There are some caveats around metadata but it’s an amazing property.
- nzmsv 5y agoBut only if the first file ends on a byte boundary of course. Just to save someone trying this some debugging :)
- sedatk 5y ago> byte boundary Do GZip files have bit-based lengths?
- nzmsv 5y agoYes they do!
- sedatk 5y agoWow, I wonder what led to that design decision. Does that mean you have to take that into account when concatenating them?
- nzmsv 5y agoWell, the whole article is about the flaws in the format :) But the answer in this case is Huffman coding. The most common sequences compress to just a few bits, but backreferences and such are not byte-aligned, they just follow literals in the bitstream. Aligning things to bytes would significantly reduce the compression ratio.
- 8note 5y agoIs that the compressed file that ends on the byte boundary, or the uncompressed one?
- nzmsv 5y agoCompressed. The compressed stream isn't byte aligned. Normally this isn't a problem because an application will flush the output and pad it to end on a byte boundary before closing a file (see https://www.bolet.org/~pornin/deflate-flush.html https://www.bolet.org/~pornin/deflate-flush.html for a good explanation) Where this becomes a problem is when someone thinks "I'll compress my logs as I write them out". The next problem is what to do on restart. It's tempting to think "I can just append to the existing file because I can concatenate two gzip files". But if the restart is after a crash rather than a clean exit you get log salad.
- brandmeyer 5y agoDoes this meet your needs? https://sqlite.org/zipfile.html https://sqlite.org/zipfile.html
- Hello71 5y agoHuh? I just checked the Info-ZIP zip man page, and it says (reformatted): > -g, --grow: Grow (append to) the specified zip archive, instead of creating a new one. If this operation fails, zip attempts to restore the archive to its original state. If the restoration fails, the archive might become corrupted. This option is ignored when there's no existing archive or when at least one archive member must be updated or deleted. I tested this on Linux and it appears to work as advertised. I assume the reason why it is not enabled by default is because "If the restoration fails, the archive might become corrupted". edit: python zipfile in append mode also appears to avoid rewriting the zip file. libarchive doesn't seem to support appending to zip files.
- EvanAnderson 5y agoWell I feel like a dunce. Thanks for this tip! I didn't see this option on the version of Info-ZIP they were using but that would definitely do what I wanted. I thought I looked at current man pages but I guess I didn't.
- EvanAnderson 5y agoI checked up on this. Turns out the "-g" option isn't listed in the "-?" help. I guess I didn't look at the man page-- just the "-?" help.
- d110af5ccf 5y agoAnother commenter pointed out that Info-ZIP supports append, but alternatively it seems like tar is built for exactly this sort of thing. Instead of the usual gzipped tar archive (ie .tar.gz), just gzip (or something else) and then append the result to an existing tar archive. (https://www.gnu.org/software/tar/manual/html_node/appending-files.html https://www.gnu.org/software/tar/manual/html_node/appending-...) Also I'd preferably use zstd because it's a significant improvement in many ways and would afford you the option of placing a custom compression dictionary at the front of the tar archive.