14 ms·
Gzip files can just be straight concatenated together. There are some caveats around metadata but it’s an amazing property.
by bradleyjg 5y ago
Gzip files can just be straight concatenated together. There are some caveats around metadata but it’s an amazing property.
- nzmsv 5y agoBut only if the first file ends on a byte boundary of course. Just to save someone trying this some debugging :)
- sedatk 5y ago> byte boundary Do GZip files have bit-based lengths?
- nzmsv 5y agoYes they do!
- sedatk 5y agoWow, I wonder what led to that design decision. Does that mean you have to take that into account when concatenating them?
- nzmsv 5y agoWell, the whole article is about the flaws in the format :) But the answer in this case is Huffman coding. The most common sequences compress to just a few bits, but backreferences and such are not byte-aligned, they just follow literals in the bitstream. Aligning things to bytes would significantly reduce the compression ratio.
- 8note 5y agoIs that the compressed file that ends on the byte boundary, or the uncompressed one?
- nzmsv 5y agoCompressed. The compressed stream isn't byte aligned. Normally this isn't a problem because an application will flush the output and pad it to end on a byte boundary before closing a file (see https://www.bolet.org/~pornin/deflate-flush.html https://www.bolet.org/~pornin/deflate-flush.html for a good explanation) Where this becomes a problem is when someone thinks "I'll compress my logs as I write them out". The next problem is what to do on restart. It's tempting to think "I can just append to the existing file because I can concatenate two gzip files". But if the restart is after a crash rather than a clean exit you get log salad.