3 ms·
But only if the first file ends on a byte boundary of course. Just to save someone trying this some debugging :)
by nzmsv 5y ago
But only if the first file ends on a byte boundary of course. Just to save someone trying this some debugging :)
- sedatk 5y ago> byte boundary Do GZip files have bit-based lengths?
- nzmsv 5y agoYes they do!
- sedatk 5y agoWow, I wonder what led to that design decision. Does that mean you have to take that into account when concatenating them?
- nzmsv 5y agoWell, the whole article is about the flaws in the format :) But the answer in this case is Huffman coding. The most common sequences compress to just a few bits, but backreferences and such are not byte-aligned, they just follow literals in the bitstream. Aligning things to bytes would significantly reduce the compression ratio.
- 8note 5y agoIs that the compressed file that ends on the byte boundary, or the uncompressed one?
- nzmsv 5y agoCompressed. The compressed stream isn't byte aligned. Normally this isn't a problem because an application will flush the output and pad it to end on a byte boundary before closing a file (see https://www.bolet.org/~pornin/deflate-flush.html https://www.bolet.org/~pornin/deflate-flush.html for a good explanation) Where this becomes a problem is when someone thinks "I'll compress my logs as I write them out". The next problem is what to do on restart. It's tempting to think "I can just append to the existing file because I can concatenate two gzip files". But if the restart is after a crash rather than a clean exit you get log salad.