2 ms·
Assuming your decompressor is as capable as gzip(1) is, you don't need to write your own compressor or rely on block alignment, because: • pigz (https://github
by jwilk 4y ago
Assuming your decompressor is as capable as gzip(1) is, you don't need to write your own compressor or rely on block alignment, because:
• pigz (https://github.com/madler/pigz https://github.com/madler/pigz) has -0 for no-compression compression;
• you can concat multiple gzip files together.
$ echo foo | gzip > foobar.gz
$ echo bar | pigz -0 >> foobar.gz
$ echo baz | gzip >> foobar.gz
$ hd foobar.gz | grep -C1 -w bar
00000010 a8 65 32 7e 04 00 00 00 1f 8b 08 00 70 bf 8c 63 |.e2~........p..c|
00000020 00 03 01 04 00 fb ff 62 61 72 0a e9 b3 a2 04 04 |.......bar......|
00000030 00 00 00 1f 8b 08 00 00 00 00 00 00 03 4b 4a ac |.............KJ.|
$ gzip -d < foobar.gz
foo
bar
baz
This also solves the CRC problem, because pigz computes the checksum for you.
- rwmj 4y agoReally interesting - I didn't know you could concatenate gzip files, and indeed this would have worked better. Another thing I found out while researching this earlier is the pigz -i / --independent option. It lets you create a seekable gzip file, albeit one which cannot easily be indexed except by brute force searching over the whole file for the block boundaries (so not as useful as seekable xz like we use here: https://libguestfs.org/nbdkit-xz-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html)