4 ms·

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ https://gregoryszorc.com/blog/2017/03/07/better-compression-...

dspillett 4y ago
pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.
- walrus01 4y ago
  yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
  - dspillett 4y ago
    Aye, pick the right tool for the target audience. If you are the target or you know everyone else who needs to read the output will have the ability to read zstd, go with that. If not consider pigz. If writing a script that others may run, have it default to gzip but use pigz if available (unless you really don't want that small % drop on compression).
- jonnycomputer 4y ago
  I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?
  - walrus01 4y ago
    zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options: HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable .gz support, by setting HAVE_ZLIB=0. Example : make zstd HAVE_ZLIB=0 It's also possible to force compilation with zlib support, using HAVE_ZLIB=1. In which case, linking stage will fail if zlib library cannot be found. This is useful to prevent silent feature disabling.
    - jonnycomputer 4y ago
      Yes, indeed I had already read that in the man page, but I did not feel it answered my question, because in my mind, and state of naivety about compression, logically at least, a compression algorithm and final file format needn't stand in a 1-1 relation. But I suppose that ZLIB is just the DEFLATE algorithm, then?
      - walrus01 4y ago
        yes, zlib is the library underpinning traditional gzip zstandard's own zstandard-own-compression format is its own separate much newer thing. zlib, for instance https://packages.debian.org/source/buster/zlib https://packages.debian.org/source/buster/zlib

dspillett 4y ago

pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.

walrus01 4y ago
yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
- dspillett 4y ago
  Aye, pick the right tool for the target audience. If you are the target or you know everyone else who needs to read the output will have the ability to read zstd, go with that. If not consider pigz. If writing a script that others may run, have it default to gzip but use pigz if available (unless you really don't want that small % drop on compression).
jonnycomputer 4y ago
I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?
- walrus01 4y ago
  zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options: HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable .gz support, by setting HAVE_ZLIB=0. Example : make zstd HAVE_ZLIB=0 It's also possible to force compilation with zlib support, using HAVE_ZLIB=1. In which case, linking stage will fail if zlib library cannot be found. This is useful to prevent silent feature disabling.
  - jonnycomputer 4y ago
    Yes, indeed I had already read that in the man page, but I did not feel it answered my question, because in my mind, and state of naivety about compression, logically at least, a compression algorithm and final file format needn't stand in a 1-1 relation. But I suppose that ZLIB is just the DEFLATE algorithm, then?
    - walrus01 4y ago
      yes, zlib is the library underpinning traditional gzip zstandard's own zstandard-own-compression format is its own separate much newer thing. zlib, for instance https://packages.debian.org/source/buster/zlib https://packages.debian.org/source/buster/zlib