3 ms·

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ htt

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller. megs 3781 test.tar 3041 test.tar.zstd (default compression 3, 3.5s) 3170 test.tar.bz2 (default compression, 8 threads, 40s)

ac29 4y ago
bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)
- sitkack 4y ago
  Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder.html#method.multithread https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...
  - iruoy 4y ago
    Decompression is multithreaded by default. Compression with an argument. However it is built-in.
walrus01 4y ago
on the other hand, bzip2 is pretty much obsoleted now by xzip
- booi 4y ago
  What is xzip? are you talking about xz?
  - walrus01 4y ago
    yes, xz section 3.6 here https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mar... https://en.wikipedia.org/wiki/XZ_Utils https://en.wikipedia.org/wiki/XZ_Utils
    - chasil 4y ago
      The author of lzip has pointed criticism for the design choices of xz. I generally use lzip for data that is important to me. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- bmn__ 4y ago
  Also see: http://enwp.org/zstd http://enwp.org/zstd parallel-friendly, trades off compression level for speed

ac29 4y ago

bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)

sitkack 4y ago
Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder.html#method.multithread https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...
- iruoy 4y ago
  Decompression is multithreaded by default. Compression with an argument. However it is built-in.

walrus01 4y ago

on the other hand, bzip2 is pretty much obsoleted now by xzip

booi 4y ago
What is xzip? are you talking about xz?
- walrus01 4y ago
  yes, xz section 3.6 here https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mar... https://en.wikipedia.org/wiki/XZ_Utils https://en.wikipedia.org/wiki/XZ_Utils
  - chasil 4y ago
    The author of lzip has pointed criticism for the design choices of xz. I generally use lzip for data that is important to me. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
bmn__ 4y ago
Also see: http://enwp.org/zstd http://enwp.org/zstd parallel-friendly, trades off compression level for speed