3 ms·

Looks great! I’m still surprised these tools (gzip, etc) haven’t been update to take advantage of multicore support.

• LeoPanthera 6y ago
  It's good that they haven't. If you're compressing a large number of individual files (rather than combining them into a single archive), it's still more efficient to run multiple copies of gzip or whatever via something like "parallel".
  • pdimitar 6y ago
    Good point. I don't think anything bad would happen if you could opt in via something like `-j8`, preserving the default backwards-compatible behaviour and offering an escape hatch for newer workloads / machines.
• bennofs 6y ago
  In the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decompress the next byte. So parallizing is not really possible, without changing the format. Newer compression formats have multiple streams that can be decompressed in parallel, or have blocks that are compressed independently without dependencies (bzip2 can be decompressed in parallel due to independent blocks).
  • ed25519FUUU 6y ago
    So is Pigz not backward compatible with gzip?
    • trav4225 6y ago
      I don't think Pigz decompresses in parallel.
    • goodside 6y ago
      Pigz is backward compatible — it breaks the input into 128K chunks, compresses each in parallel, and concatenates them. Gzip supports naive concatenation of compressed files so the result can be decompressed by standard gzip. Pigz does not implement parallel decompression, but pigz-compressed files could, in principle, be decompressed in parallel. I assume there's not much urgency for this because gzip decompression is fast enough: >3x faster than compression and >10x faster than bzip2 decompression on default settings. (See: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ )