5 ms·
Parallel Gzip – Pigz
- LeoPanthera 6y agoSee also: pbzip2: https://github.com/ruanhuabin/pbzip2 https://github.com/ruanhuabin/pbzip2 plzip: https://www.nongnu.org/lzip/plzip.html https://www.nongnu.org/lzip/plzip.html And possibly the cleverest of the parallel compressors, tarlz, which, while still being backwards compatible with tar, also allows parallel decompression, something none of the others can do: https://www.nongnu.org/lzip/tarlz.html https://www.nongnu.org/lzip/tarlz.html
- Yajirobe 6y agoThe source code looks a little extreme... Is it auto-generated? https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#L3367 https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#...
- simcop2387 6y agoI doubt it, it does look uglier than modern C but it looks like it's written to work in ANSI-C/C89 entirely without any language extensions since then. That's going to shape things quite a bit (in particular the dense bit of variable declarations at the top of every function) simply because a lot of the things like being able to declare variables later on in the source code just wasn't a thing in that version of the language.
- trav4225 6y agoSeems pretty standard to me. (Although I also typically only write C in C89, so modern C might be a lot prettier.)
- ak217 6y agoAlso https://github.com/vasi/pixz https://github.com/vasi/pixz, which uses a more computationally efficient yet widely supported compression algorithm (lzma/xz) and adds indexing, making byte range access possible in large files within the archive (and does also support parallel decompression).
- goodside 6y agoZstandard also has experimental support for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395
- goodside 6y agoNote that pbzip2 can only parallel-decompress files that are compressed with pbzip2. lbzip2 (https://lbzip2.org/ https://lbzip2.org/) is capable of decompressing arbitrary bz2 files in parallel, which is a more common need IMHO. bz2 is very CPU- and RAM-intensive, but gives the best ratios of any mainstream algorithm, so it's really only useful for publicly distributed files where bandwidth considerations dominate.
- chrchang523 6y agoThe bgzip format (widely used in bioinformatics) supports parallel decompression, while being readable by gzip. (Compression ratio isn’t the best, though.)
- ed25519FUUU 6y agoLooks great! I’m still surprised these tools (gzip, etc) haven’t been update to take advantage of multicore support.
- LeoPanthera 6y agoIt's good that they haven't. If you're compressing a large number of individual files (rather than combining them into a single archive), it's still more efficient to run multiple copies of gzip or whatever via something like "parallel".
- pdimitar 6y agoGood point. I don't think anything bad would happen if you could opt in via something like `-j8`, preserving the default backwards-compatible behaviour and offering an escape hatch for newer workloads / machines.
- bennofs 6y agoIn the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decompress the next byte. So parallizing is not really possible, without changing the format. Newer compression formats have multiple streams that can be decompressed in parallel, or have blocks that are compressed independently without dependencies (bzip2 can be decompressed in parallel due to independent blocks).
- ed25519FUUU 6y agoSo is Pigz not backward compatible with gzip?
- goodside 6y agoThis is a low-effort post, showing basic CLI usage and nothing else. Pigz does not actually decompress in parallel, as the author implies, underscoring the importance of benchmarking things — especially when you're only considering them for performance reasons. See: https://github.com/madler/pigz/issues/36 https://github.com/madler/pigz/issues/36 For compression, pigz is only useful for compressing a single large file in a scenario where you must use gzip. For large collections of files, wrapping normal gzip with GNU Parallel would be faster. In scenarios where you control both compression and decompression, you'd be better off picking a more modern compression algorithm than gzip. ZStandard is almost always the right choice, but this benchmark is a great resource for comparing other options: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/
- skrause 6y agoYou basically only need to learn about "zstd -T0". Better and faster algorithm and multi-threading directly included.
- dispat0r 6y agosadly also doesn't work for decompression.
- goodside 6y agoEverybody's needs are different, but zstd is so fast the lack of multithreaded decompression doesn't matter very often. I've only ever found parallel decompression useful with bz2 files (which are insanely CPU-intensive to decompress), using lbzip2. Zstd also has experimental code in the `contrib` dir for parallel (and, more importantly, seekable) decompression: https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395
- dispat0r 6y agoThanks for the Information. You're right zstd even on a single core is pretty fast but for multicore systems with nvme drives multithreading would be beneficial.
- deleted 6y ago[deleted]