4 ms·
The real trick is to use a parallelized implementation of the compression tool, viz: tar --use-compress-program /usr/bin/lbzip2 --create --file file.tar.bz
by gjvc 4y ago
The real trick is to use a parallelized implementation of the compression tool, viz:
tar --use-compress-program /usr/bin/lbzip2 --create --file file.tar.bz2
or if you must use .gz
tar --use-compress-program /usr/bin/pigz --create --file file.tar.gz
- sidewndr46 4y agoWow, that's really neat. I have never heard of pigz before despite being an avid Ubuntu user for a long time. Also, I see it supports the "-11" level which is using the zopfli algorithm! Appreciate you sharing.
- digitallyfree 4y agoAs a FYI pigz also can speed up decompression as well, but the performance boost is minimal. The decompression is still in a single thread but it spawns other threads for i/o and so forth. For compression the speedup is quite linear.
- optionalsquid 4y agoigzip (https://github.com/intel/isa-l https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files. However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.
- lifthrasiir 4y ago> However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. [EDIT: I misread a benchmark result [1] and wrongly concluded that igzip -3 is better than gzip -9, in reality it is somewhere in the middle of -1 and -9.] [1] https://encode.su/threads/3398-Sequence-Compression-Benchmark?p=66191&viewfull=1#post66191 https://encode.su/threads/3398-Sequence-Compression-Benchmar...
- optionalsquid 4y agoThat benchmark seems to be comparing igzip with gzip -1, since the reported gzip output size is 42298786 bytes and not 36548933 bytes (from the earlier benchmarks). My own experience based mostly on FASTQ/text data is that igzip -3 gives results very similar to gzip -3.
- lifthrasiir 4y agoOops! Yes, you are correct. I confused it with an other reply with level 9 benchmarks, please disregard the parent comment.
- kardos 4y agoIs there a way to decompress in parallel? Afaik pigz can use a few threads to separate decompressing and i/o -- is there anything can we can pass to tar to ungzip faster?
- jandrese 4y agoNot without changing the structure of the gzip unfortunately.
- fwip 4y agoGzip files are allowed to consist of multiple streams. Bgzip (block gzip) is one tool that reads and writes gzip-compatible files in parallel. The output can be decompressed in parallel by bgzip, or semi-random access, or decompressed normally by gzip.
- LeoPanthera 4y agoLzip can be decompressed in parallel if compressed with "tarlz". https://www.nongnu.org/lzip/tarlz.html https://www.nongnu.org/lzip/tarlz.html
- usefulcat 4y agopzstd can decompress in parallel if the data was compressed using pzstd. Although zstd is so much faster for decompression that you rarely need multiple threads. For xz, pixz can decompress in parallel.
- mxmlnkn 4y agoI have been working on exactly that: pragzip. It decompresses in parallel in a similar manner to the prototype pugz by implementing a two-staged decompression. https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip I also did a Show HN: https://news.ycombinator.com/item?id=32366959 https://news.ycombinator.com/item?id=32366959
- biglost 4y agolrzip