3 ms·
Is there a way to decompress in parallel? Afaik pigz can use a few threads to separate decompressing and i/o -- is there anything can we can pass to tar to ungz
by kardos 4y ago
Is there a way to decompress in parallel? Afaik pigz can use a few threads to separate decompressing and i/o -- is there anything can we can pass to tar to ungzip faster?
- jandrese 4y agoNot without changing the structure of the gzip unfortunately.
- fwip 4y agoGzip files are allowed to consist of multiple streams. Bgzip (block gzip) is one tool that reads and writes gzip-compatible files in parallel. The output can be decompressed in parallel by bgzip, or semi-random access, or decompressed normally by gzip.
- LeoPanthera 4y agoLzip can be decompressed in parallel if compressed with "tarlz". https://www.nongnu.org/lzip/tarlz.html https://www.nongnu.org/lzip/tarlz.html
- usefulcat 4y agopzstd can decompress in parallel if the data was compressed using pzstd. Although zstd is so much faster for decompression that you rarely need multiple threads. For xz, pixz can decompress in parallel.
- mxmlnkn 4y agoI have been working on exactly that: pragzip. It decompresses in parallel in a similar manner to the prototype pugz by implementing a two-staged decompression. https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip I also did a Show HN: https://news.ycombinator.com/item?id=32366959 https://news.ycombinator.com/item?id=32366959