4 ms·

And the same for LZMA: https://github.com/vasi/pixz https://github.com/vasi/pixz (it's relatively easy to remember those commands)

by tobias3 10y ago

stouset 10y ago
Doesn't xz already support threads out of the box, by the --threads flag?
- mnordhoff 10y ago
  The first stable version of xz with threading was 5.2.0, released in December 2014. A lot of people are running older packages, or were until recently, and habitually use pixz instead.
  - _delirium 10y ago
    Current versions of Debian and Ubuntu ship 5.1.1 fwiw, so it's still quite a large number of people. In fact 5.2.0 doesn't even seem to be in Debian unstable yet. Not 100% sure why, but browsing through the wishlist bug open for years [1], it seems to be due to an unfortunately common reason: even widely used open-source software sometimes has surprisingly few maintainers, in this case seemingly one person, who was maintaining the xz package for years but got busy with other things, and nobody else has picked it up. The good-ish news is that the 5.1.x version now seems to be growing old enough that it's starting to block other packages' upgrades (because their upstream now assumes a newer version), which will probably cause enough other Debian maintainers to notice for the situation to be sorted out. Though I did of course use passive voice in that last sentence, as I wait for "the situation to be sorted out". [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634
0x0 10y ago
The main purpose of this "pixz" appears to be its chunking of the compressed data so that is it partially decompressible (i.e. random access). "xz" has -T/--threads= already for multithreaded processing (although it does seem like pixz has a different default value "all cores" instead of "1 thread") .
- vasi 10y ago
  xz in multithreaded mode supports random access too, at least theoretically. But there's no reasonable way with xz to actually find the file in a tarball you want to access, it's that bit that pixz provides. Another nice thing about pixz is it does parallel decompression, as well as compression. (Disclaimer: I'm the original author of pixz.)
  - mgerdts 10y ago
    I was thinking about that "no reasonable way" comment. When you uncompress the first block, you will find the first tar header. From that you can know the uncompressed offset of the next tar header. If the compressed stream does support random access, you should be able to uncompress a block (assuming uncompressed block size was a multitple of 512 bytes) to get to the next tar header. You can repeat this until you get to the file you are looking for. With large files, this approach would be of huge value. If the files tend to be no larger than block_size - 512, there will be no speedup. Of course, this would need to be implemented directly in tar, not by piping the output of a decompression command through tar.
- mgerdts 10y ago
  The --index option that I added (see another comment) to pigz allows random access so that uncompression could be multi-threaded. The workload I was interested in (compressing virtual machine memory during suspend) tends to be I/O bound. pigz struck a good balance to make it so that cpus and disks both stayed busy in an important subset of the machines of interest. This balanced saturation seemed to be optimal to minimize the suspend time. If the suspend image was on fast storage (that is, could saturate a 10 gig link) multi-threading uncompression for resume made a big difference. I tried pixz and I think pbzip2 and found that while the compress ratio was better than pigz offers, the delta in CPU-bound run time during compression was unacceptable. The scales may tip in favor of pixz when compression speed is less important than (e.g.) minimizing the amount of bandwidth required for 1000's of downloads of the compressed content.
mikepurvis 10y ago
Upstream xz supports it too, but not until a relatively recent release. If you're interested, I have a PPA for Ubuntu Trusty which includes up-to-date xz-utils + dpkg-deb set up to use it. This is a lifesaver if you deploy large bundles of software in the deb format: https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg

And the same for LZMA: https://github.com/vasi/pixz https://github.com/vasi/pixz (it's relatively easy to remember those commands)

by tobias3 10y ago

And the same for LZMA: https://github.com/vasi/pixz https://github.com/vasi/pixz (it's relatively easy to remember those commands)

stouset 10y ago

Doesn't xz already support threads out of the box, by the --threads flag?

mnordhoff 10y ago
The first stable version of xz with threading was 5.2.0, released in December 2014. A lot of people are running older packages, or were until recently, and habitually use pixz instead.
- _delirium 10y ago
  Current versions of Debian and Ubuntu ship 5.1.1 fwiw, so it's still quite a large number of people. In fact 5.2.0 doesn't even seem to be in Debian unstable yet. Not 100% sure why, but browsing through the wishlist bug open for years [1], it seems to be due to an unfortunately common reason: even widely used open-source software sometimes has surprisingly few maintainers, in this case seemingly one person, who was maintaining the xz package for years but got busy with other things, and nobody else has picked it up. The good-ish news is that the 5.1.x version now seems to be growing old enough that it's starting to block other packages' upgrades (because their upstream now assumes a newer version), which will probably cause enough other Debian maintainers to notice for the situation to be sorted out. Though I did of course use passive voice in that last sentence, as I wait for "the situation to be sorted out". [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634

0x0 10y ago

The main purpose of this "pixz" appears to be its chunking of the compressed data so that is it partially decompressible (i.e. random access). "xz" has -T/--threads= already for multithreaded processing (although it does seem like pixz has a different default value "all cores" instead of "1 thread") .

vasi 10y ago
xz in multithreaded mode supports random access too, at least theoretically. But there's no reasonable way with xz to actually find the file in a tarball you want to access, it's that bit that pixz provides. Another nice thing about pixz is it does parallel decompression, as well as compression. (Disclaimer: I'm the original author of pixz.)
- mgerdts 10y ago
  I was thinking about that "no reasonable way" comment. When you uncompress the first block, you will find the first tar header. From that you can know the uncompressed offset of the next tar header. If the compressed stream does support random access, you should be able to uncompress a block (assuming uncompressed block size was a multitple of 512 bytes) to get to the next tar header. You can repeat this until you get to the file you are looking for. With large files, this approach would be of huge value. If the files tend to be no larger than block_size - 512, there will be no speedup. Of course, this would need to be implemented directly in tar, not by piping the output of a decompression command through tar.
mgerdts 10y ago
The --index option that I added (see another comment) to pigz allows random access so that uncompression could be multi-threaded. The workload I was interested in (compressing virtual machine memory during suspend) tends to be I/O bound. pigz struck a good balance to make it so that cpus and disks both stayed busy in an important subset of the machines of interest. This balanced saturation seemed to be optimal to minimize the suspend time. If the suspend image was on fast storage (that is, could saturate a 10 gig link) multi-threading uncompression for resume made a big difference. I tried pixz and I think pbzip2 and found that while the compress ratio was better than pigz offers, the delta in CPU-bound run time during compression was unacceptable. The scales may tip in favor of pixz when compression speed is less important than (e.g.) minimizing the amount of bandwidth required for 1000's of downloads of the compressed content.

mikepurvis 10y ago

Upstream xz supports it too, but not until a relatively recent release. If you're interested, I have a PPA for Ubuntu Trusty which includes up-to-date xz-utils + dpkg-deb set up to use it. This is a lifesaver if you deploy large bundles of software in the deb format: https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg