3 ms·

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/components/pigz/patc

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/components/pigz/patches/202_index.patch https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.

gpderetta 3y ago
Nice! You should be able to do it without an index by periodically restarting the dictionary on compression and then looking for something resembling the dictionary, right?
- mgerdts 3y ago
  Yeah, probably so at the cost of compatibility. As implemented, the .gz file can be used with `gzip -d`.
gcflymoto 3y ago
Would you mind creating a fork of pigz in GitHub and add this patch? I would be interested in testing it out!
- mgerdts 3y ago
  You can grab the version from the solaris userland repo I linked and use it without me completing a homework assignment. Just grab the pigz-2.3.4 source then apply the patches from [1] in the proper order. Maybe some of them aren't needed for non-Solaris. 1. https://github.com/oracle/solaris-userland/tree/master/components/pigz/patches https://github.com/oracle/solaris-userland/tree/master/compo... I thought I had opened a PR for that a long while ago, but it doesn't show up on github these days. In any case, I did ask Mark Adler to review it. It was never a priority, then the code changed in ways that I don't really want to deal with. While looking through the PRs, I noticed a PR for Blocked GZip Format (BGZF) [2]. That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in. 2. https://github.com/madler/pigz/pull/19 https://github.com/madler/pigz/pull/19

gpderetta 3y ago

Nice! You should be able to do it without an index by periodically restarting the dictionary on compression and then looking for something resembling the dictionary, right?

mgerdts 3y ago
Yeah, probably so at the cost of compatibility. As implemented, the .gz file can be used with `gzip -d`.

gcflymoto 3y ago

Would you mind creating a fork of pigz in GitHub and add this patch? I would be interested in testing it out!

mgerdts 3y ago
You can grab the version from the solaris userland repo I linked and use it without me completing a homework assignment. Just grab the pigz-2.3.4 source then apply the patches from [1] in the proper order. Maybe some of them aren't needed for non-Solaris. 1. https://github.com/oracle/solaris-userland/tree/master/components/pigz/patches https://github.com/oracle/solaris-userland/tree/master/compo... I thought I had opened a PR for that a long while ago, but it doesn't show up on github these days. In any case, I did ask Mark Adler to review it. It was never a priority, then the code changed in ways that I don't really want to deal with. While looking through the PRs, I noticed a PR for Blocked GZip Format (BGZF) [2]. That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in. 2. https://github.com/madler/pigz/pull/19 https://github.com/madler/pigz/pull/19