3 ms·

See also: pbzip2: https://github.com/ruanhuabin/pbzip2 https://github.com/ruanhuabin/pbzip2 plzip: https://www.nongnu.org/lzip/plzip.html https://www.nongnu.o

See also: pbzip2: https://github.com/ruanhuabin/pbzip2 https://github.com/ruanhuabin/pbzip2 plzip: https://www.nongnu.org/lzip/plzip.html https://www.nongnu.org/lzip/plzip.html And possibly the cleverest of the parallel compressors, tarlz, which, while still being backwards compatible with tar, also allows parallel decompression, something none of the others can do: https://www.nongnu.org/lzip/tarlz.html https://www.nongnu.org/lzip/tarlz.html

Yajirobe 6y ago
The source code looks a little extreme... Is it auto-generated? https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#L3367 https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#...
- simcop2387 6y ago
  I doubt it, it does look uglier than modern C but it looks like it's written to work in ANSI-C/C89 entirely without any language extensions since then. That's going to shape things quite a bit (in particular the dense bit of variable declarations at the top of every function) simply because a lot of the things like being able to declare variables later on in the source code just wasn't a thing in that version of the language.
- trav4225 6y ago
  Seems pretty standard to me. (Although I also typically only write C in C89, so modern C might be a lot prettier.)
  - Yajirobe 6y ago
    Where are the 20 line functions, the modularity, the refactored nested if's?
    - trav4225 6y ago
      Like I said, pretty standard! ;-)
ak217 6y ago
Also https://github.com/vasi/pixz https://github.com/vasi/pixz, which uses a more computationally efficient yet widely supported compression algorithm (lzma/xz) and adds indexing, making byte range access possible in large files within the archive (and does also support parallel decompression).
- goodside 6y ago
  Zstandard also has experimental support for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395
goodside 6y ago
Note that pbzip2 can only parallel-decompress files that are compressed with pbzip2. lbzip2 (https://lbzip2.org/ https://lbzip2.org/) is capable of decompressing arbitrary bz2 files in parallel, which is a more common need IMHO. bz2 is very CPU- and RAM-intensive, but gives the best ratios of any mainstream algorithm, so it's really only useful for publicly distributed files where bandwidth considerations dominate.
chrchang523 6y ago
The bgzip format (widely used in bioinformatics) supports parallel decompression, while being readable by gzip. (Compression ratio isn’t the best, though.)

Yajirobe 6y ago

The source code looks a little extreme... Is it auto-generated? https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#L3367 https://github.com/ruanhuabin/pbzip2/blob/master/pbzip2.cpp#...

simcop2387 6y ago
I doubt it, it does look uglier than modern C but it looks like it's written to work in ANSI-C/C89 entirely without any language extensions since then. That's going to shape things quite a bit (in particular the dense bit of variable declarations at the top of every function) simply because a lot of the things like being able to declare variables later on in the source code just wasn't a thing in that version of the language.
trav4225 6y ago
Seems pretty standard to me. (Although I also typically only write C in C89, so modern C might be a lot prettier.)
- Yajirobe 6y ago
  Where are the 20 line functions, the modularity, the refactored nested if's?
  - trav4225 6y ago
    Like I said, pretty standard! ;-)

ak217 6y ago

Also https://github.com/vasi/pixz https://github.com/vasi/pixz, which uses a more computationally efficient yet widely supported compression algorithm (lzma/xz) and adds indexing, making byte range access possible in large files within the archive (and does also support parallel decompression).

goodside 6y ago
Zstandard also has experimental support for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395

goodside 6y ago

Note that pbzip2 can only parallel-decompress files that are compressed with pbzip2. lbzip2 (https://lbzip2.org/ https://lbzip2.org/) is capable of decompressing arbitrary bz2 files in parallel, which is a more common need IMHO. bz2 is very CPU- and RAM-intensive, but gives the best ratios of any mainstream algorithm, so it's really only useful for publicly distributed files where bandwidth considerations dominate.

chrchang523 6y ago

The bgzip format (widely used in bioinformatics) supports parallel decompression, while being readable by gzip. (Compression ratio isn’t the best, though.)