3 ms·

really? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than t

really? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than the others we tested.

thedance 7y ago
Well yah, those are all ratio-tuned codecs that are very slow. LZMA beats bzip2 on both ratio and speed so you might as well forget about bzip2 forever. Zstd, snappy, LZ4, or even brotli are probably better choices than zlib for most people. Brotli has LZMA-like ratios at dramatically higher speeds.
blattimwind 7y ago
The algorithms you name are all rather outdated. Typical gzip decompression speed is somewhere in the 200-250 MB/s region, compression is much slower. LZ4 for example tends to compress at ~600-700 MB/s, and decompress at several GB/s. zstd is tweakable over a very wide range of ratio-speed trade-offs. LZMA(2) (xz) is a rather troubled format and should not be used any more. bzip2 has always been slower than gzip with usually marginally better compression. It has been irrelevant for a long time.
- oefrha 7y ago
  > LZMA(2) (xv) is a rather troubled format and should not be used any more. You mean xz. Not sure where you got the idea that it’s a troubled format, but if you’re talking about the infamous “Xz format inadequate for long-term archiving”, IMO that’s just bzip2 authors taking a dump on xz for no good reason, and fortunately for us it’s bzip2 that’s basically irrelevant today, not xz.
- gnulinux 7y ago
  > It has been irrelevant for a long time. Funny you say that, my company uses bz2 for compressing pretty much everything.
  - blattimwind 7y ago
    And many companies use fixed-width record formats for data exchange... what's your point exactly?
    - gnulinux 7y ago
      I guess that it's not irrelevant. Deprecated, old, etc sure, but irrelevant?
deleted 7y ago
[deleted]
ants_a 7y ago
https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ Brotli is better, zstd is faster, lz4 is often good enough and a lot faster.

really? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than the others we tested.

thedance 7y ago

Well yah, those are all ratio-tuned codecs that are very slow. LZMA beats bzip2 on both ratio and speed so you might as well forget about bzip2 forever. Zstd, snappy, LZ4, or even brotli are probably better choices than zlib for most people. Brotli has LZMA-like ratios at dramatically higher speeds.

blattimwind 7y ago

The algorithms you name are all rather outdated. Typical gzip decompression speed is somewhere in the 200-250 MB/s region, compression is much slower. LZ4 for example tends to compress at ~600-700 MB/s, and decompress at several GB/s. zstd is tweakable over a very wide range of ratio-speed trade-offs. LZMA(2) (xz) is a rather troubled format and should not be used any more. bzip2 has always been slower than gzip with usually marginally better compression. It has been irrelevant for a long time.

oefrha 7y ago
> LZMA(2) (xv) is a rather troubled format and should not be used any more. You mean xz. Not sure where you got the idea that it’s a troubled format, but if you’re talking about the infamous “Xz format inadequate for long-term archiving”, IMO that’s just bzip2 authors taking a dump on xz for no good reason, and fortunately for us it’s bzip2 that’s basically irrelevant today, not xz.
gnulinux 7y ago
> It has been irrelevant for a long time. Funny you say that, my company uses bz2 for compressing pretty much everything.
- blattimwind 7y ago
  And many companies use fixed-width record formats for data exchange... what's your point exactly?
  - gnulinux 7y ago
    I guess that it's not irrelevant. Deprecated, old, etc sure, but irrelevant?

deleted 7y ago

[deleted]

ants_a 7y ago

https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ Brotli is better, zstd is faster, lz4 is often good enough and a lot faster.