3 ms·

lrzip is a preprocessor that finds matches in the distant past that the backend compressor (xz) couldn't normally find. zstd has a new long range matcher mode i

lrzip is a preprocessor that finds matches in the distant past that the backend compressor (xz) couldn't normally find. zstd has a new long range matcher mode inspired by the ideas behind rzip/lrzip with some extra tricks. It produces data in the standard zstd format, so can be decompressed the the normal zstd decompressor. There is a short article about it in the latest release notes https://github.com/facebook/zstd/releases/tag/v1.3.2 https://github.com/facebook/zstd/releases/tag/v1.3.2

Twirrim 9y ago
I tried a few times with zstd at various levels of compression with the linux kernel sources. While I've been impressed with zstd, and have some projects lined up to use it, it seems in the case of the linux kernel sources, it's not a great fit. xz handily beats it, and not by a small margin either. I had to really ratchet up the compression levels (20+) before I could get close to 100Mb.
- Iburinoc 9y ago
  In general, xz beats zstd in compression ratio, as xz is very committed to providing the strongest compression, at the expense of speed, while zstd provides a range of compression ratio vs speed tradeoffs [0]. At the lower levels, zstd isn't approaching xz's compression level, but it's doing it much much faster. Additionally, zstd generally massively outperforms xz in decompression speed $ time xz linux-4.14-rc6.tar real 4m26.009s user 4m24.828s sys 0m0.724s $ wc -c linux-4.14-rc6.tar.xz 103705148 linux-4.14-rc6.tar.xz $ time zstd --ultra -20 linux-4.14-rc6.tar linux-4.14-rc6.tar : 12.81% (824350720 => 105564246 bytes, linux-4.14-rc6.tar.zst) real 4m34.129s user 4m33.484s sys 0m0.432s $ time cat linux-4.14-rc6.tar.xz | xz -d > out1 real 0m9.677s user 0m6.608s sys 0m0.704s $ time cat linux-4.14-rc6.tar.zst | zstd -d > out2 real 0m1.702s user 0m1.220s sys 0m0.520s [0]: https://github.com/facebook/zstd/blob/dev/doc/images/DCspeed5.png https://github.com/facebook/zstd/blob/dev/doc/images/DCspeed...

Twirrim 9y ago

I tried a few times with zstd at various levels of compression with the linux kernel sources. While I've been impressed with zstd, and have some projects lined up to use it, it seems in the case of the linux kernel sources, it's not a great fit. xz handily beats it, and not by a small margin either. I had to really ratchet up the compression levels (20+) before I could get close to 100Mb.

Iburinoc 9y ago
In general, xz beats zstd in compression ratio, as xz is very committed to providing the strongest compression, at the expense of speed, while zstd provides a range of compression ratio vs speed tradeoffs [0]. At the lower levels, zstd isn't approaching xz's compression level, but it's doing it much much faster. Additionally, zstd generally massively outperforms xz in decompression speed $ time xz linux-4.14-rc6.tar real 4m26.009s user 4m24.828s sys 0m0.724s $ wc -c linux-4.14-rc6.tar.xz 103705148 linux-4.14-rc6.tar.xz $ time zstd --ultra -20 linux-4.14-rc6.tar linux-4.14-rc6.tar : 12.81% (824350720 => 105564246 bytes, linux-4.14-rc6.tar.zst) real 4m34.129s user 4m33.484s sys 0m0.432s $ time cat linux-4.14-rc6.tar.xz | xz -d > out1 real 0m9.677s user 0m6.608s sys 0m0.704s $ time cat linux-4.14-rc6.tar.zst | zstd -d > out2 real 0m1.702s user 0m1.220s sys 0m0.520s [0]: https://github.com/facebook/zstd/blob/dev/doc/images/DCspeed5.png https://github.com/facebook/zstd/blob/dev/doc/images/DCspeed...