5 ms·

Part of my infrastructure uses compressed "dd" disc images. Last week I switched from using "xz" to "zstd" for them and "zstd -19" compresses almost as well as

Part of my infrastructure uses compressed "dd" disc images. Last week I switched from using "xz" to "zstd" for them and "zstd -19" compresses almost as well as xz (1.8GB vs 1.7GB), but the threaded decompression is >10x faster (72s -> 6s). I made this change coincidentally, the day before the xz compromise.

cogman10 3y ago
I've been nothing but happy with ZSTD. Really good compression trade offs with ratios close to the best of LZMA. ZSTD 3 is nearly as fast as LZ4 while getting a much better ratio. Consiquentially, on my linux boxes I use either ZFS or btrfs and use zstd 3 as the default for the file system.
- dralley 3y ago
  Nearly as fast to compress, but much slower to decompress. Zstd is pretty good on decompression but LZ4 is top-tier. Zstd is often "fast enough" in decompression for most cases, though.
  - chronogram 3y ago
    It's why I use LZ4 for ZRAM. Only at RAM speeds do LZ4's speeds matter.
    - everybackdoor 3y ago
      [dead]
wood_spirit 3y ago
(JiaT75 contributed to zstd as well as xz)
- cipherboy 3y ago
  Citation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75 https://github.com/facebook/zstd/commits/dev/?author=jiaT75
  - wood_spirit 3y ago
    https://news.ycombinator.com/item?id=39873017 https://news.ycombinator.com/item?id=39873017
    - cipherboy 3y ago
      That looks like a(n unofficial) fork, not contributions to the official upstream repository.
- geggo98 3y ago
  I think there were some open PRs from that account, that got scrubbed from the account after the back door in xz was discovered. But probably nothing got merged.
  - ikekkdcjkfke 3y ago
    Is it possible to retrieve the old branches?
linsomniac 3y ago
Additional note: I run these on servers that are not very tall but are pretty wide: 40 1.8GHz cores or so. So multi-threaded decompression really helps. I was using xz before, which doesn't support threaded decompression (according tom man page), decided to switch to zstd rather than pixz.
1vuio0pswjnm7 3y ago
zstd (tries to) link to liblzma by default. https://raw.githubusercontent.com/facebook/zstd/dev/CHANGELOG https://raw.githubusercontent.com/facebook/zstd/dev/CHANGELO... See v1.3.0 I compile it statically without xz/lzma support

cogman10 3y ago

I've been nothing but happy with ZSTD. Really good compression trade offs with ratios close to the best of LZMA. ZSTD 3 is nearly as fast as LZ4 while getting a much better ratio. Consiquentially, on my linux boxes I use either ZFS or btrfs and use zstd 3 as the default for the file system.

dralley 3y ago
Nearly as fast to compress, but much slower to decompress. Zstd is pretty good on decompression but LZ4 is top-tier. Zstd is often "fast enough" in decompression for most cases, though.
- chronogram 3y ago
  It's why I use LZ4 for ZRAM. Only at RAM speeds do LZ4's speeds matter.
  - everybackdoor 3y ago
    [dead]

wood_spirit 3y ago

(JiaT75 contributed to zstd as well as xz)

cipherboy 3y ago
Citation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75 https://github.com/facebook/zstd/commits/dev/?author=jiaT75
- wood_spirit 3y ago
  https://news.ycombinator.com/item?id=39873017 https://news.ycombinator.com/item?id=39873017
  - cipherboy 3y ago
    That looks like a(n unofficial) fork, not contributions to the official upstream repository.
geggo98 3y ago
I think there were some open PRs from that account, that got scrubbed from the account after the back door in xz was discovered. But probably nothing got merged.
- ikekkdcjkfke 3y ago
  Is it possible to retrieve the old branches?

linsomniac 3y ago

Additional note: I run these on servers that are not very tall but are pretty wide: 40 1.8GHz cores or so. So multi-threaded decompression really helps. I was using xz before, which doesn't support threaded decompression (according tom man page), decided to switch to zstd rather than pixz.

1vuio0pswjnm7 3y ago

zstd (tries to) link to liblzma by default. https://raw.githubusercontent.com/facebook/zstd/dev/CHANGELOG https://raw.githubusercontent.com/facebook/zstd/dev/CHANGELO... See v1.3.0 I compile it statically without xz/lzma support