7 ms·

I have been incredibly impressed with Zstandard, and if you are not currently using I suggest giving it a try. I'm a sysadmin so this has worked it's way into

I have been incredibly impressed with Zstandard, and if you are not currently using I suggest giving it a try. I'm a sysadmin so this has worked it's way into every corner of my servers, from backups, to log compression, and more. Most of my servers run FreeBSD, and with the merge of linux/FreeBSD ZFS I am looking forward to the integration of ZSTD into ZFS so I can use that for on-the-fly Zpool compression. I can safely say that ZSTD has saved me TB of space over LZ4 & GZIP.

kzrdude 6y ago
What kind of compression level do you use?
- dmw_ng 6y ago
  You probably have some conditioning that discourages use of '-9', but that's generally totally fine to use with zstd in terms of speed. I've been a zstd convert for a few years now and haven't settled on any particular ideal setting. Sometimes the very high compression levels (-19) produce a worse result, sometimes they kill it. I think most of the time I'm using -9. I've seen cases where '--ultra --long -22' was bringing size down by 10% or more compared to the normal settings, and also cases where changing the defaults at all produced significantly lower compression.
  - loeg 6y ago
    For approximate reference (N.B., this graph is somewhat dated — zstd 1.3.4 is from March 2018): https://raw.githubusercontent.com/facebook/zstd/v1.3.4/doc/images/CSpeed2.png https://raw.githubusercontent.com/facebook/zstd/v1.3.4/doc/i... (On the dataset used for the figure, zstd -9 was about the same MB/s as gzip -4, and compressed at ~3.5x to gzip's ~3.0x.)
alecco 6y ago
Compressing large files/backups using all available cores at the highest setting is a bliss. Sadly the choke point now is encryption. Hopefully some kind soul will some day add this to gpg or equivalent. There are parallelizable cipher modes like CTR and GCM. https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
- Algent 6y ago
  If your need of encryption is related to backups, look into "Borg backup". It can handle both encryption and zstd, also do dedupe natively since it's storing data as blocks.
  - formerly_proven 6y ago
    Borg only utilizes one thread and does not support multithreaded zstd, so the write throughput will likely be (much) lower than ZFS -- typically 60-120 MB/s. Since ZFS can also dedupe, Borg might not even be faster for heavily deduped backups.
    - doublepg23 6y ago
      ZFS dedupe is broken and I don’t recommend using it unless you really know what you’re getting into.
      - formerly_proven 6y ago
        Broken in what way?
        tatersolid 6y ago
        It uses a ludicrous amount of RAM and a fixed block size and offset boundry, making it very slow and fairly ineffective.
wongarsu 6y ago
I used to be a big fan of XZ(which uses lzma2), but Zstandard has good-enough compression at high levels with orders of magnitude faster decompression. It's also more versatile in its compression levels. It's now also my go-to for new projects.
- formerly_proven 6y ago
  XZ/LZMA2 has the problem that it slows down a lot on incompressible data and can even choke pretty badly on compressible data, which Yann Collet's algorithms (LZ4 and ze standard) don't do nearly as much. This means zstd can be used indiscriminately, while LZMA would frequently need some sort of qualifier to avoid processing incompressible data and the associated loss of performance. To put some numbers to this, on the machine (>10yo laptop) I'm typing this on: trivial random .wav file LZMA ~20 MB/s ~2 MB/s ~1 MB/s zstd ~500 MB/s ~250 MB/s ~70 MB/s zlib ~40 MB/s ~13 MB/s ~6 MB/s
- saurik 6y ago
  (xz is a compression container format that doesn't imply any particular compression algorithm. It has a concept of "filters" that are applied in "chains" over "blocks". One filter happens to be "lzma2", but to say xz "uses lzma2" is a bit of a confusing mental model, as xz could just as easily "use zstd" if someone allocated a filter ID for such.)
  - wongarsu 6y ago
    In principle yes, but in practice the only filters included in my version are lzma1, lzma2, a basic delta filter and a few filters for executable code. As far as I can tell all the presets that are reachable without explicitly specifying a filter chain are just lzma2 with various options set. I would love if xz had a zstd filter and a filter that adds/processes some error correction code. But with what's there right now I think it's fair to treat it as a pure lzma2 container format.
walrus01 6y ago
For system backups and logs and stuff that rarely need to be decompressed, but stored for a while and then deleted in rotation, I still strongly prefer xzip. It's much slower to create a .tar.xz file with the default settings, but also saves a considerable amount of disk space. Side by side comparison right now between a full system file tree backup of a mediawiki server, the .tar.xz file is about 1.8GB, the .tar.zstd file is 2.15GB. The same thing in traditional .tar.gz format is obviously even larger. From the research I've done on common uses of zstd it's much better suited to applications where very high speed of compression and decompression is a major requirement, but absolute smallest file size is not. Typically as a replacement for gzip.
- petre 6y ago
  Try xz -1. Faster than gzip, smaller size. We use it for regular backups. Although the xz format is not recommended for long term archiving because it's unsafe. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
  - walrus01 6y ago
    I'm going to read this in more detail later today when I have time. But I just skimmed it, and most of the points about xz being fragile and having a weird format would seem to apply in a scenario in which a .tar.xz file became corrupted or bit-flipped from its original? eg: in which the .xz file, after some long period of archiving, no longer matched the same sha256sum as when it was originally created. Thereby making it difficult or a huge hassle to get authentic lossless/uncorrupted data out of it?
  - loeg 6y ago
    The "unsafe" thing is nonsense. The expressed concern is how the contents degrade in the presence of arbitrary bit flips. Any highly-compressed format is going to be susceptible to corruption amplification from random bitflips, and the higher compression level, the larger fallout. Ensure your compressed archives are not corrupted using some external FEC; either in your hardware, or something like par2.
- 6y ago
deleted 6y ago
[deleted]
baybal2 6y ago
I wonder if software compression can now cost compete with compression cards like Intel QuickAssist on constantly loaded servers.
moonchild 6y ago
I don't think zstd makes sense for transparent disc encryption (compared with lz4). The table at [1] shows that zstd is half the speed of lz4, and provides only a modest improvement in compression ratio. 1. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/

kzrdude 6y ago

What kind of compression level do you use?

dmw_ng 6y ago
You probably have some conditioning that discourages use of '-9', but that's generally totally fine to use with zstd in terms of speed. I've been a zstd convert for a few years now and haven't settled on any particular ideal setting. Sometimes the very high compression levels (-19) produce a worse result, sometimes they kill it. I think most of the time I'm using -9. I've seen cases where '--ultra --long -22' was bringing size down by 10% or more compared to the normal settings, and also cases where changing the defaults at all produced significantly lower compression.
- loeg 6y ago
  For approximate reference (N.B., this graph is somewhat dated — zstd 1.3.4 is from March 2018): https://raw.githubusercontent.com/facebook/zstd/v1.3.4/doc/images/CSpeed2.png https://raw.githubusercontent.com/facebook/zstd/v1.3.4/doc/i... (On the dataset used for the figure, zstd -9 was about the same MB/s as gzip -4, and compressed at ~3.5x to gzip's ~3.0x.)

alecco 6y ago

Compressing large files/backups using all available cores at the highest setting is a bliss. Sadly the choke point now is encryption. Hopefully some kind soul will some day add this to gpg or equivalent. There are parallelizable cipher modes like CTR and GCM. https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation

Algent 6y ago
If your need of encryption is related to backups, look into "Borg backup". It can handle both encryption and zstd, also do dedupe natively since it's storing data as blocks.
- formerly_proven 6y ago
  Borg only utilizes one thread and does not support multithreaded zstd, so the write throughput will likely be (much) lower than ZFS -- typically 60-120 MB/s. Since ZFS can also dedupe, Borg might not even be faster for heavily deduped backups.
  - doublepg23 6y ago
    ZFS dedupe is broken and I don’t recommend using it unless you really know what you’re getting into.
    - formerly_proven 6y ago
      Broken in what way?
      - tatersolid 6y ago
        It uses a ludicrous amount of RAM and a fixed block size and offset boundry, making it very slow and fairly ineffective.

wongarsu 6y ago

I used to be a big fan of XZ(which uses lzma2), but Zstandard has good-enough compression at high levels with orders of magnitude faster decompression. It's also more versatile in its compression levels. It's now also my go-to for new projects.

formerly_proven 6y ago
XZ/LZMA2 has the problem that it slows down a lot on incompressible data and can even choke pretty badly on compressible data, which Yann Collet's algorithms (LZ4 and ze standard) don't do nearly as much. This means zstd can be used indiscriminately, while LZMA would frequently need some sort of qualifier to avoid processing incompressible data and the associated loss of performance. To put some numbers to this, on the machine (>10yo laptop) I'm typing this on: trivial random .wav file LZMA ~20 MB/s ~2 MB/s ~1 MB/s zstd ~500 MB/s ~250 MB/s ~70 MB/s zlib ~40 MB/s ~13 MB/s ~6 MB/s
saurik 6y ago
(xz is a compression container format that doesn't imply any particular compression algorithm. It has a concept of "filters" that are applied in "chains" over "blocks". One filter happens to be "lzma2", but to say xz "uses lzma2" is a bit of a confusing mental model, as xz could just as easily "use zstd" if someone allocated a filter ID for such.)
- wongarsu 6y ago
  In principle yes, but in practice the only filters included in my version are lzma1, lzma2, a basic delta filter and a few filters for executable code. As far as I can tell all the presets that are reachable without explicitly specifying a filter chain are just lzma2 with various options set. I would love if xz had a zstd filter and a filter that adds/processes some error correction code. But with what's there right now I think it's fair to treat it as a pure lzma2 container format.

walrus01 6y ago

For system backups and logs and stuff that rarely need to be decompressed, but stored for a while and then deleted in rotation, I still strongly prefer xzip. It's much slower to create a .tar.xz file with the default settings, but also saves a considerable amount of disk space. Side by side comparison right now between a full system file tree backup of a mediawiki server, the .tar.xz file is about 1.8GB, the .tar.zstd file is 2.15GB. The same thing in traditional .tar.gz format is obviously even larger. From the research I've done on common uses of zstd it's much better suited to applications where very high speed of compression and decompression is a major requirement, but absolute smallest file size is not. Typically as a replacement for gzip.

petre 6y ago
Try xz -1. Faster than gzip, smaller size. We use it for regular backups. Although the xz format is not recommended for long term archiving because it's unsafe. https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- walrus01 6y ago
  I'm going to read this in more detail later today when I have time. But I just skimmed it, and most of the points about xz being fragile and having a weird format would seem to apply in a scenario in which a .tar.xz file became corrupted or bit-flipped from its original? eg: in which the .xz file, after some long period of archiving, no longer matched the same sha256sum as when it was originally created. Thereby making it difficult or a huge hassle to get authentic lossless/uncorrupted data out of it?
- loeg 6y ago
  The "unsafe" thing is nonsense. The expressed concern is how the contents degrade in the presence of arbitrary bit flips. Any highly-compressed format is going to be susceptible to corruption amplification from random bitflips, and the higher compression level, the larger fallout. Ensure your compressed archives are not corrupted using some external FEC; either in your hardware, or something like par2.
6y ago

deleted 6y ago

[deleted]

baybal2 6y ago

I wonder if software compression can now cost compete with compression cards like Intel QuickAssist on constantly loaded servers.

moonchild 6y ago

I don't think zstd makes sense for transparent disc encryption (compared with lz4). The table at [1] shows that zstd is half the speed of lz4, and provides only a modest improvement in compression ratio. 1. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/