6 ms·

zstd beats gzip handily, either in terms of speed for a given compression ratio or compression ratio for a given amount of wall clock time. And that's without e

zstd beats gzip handily, either in terms of speed for a given compression ratio or compression ratio for a given amount of wall clock time. And that's without even using multiple threads, which zstd supports out of the box but for gzip you would need to use a different tool like pigz. And on top of that zstd is also much faster to decompress. In 2022, if you care about compression performance at all, there's no reason to even consider gzip unless you have a specific need for it (e.g. you have to deal with some other thing that can't handle zstd).

jamal-kumar 4y ago
Thanks for that information, I wasn't aware until reading into it from your suggestion here that it's what underpins the encryption in btrfs, which is something I've experimented with a bit but didn't realize had zstd as a part of it. Very useful!
twotwotwo 4y ago
For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
- GoblinSlayer 4y ago
  So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.
  - nezirus 4y ago
    If you are concerned about RAM usage, there are some trade-offs available (slightly lower compression ratio), for more effective resource usage. As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz https://github.com/wtarreau/libslz
  - lifthrasiir 4y ago
    Virtually every compression format utilizing LZ77 has a way to declare the window size used and reject any archive requiring too large window. This is even the case with the traditional DEFLATE.
  - twotwotwo 4y ago
    When compressing, window size is a tunable option. I think the compression level sets it implicitly, but it can be set explicitly using brotli -w or windowLog in the advanced zstd API (https://facebook.github.io/zstd/zstd_manual.html https://facebook.github.io/zstd/zstd_manual.html). Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.
  - semi 4y ago
    In the age of cloud computing, compression can easily be offloaded to another layer. CDNs can do brotli compression for you for example.
solardev 4y ago
Is zstd readily available by default in distros and browsers and such?
- josephg 4y ago
  Zstd is available in distros but not browsers. Brotli is available in all web modern browsers for sending http resources to the browser. Very handy for wasm and things like that: https://caniuse.com/brotli https://caniuse.com/brotli
  - yangcheng 4y ago
    Brotli is not enabled by default on Android's webview yet.
- chasil 4y ago
  If I remember correctly, it's in EPEL for CentOS and peer distributions. Maybe RHEL v9 took it into the main repositories, would have to check.
- nijave 4y ago
  In distros, I would say no anecdotally (not sure about latest versions). I frequently find it not installed on things like Ubuntu 18/20 and RHEL-based systems whereas gzip is pretty much universally available on everything from embedded to Windows
TylerE 4y ago
Of course, if you care about (random access) performance you aren’t using that, either. I don’t really understand why anyone uses tar anymore, really.
- awestroke 4y ago
  What would you use instead of tar?
  - selfhoster11 4y ago
    ZIP with LZMA compression is one possibility. Random seek were always supported, and later versions also gained support for modern encryption, modern compression, and IIRC Unix permissions too.
    - wruza 4y ago
      ZIP always had filename encoding issues, afair. At least in a “real-world zip”. Have they been resolved? (And does tar have them?)
      - deleted 4y ago
        [deleted]
      - pastage 4y ago
        If you use pax format all metadata is assumed to be UTF-8. So that would work with tar.
    - account42 4y ago
      ZIP with non-deflate compression is just silly. If you are not going to create defacto-standard compatible .zips then you might as well use a better archiving format.
      - selfhoster11 4y ago
        Your choice of tools always depends on the use case. If you're only going to be creating these archives for internal use, or will be exchanging them/reading them on modern Unix-like systems, then there is no worry about the decompressor not supporting LZMA. If you need to exchange files with feature-poor systems, then yes - stick to the de-facto standard. Besides, what "better" archiving format are you going to use? Tar clearly isn't one (the great... grandparent comment wouldn't be asking for alternatives if it were). "Defacto standard" ZIP files have a poor compression ratio. 7z feels strange to use in a professional setting or for long-term storage. RAR is a closed-source abomination. Are there any other "better" archiving formats that have wider (de)compressor adoption than ZIP+LZMA?
systemvoltage 4y ago
Intel's new Sapphire Rapids Xeons will have special on-chip gzip IP: https://www.anandtech.com/show/17596/intel-demos-sapphire-rapids-accelerators-at-innovation-2022 https://www.anandtech.com/show/17596/intel-demos-sapphire-ra...
- happymellon 4y ago
  That sounds about 30 years too late.
- tecleandor 4y ago
  That sounds like Intel Quick Assist (QAT). That's a not-greatly-documented technology that's included in some processors (including a bunch of Celerons, but mostly Xeons) and in PCIe cards. They are used in firewalls, ssl terminators and storage systems, as they can provide Gbps of compression or cryptography. See one of the older cards: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/quickassist-adapter-8920-brief.pdf https://www.intel.com/content/dam/www/public/us/en/documents... Netgate sells one 8950 based card for it's firewalls: https://shop.netgate.com/products/netgate-cpic-8955-cryptographic-accelerator-card-with-qat https://shop.netgate.com/products/netgate-cpic-8955-cryptogr... STH had a report this summer: https://www.servethehome.com/intel-quickassist-parts-and-cards-by-qat-generation/ https://www.servethehome.com/intel-quickassist-parts-and-car... This cards are expensive when bought new, but you can probably buy some of them cheaply in eBay. Please don't drive the prices up ;)
- hulitu 4y ago
  They shall patent it quick before Fraunhofer or Dolby does it.
hrbf 4y ago
I’d like to throw in lz4 compression as a ridiculously fast, low CPU alternative while still providing acceptable ratios for the presented use-case. Maybe obvious for many but still worth mentioning is that on Unix-like systems any external compression can be used by piping to stdout, no need to rely on built-in support. Especially relevant with BSD tar implementations. Example: tar cfv - /path/to/files | lz4 > output.tar.lz4

jamal-kumar 4y ago

Thanks for that information, I wasn't aware until reading into it from your suggestion here that it's what underpins the encryption in btrfs, which is something I've experimented with a bit but didn't realize had zstd as a part of it. Very useful!

twotwotwo 4y ago

For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.

GoblinSlayer 4y ago
So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.
- nezirus 4y ago
  If you are concerned about RAM usage, there are some trade-offs available (slightly lower compression ratio), for more effective resource usage. As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz https://github.com/wtarreau/libslz
- lifthrasiir 4y ago
  Virtually every compression format utilizing LZ77 has a way to declare the window size used and reject any archive requiring too large window. This is even the case with the traditional DEFLATE.
- twotwotwo 4y ago
  When compressing, window size is a tunable option. I think the compression level sets it implicitly, but it can be set explicitly using brotli -w or windowLog in the advanced zstd API (https://facebook.github.io/zstd/zstd_manual.html https://facebook.github.io/zstd/zstd_manual.html). Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.
- semi 4y ago
  In the age of cloud computing, compression can easily be offloaded to another layer. CDNs can do brotli compression for you for example.

solardev 4y ago

Is zstd readily available by default in distros and browsers and such?

josephg 4y ago
Zstd is available in distros but not browsers. Brotli is available in all web modern browsers for sending http resources to the browser. Very handy for wasm and things like that: https://caniuse.com/brotli https://caniuse.com/brotli
- yangcheng 4y ago
  Brotli is not enabled by default on Android's webview yet.
chasil 4y ago
If I remember correctly, it's in EPEL for CentOS and peer distributions. Maybe RHEL v9 took it into the main repositories, would have to check.
nijave 4y ago
In distros, I would say no anecdotally (not sure about latest versions). I frequently find it not installed on things like Ubuntu 18/20 and RHEL-based systems whereas gzip is pretty much universally available on everything from embedded to Windows

TylerE 4y ago

Of course, if you care about (random access) performance you aren’t using that, either. I don’t really understand why anyone uses tar anymore, really.

awestroke 4y ago
What would you use instead of tar?
- selfhoster11 4y ago
  ZIP with LZMA compression is one possibility. Random seek were always supported, and later versions also gained support for modern encryption, modern compression, and IIRC Unix permissions too.
  - wruza 4y ago
    ZIP always had filename encoding issues, afair. At least in a “real-world zip”. Have they been resolved? (And does tar have them?)
    - deleted 4y ago
      [deleted]
    - pastage 4y ago
      If you use pax format all metadata is assumed to be UTF-8. So that would work with tar.
  - account42 4y ago
    ZIP with non-deflate compression is just silly. If you are not going to create defacto-standard compatible .zips then you might as well use a better archiving format.
    - selfhoster11 4y ago
      Your choice of tools always depends on the use case. If you're only going to be creating these archives for internal use, or will be exchanging them/reading them on modern Unix-like systems, then there is no worry about the decompressor not supporting LZMA. If you need to exchange files with feature-poor systems, then yes - stick to the de-facto standard. Besides, what "better" archiving format are you going to use? Tar clearly isn't one (the great... grandparent comment wouldn't be asking for alternatives if it were). "Defacto standard" ZIP files have a poor compression ratio. 7z feels strange to use in a professional setting or for long-term storage. RAR is a closed-source abomination. Are there any other "better" archiving formats that have wider (de)compressor adoption than ZIP+LZMA?

systemvoltage 4y ago

Intel's new Sapphire Rapids Xeons will have special on-chip gzip IP: https://www.anandtech.com/show/17596/intel-demos-sapphire-rapids-accelerators-at-innovation-2022 https://www.anandtech.com/show/17596/intel-demos-sapphire-ra...

happymellon 4y ago
That sounds about 30 years too late.
tecleandor 4y ago
That sounds like Intel Quick Assist (QAT). That's a not-greatly-documented technology that's included in some processors (including a bunch of Celerons, but mostly Xeons) and in PCIe cards. They are used in firewalls, ssl terminators and storage systems, as they can provide Gbps of compression or cryptography. See one of the older cards: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/quickassist-adapter-8920-brief.pdf https://www.intel.com/content/dam/www/public/us/en/documents... Netgate sells one 8950 based card for it's firewalls: https://shop.netgate.com/products/netgate-cpic-8955-cryptographic-accelerator-card-with-qat https://shop.netgate.com/products/netgate-cpic-8955-cryptogr... STH had a report this summer: https://www.servethehome.com/intel-quickassist-parts-and-cards-by-qat-generation/ https://www.servethehome.com/intel-quickassist-parts-and-car... This cards are expensive when bought new, but you can probably buy some of them cheaply in eBay. Please don't drive the prices up ;)
hulitu 4y ago
They shall patent it quick before Fraunhofer or Dolby does it.

hrbf 4y ago

I’d like to throw in lz4 compression as a ridiculously fast, low CPU alternative while still providing acceptable ratios for the presented use-case. Maybe obvious for many but still worth mentioning is that on Unix-like systems any external compression can be used by piping to stdout, no need to rely on built-in support. Especially relevant with BSD tar implementations. Example: tar cfv - /path/to/files | lz4 > output.tar.lz4