4 ms·

That being said, can anyone able to decode this share some cases in which this would be better suited than zlib?

by jclay 8y ago

terrelln 8y ago
The x axis is compression speed, and the y axis is compression ratio. Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with systems that expect zlib.
- jclay 8y ago
  That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
  - Cyan4973 8y ago
    There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper https://github.com/facebook/zstd/tree/master/zlibWrapper
  - megous 8y ago
    Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib: https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6b2538401ec41f0f19f881b06 https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
    - kstrauser 8y ago
      Awesome! What kind of metrics differences have you seen from the change?
      - megous 8y ago
        Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case. But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones. It's certainly better than zlib in any case.
  - terrelln 8y ago
    Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper
    - jclay 8y ago
      Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
      - terrelln 8y ago
        There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...
- hinkley 8y ago
  The informatics dysfunction on this graph is just off the charts. Here's the problem. The graph is designed to make your conclusion sound right, but it doesn't actually prove that. Let's look at what the numbers really say: For the sample data, and the best case, gzip gets you a file that's about 31% of the original size. zstandard can get you a file that is 25% of the original size but it will take you four times as long to get it. If you allow it the same time as gzip, you can get 27% compression instead of 31%. That's only 13% improvement on the wire. That's nice, but it's not impressive at all. It's not a good enough reason to change your stack. The only thing that is impressive is that if you want the same compression ratio as gzip you can do it up to 20 times faster. On some hardware that's totally worth it, but not on all (because who is streaming at 700 MBps?)
  - jclay 8y ago
    There's so much that _feels_ misleading here. Paragraph before the chart: "The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed." In what cases? What methodology was used to evaluate this? I certainly expected a scientific (honest) treatment of how the performance was evaluated and the corresponding trade-offs to follow on at some point.
    - vjeux 8y ago
      If you read the full article there are many examples of real systems being migrated with their associated wins.
    - hinkley 8y ago
      I will say that we've been 'doing just fine' with zlib for longer than I'm comfortable with. I hung out on comp.compression when I was still in college and shared the dream of writing the next great compression library. Maybe the only thing I ever accomplished though was altering a Java minifier to improve compressibility of the class files. On a very deep level I'm pretty disappointed that zlib has been 'good enough' for almost 30 years. When it was Google proposing a change, I wasn't enthused about handing more control over HTTP to Google. They have too much already. The ways I'm concerned about Facebook have nothing to do with standards bodies or protocols. So maybe this is good enough. There are more options for files-in-motion and files-at-rest, and that could put it over the top. But you need to be filing high quality PRs on both HAProxy and Nginx if you want anybody to care.
danielvf 8y ago
zstd totally dominates zlib. There is nothing that zlib does that zstd does not do better. It’s just that simple. For a given compression speed, zstd will compress the file more that zlib would. For a given compression amount, zstd will compress the data much faster than zlib would.
- jandrese 8y ago
  Except for compatibility. If you are compressing files for yourself it's no big deal, but when you're compressing them for someone else you have to consider if your bleeding edge compression software will be an issue, especially if they are on some corporate locked down machine and can't install new software.
  - Cyan4973 8y ago
    zstd availability is becoming more and more common nowadays. See for example this tracker : https://repology.org/metapackage/zstd/versions https://repology.org/metapackage/zstd/versions
    - jandrese 8y ago
      Not listed: Windows
      - danielvf 8y ago
        I’ve used it for years on both Windows and OS X.
      - Cyan4973 8y ago
        On Windows, I'm using Chocolatey package manager : https://chocolatey.org/ https://chocolatey.org/ zstd is among the available packages.
        tatersolid 8y ago
        To the GGP’s point, not a single machine at $dayjob (financial services) has chocolatey on the whitelist of software allowed. We couldn’t even use GoToMeeting in our industry for ages because it required an executable download that was mostly not on the whitelist of our 2000+ customer banks & credit unions. Windows has this thing called Software Restriction Policies, think of it as an Apple App Store curated by windows sysadmins. You can whitelist by code signer, SHA256, or even file path and name (which is insecure). https://docs.microsoft.com/en-us/windows-server/identity/software-restriction-policies/software-restriction-policies https://docs.microsoft.com/en-us/windows-server/identity/sof...
cmurphycode 8y ago
The short answer is that zstd is faster and better at compressing than zlib for most input data. For more details, here's a comment I made a couple years ago: There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK (the original comment: https://news.ycombinator.com/item?id=12400503 https://news.ycombinator.com/item?id=12400503 )
- JyrkiAlakuijala 8y ago
  Note, that this benchmark uses an old version of zstd and doesn't use large window brotli.

That being said, can anyone able to decode this share some cases in which this would be better suited than zlib?

by jclay 8y ago

That being said, can anyone able to decode this share some cases in which this would be better suited than zlib?

The x axis is compression speed, and the y axis is compression ratio. Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with systems that expect zlib.

jclay 8y ago
That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
- Cyan4973 8y ago
  There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper https://github.com/facebook/zstd/tree/master/zlibWrapper
- megous 8y ago
  Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib: https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6b2538401ec41f0f19f881b06 https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
  - kstrauser 8y ago
    Awesome! What kind of metrics differences have you seen from the change?
    - megous 8y ago
      Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case. But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones. It's certainly better than zlib in any case.
- terrelln 8y ago
  Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper
  - jclay 8y ago
    Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
    - terrelln 8y ago
      There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...
hinkley 8y ago
The informatics dysfunction on this graph is just off the charts. Here's the problem. The graph is designed to make your conclusion sound right, but it doesn't actually prove that. Let's look at what the numbers really say: For the sample data, and the best case, gzip gets you a file that's about 31% of the original size. zstandard can get you a file that is 25% of the original size but it will take you four times as long to get it. If you allow it the same time as gzip, you can get 27% compression instead of 31%. That's only 13% improvement on the wire. That's nice, but it's not impressive at all. It's not a good enough reason to change your stack. The only thing that is impressive is that if you want the same compression ratio as gzip you can do it up to 20 times faster. On some hardware that's totally worth it, but not on all (because who is streaming at 700 MBps?)
- jclay 8y ago
  There's so much that _feels_ misleading here. Paragraph before the chart: "The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed." In what cases? What methodology was used to evaluate this? I certainly expected a scientific (honest) treatment of how the performance was evaluated and the corresponding trade-offs to follow on at some point.
  - vjeux 8y ago
    If you read the full article there are many examples of real systems being migrated with their associated wins.
  - hinkley 8y ago
    I will say that we've been 'doing just fine' with zlib for longer than I'm comfortable with. I hung out on comp.compression when I was still in college and shared the dream of writing the next great compression library. Maybe the only thing I ever accomplished though was altering a Java minifier to improve compressibility of the class files. On a very deep level I'm pretty disappointed that zlib has been 'good enough' for almost 30 years. When it was Google proposing a change, I wasn't enthused about handing more control over HTTP to Google. They have too much already. The ways I'm concerned about Facebook have nothing to do with standards bodies or protocols. So maybe this is good enough. There are more options for files-in-motion and files-at-rest, and that could put it over the top. But you need to be filing high quality PRs on both HAProxy and Nginx if you want anybody to care.

danielvf 8y ago

zstd totally dominates zlib. There is nothing that zlib does that zstd does not do better. It’s just that simple. For a given compression speed, zstd will compress the file more that zlib would. For a given compression amount, zstd will compress the data much faster than zlib would.

jandrese 8y ago
Except for compatibility. If you are compressing files for yourself it's no big deal, but when you're compressing them for someone else you have to consider if your bleeding edge compression software will be an issue, especially if they are on some corporate locked down machine and can't install new software.
- Cyan4973 8y ago
  zstd availability is becoming more and more common nowadays. See for example this tracker : https://repology.org/metapackage/zstd/versions https://repology.org/metapackage/zstd/versions
  - jandrese 8y ago
    Not listed: Windows
    - danielvf 8y ago
      I’ve used it for years on both Windows and OS X.
    - Cyan4973 8y ago
      On Windows, I'm using Chocolatey package manager : https://chocolatey.org/ https://chocolatey.org/ zstd is among the available packages.
      - tatersolid 8y ago
        To the GGP’s point, not a single machine at $dayjob (financial services) has chocolatey on the whitelist of software allowed. We couldn’t even use GoToMeeting in our industry for ages because it required an executable download that was mostly not on the whitelist of our 2000+ customer banks & credit unions. Windows has this thing called Software Restriction Policies, think of it as an Apple App Store curated by windows sysadmins. You can whitelist by code signer, SHA256, or even file path and name (which is insecure). https://docs.microsoft.com/en-us/windows-server/identity/software-restriction-policies/software-restriction-policies https://docs.microsoft.com/en-us/windows-server/identity/sof...

cmurphycode 8y ago

The short answer is that zstd is faster and better at compressing than zlib for most input data. For more details, here's a comment I made a couple years ago: There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK (the original comment: https://news.ycombinator.com/item?id=12400503 https://news.ycombinator.com/item?id=12400503 )

JyrkiAlakuijala 8y ago
Note, that this benchmark uses an old version of zstd and doesn't use large window brotli.