10 ms·

Improving compression at scale with Zstandard

139 points by felixhandte 8y ago

jclay 8y ago
I find the first chart so hard to understand. The axes need labels, and the color scheme is not ideal. They should use different line styles, and add a caption below summarizing the findings. There's a reason journals often require graphs to be formatted this way. This is a resource I've found helpful: https://www3.nd.edu/~pkamat/pdf/graphs.pdf https://www3.nd.edu/~pkamat/pdf/graphs.pdf "Consider readers with color blindness or deficiencies" "Avoid colors that are difficult to distinguish"
- jclay 8y ago
  That being said, can anyone able to decode this share some cases in which this would be better suited than zlib?
  - terrelln 8y ago
    The x axis is compression speed, and the y axis is compression ratio. Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with systems that expect zlib.
    - jclay 8y ago
      That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
      - Cyan4973 8y ago
        There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper https://github.com/facebook/zstd/tree/master/zlibWrapper
      - megous 8y ago
        Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib: https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6b2538401ec41f0f19f881b06 https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
        kstrauser 8y ago
        Awesome! What kind of metrics differences have you seen from the change?
        megous 8y ago
        Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case. But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones. It's certainly better than zlib in any case.
      - terrelln 8y ago
        Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper
        jclay 8y ago
        Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
        terrelln 8y ago
        There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...
    - hinkley 8y ago
      The informatics dysfunction on this graph is just off the charts. Here's the problem. The graph is designed to make your conclusion sound right, but it doesn't actually prove that. Let's look at what the numbers really say: For the sample data, and the best case, gzip gets you a file that's about 31% of the original size. zstandard can get you a file that is 25% of the original size but it will take you four times as long to get it. If you allow it the same time as gzip, you can get 27% compression instead of 31%. That's only 13% improvement on the wire. That's nice, but it's not impressive at all. It's not a good enough reason to change your stack. The only thing that is impressive is that if you want the same compression ratio as gzip you can do it up to 20 times faster. On some hardware that's totally worth it, but not on all (because who is streaming at 700 MBps?)
      - jclay 8y ago
        There's so much that _feels_ misleading here. Paragraph before the chart: "The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed." In what cases? What methodology was used to evaluate this? I certainly expected a scientific (honest) treatment of how the performance was evaluated and the corresponding trade-offs to follow on at some point.
        vjeux 8y ago
        If you read the full article there are many examples of real systems being migrated with their associated wins.
        hinkley 8y ago
        I will say that we've been 'doing just fine' with zlib for longer than I'm comfortable with. I hung out on comp.compression when I was still in college and shared the dream of writing the next great compression library. Maybe the only thing I ever accomplished though was altering a Java minifier to improve compressibility of the class files. On a very deep level I'm pretty disappointed that zlib has been 'good enough' for almost 30 years. When it was Google proposing a change, I wasn't enthused about handing more control over HTTP to Google. They have too much already. The ways I'm concerned about Facebook have nothing to do with standards bodies or protocols. So maybe this is good enough. There are more options for files-in-motion and files-at-rest, and that could put it over the top. But you need to be filing high quality PRs on both HAProxy and Nginx if you want anybody to care.
  - danielvf 8y ago
    zstd totally dominates zlib. There is nothing that zlib does that zstd does not do better. It’s just that simple. For a given compression speed, zstd will compress the file more that zlib would. For a given compression amount, zstd will compress the data much faster than zlib would.
    - jandrese 8y ago
      Except for compatibility. If you are compressing files for yourself it's no big deal, but when you're compressing them for someone else you have to consider if your bleeding edge compression software will be an issue, especially if they are on some corporate locked down machine and can't install new software.
      - Cyan4973 8y ago
        zstd availability is becoming more and more common nowadays. See for example this tracker : https://repology.org/metapackage/zstd/versions https://repology.org/metapackage/zstd/versions
        jandrese 8y ago
        Not listed: Windows
        danielvf 8y ago
        I’ve used it for years on both Windows and OS X.
        Cyan4973 8y ago
        On Windows, I'm using Chocolatey package manager : https://chocolatey.org/ https://chocolatey.org/ zstd is among the available packages.
        tatersolid 8y ago
        To the GGP’s point, not a single machine at $dayjob (financial services) has chocolatey on the whitelist of software allowed. We couldn’t even use GoToMeeting in our industry for ages because it required an executable download that was mostly not on the whitelist of our 2000+ customer banks & credit unions. Windows has this thing called Software Restriction Policies, think of it as an Apple App Store curated by windows sysadmins. You can whitelist by code signer, SHA256, or even file path and name (which is insecure). https://docs.microsoft.com/en-us/windows-server/identity/software-restriction-policies/software-restriction-policies https://docs.microsoft.com/en-us/windows-server/identity/sof...
  - cmurphycode 8y ago
    The short answer is that zstd is faster and better at compressing than zlib for most input data. For more details, here's a comment I made a couple years ago: There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK (the original comment: https://news.ycombinator.com/item?id=12400503 https://news.ycombinator.com/item?id=12400503 )
    - JyrkiAlakuijala 8y ago
      Note, that this benchmark uses an old version of zstd and doesn't use large window brotli.
- kzrdude 8y ago
  Sure, this is the chart they usually show so it should be: x-axis: compression speed (in input data) y-axis: compression ratio (1x is no compression).
facecrook 8y ago
Good to know their data pipeline was performant as they sent my personal data to all sorts of third party platforms without my knowledge https://www.nytimes.com/2018/12/18/technology/facebook-privacy.html https://www.nytimes.com/2018/12/18/technology/facebook-priva...
josephg 8y ago
How does zstd compare with brotli? Would it be a better compression standard for http responses?
- stochastic_monk 8y ago
  Both brotli and zstd support user-provided dictionaries, which helps with compressing files which contain items from the dictionary. This is particularly helpful for small files. Zstd is both faster and more effective at compressing.
  - felixhandte 8y ago
    At the moment, Brotli doesn't accept a user-provided dictionary. I know they're working to re-introduce that functionality, but it's not currently present.
  - JyrkiAlakuijala 8y ago
    Actually brotli compresses some more, often around 5 % more density. If you compress to same density, brotli tends to be significantly faster. Zstd decompresses faster. Brotli is naturally limited to 16 MB of memory use whereas ZStd makes less attempts to reduce memory use. If you want to use more memory with Brotli, you need to use another flag '--large_window' instead of '--window', to indicate that you really want it, whereas zstd uses a lot of memory more silently. When you compare compression levels, you should compare with same memory consumption -- and if you do that, brotli tends to win by 5 %. When in doubt in benchmarking, run brotli with --large_window 30 https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a... is a summary of a recent large-scale experiment of a large variety of compressors, including brotli, zstd and LZMA.
    - twitch_checksum 8y ago
      > ZStd makes less attempts to reduce memory use Ever heard of flag `--ultra` ? Without it, zstd memory usage will be limited to 8 MB. It has been there since, forever.
stochastic_monk 8y ago
The best thing about zstd is its zlibWrapper, which lets you write code as if you’re consuming zlib-compressed files while transparently working with zlib-, zstd-, or uncompressed files. I build several of my tools with zstd for this reason.
- loeg 8y ago
  zlibWrapper seems useful if you're specifically using the low level zlib APIs already and don't want to change your code very much. But if you just use zopen() or similar, or are willing to make minor changes, I don't see much benefit. (Especially given the performance gap vs native zstd APIs). I have seen some fopencookie(3)-based zstd/zlib/xz/etc FILE-object wrappers floating around that make it pretty easy to work with any compression library's streaming APIs.
  - stochastic_monk 8y ago
    zlib is standard in my field, so being compatible is a big plus. I should do some testing with the zstd API, though. Thanks for the performance discrepancy heads-up!
- hyc_symas 8y ago
  Did someone say compression wrapper? https://github.com/hyc/polyZ https://github.com/hyc/polyZ Did that 4 years ago. > The snappy wrapper can serve as a frontend for bzip2, lz4, lz4hc, lzma, lzo, and zlib. > The zlib wrapper can serve as a frontend for bzip2, lz4, lz4hc, lzma, lzo, and snappy. Very handy for quickly benchmarking multiple compressors without having to write multiple implementations of the same test code. As illustrated here http://www.lmdb.tech/bench/inmem/compress/ http://www.lmdb.tech/bench/inmem/compress/ Feel free to PR support for brotli or zstd or anything else that comes along.
IvanK_net 8y ago
My browser loaded that website with a header: accept-encoding: gzip, deflate, br ("br" means Brotli by Google) The response had a header: content-encoding: gzip Zstandard looks like an improvement of DEFLATE (= gzip = zlib) and its specification is only 3x longer (even though it is introduced 22 years later): https://tools.ietf.org/html/rfc8478 https://tools.ietf.org/html/rfc8478 Since Zstandard is so simple and efficient, I thought it would get into browsers very quickly. Then, it could make sense to compress even PNG or JPG images, which are usually impossible to compress with DEFLATE.
- felixhandte 8y ago
  It's something we're actively working on!
- wolf550e 8y ago
  Brotli is a competitor of zstd from Google. Google integrated Brotli into Chrome. Facebook are trying to get zstd into Chrome.
- JyrkiAlakuijala 8y ago
  With internet speed significantly above the decompression speeds zstd is favorable. With internet speeds below the decompression speeds brotli is favorable because less bytes need to be transmitted. Usual users have internet speeds of a 10 MB/s or so, and brotli is more favorable up to around 200 MB/s (2 gbps) internet speeds. It is also not just about speed, but mobile users need to pay less with brotli as less bytes are transferred. Further (I'm not an expert on this, somewhat speculative) but likely the streaming properties of brotli are slightly better, i.e., less bytes are needed to hide in a buffer to be able to decode bytes. This may allow the browser to issue new fetches for urls in an html document earlier with brotli than with zstd.
  - IvanK_net 8y ago
    From what you say, it sounds like you think, that Brotli has a better compression ratio, than Zstdandard. According to this chart, Zstandard has a better compression ratio, while compressing and decompressing faster than Brotli: https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
m0zg 8y ago
I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of speed and memory savings. On the high end it'd be great to exceed XZ/LZMA. Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy). At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.
- terrelln 8y ago
  We recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compress stronger than xz, but we hope to close the gap some.
  - m0zg 8y ago
    Ah, I didn't know, thanks! I'll have to re-test.
    - JyrkiAlakuijala 8y ago
      With zstd you end up on average 5-6 % worse than LZMA in density on a variety of large test corpora, but decodes 8x faster. Brotli with --large_window 30 can get within 0.6 % of LZMA, and decodes 5x faster than LZMA. https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a...
- praseodym 8y ago
  It does depend on your dataset. In my experiments, Zstandard did outperform XZ/LZMA in terms of compression ratio, but not by much. However, the massive improvement in decompression speed is what won me over.
jzawodn 8y ago
Am I the only one getting sick of "at scale"?
- erikb 8y ago
  It's Enterprise slang for "we are big, so we assume everything we do works better at big scale, but please don't check if it's true, just trust us". (in this thread it might be true, usually it isn't though)
sammycdubs 8y ago
What's the Weissman score?
- felixhandte 8y ago
  Ha, that just came up: https://github.com/facebook/zstd/issues/1087 https://github.com/facebook/zstd/issues/1087
- wolf550e 8y ago
  Charles Bloom is a data compression expert working on the compression suite at RAD game tools (proprietary library for game developers that delivers better compression ratios and better decompression speed over the best available in open source, i.e. lz4, zstd and lzma): http://www.radgametools.com/oodlecompressors.htm http://www.radgametools.com/oodlecompressors.htm Here is a blog post of his about Weissman score for compressors, including his and Yann Collet's: https://cbloomrants.blogspot.com/2018/01/05-17-16-weissman-score.html https://cbloomrants.blogspot.com/2018/01/05-17-16-weissman-s...
  - tpetry 8y ago
    But no third party not involved with RAD game tools has _ever_ verified these bold claims. Heck, everyone can construct benchmarks to make almost any compression algorithm outperform any other. But if it happens with real world data too is another story.
    - Legogris 8y ago
      Thing is, compression performance characteristics depend on data. You will have to run benchmarks on your own data to get real representative and conclusive results.
    - wolf550e 8y ago
      https://richg42.blogspot.com/2016/08/rads-ground-breaking-lossless.html https://richg42.blogspot.com/2016/08/rads-ground-breaking-lo...
valarauca1 8y ago
I'd really like to thank Cyan for their contributions. `zstd` and `lz4` are great. I'm pretty much exclusively using `zstd` for my tarball needs in the present day as it beats the pants off `gzip` and for plane text code (most of what I compress) it performs amazingly. (shameless self promotion) I wrote my own tar clone to make usage of it [1]. It is nice to have disk IO be the limiting factor on decompression even when you are using NVMe drives. [1] https://github.com/valarauca/car https://github.com/valarauca/car
koolba 8y ago
> And the zstd binary itself includes a dictionary trainer (zstd --train). Building a pipeline for handling compression dictionaries can therefore be reduced to being a matter of gluing these building blocks together in relatively straightforward ways. What happens if your user data trained dictionary ends up storing user data and you receive a GPDR destruction request?
- bowmessage 8y ago
  standard GDPR cop out disclaimer that's something about needing to keep the data for technical reasons I wish I knew more about it, but that's what I keep hearing.
  - karavelov 8y ago
    The dictionary data in not personally identifiable, so no need to do anything. At least this is my understanding of GDRP, I am not a loyer.
- Legogris 8y ago
  It would not be covered by GDPR, just like e.g. private emails containing PII about a third party are not in scope from perspective of the e-mail provider.
karavelov 8y ago
> Two years ago, Facebook open-sourced Zstandard v1.0... Bullshit, Zstd was open-source from the very beginning, they just hired Yann and moved the project under facebook org. How do I know? I have written the JVM bindings [1] since v0.1 that are now used by Spark, Kafka, etc. EDIT: Actually, my initial bindings were against v0.0.2 [2] Kudos to FB for hiring him and helping Zstd getting production ready. This is just a PR false claim. [1] https://github.com/luben/zstd-jni https://github.com/luben/zstd-jni [2] https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da3268af6aa73dce6014298ef https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da32...
- deleted 8y ago
  [deleted]
- hnbroseph 8y ago
  on the github releases page, 1.0 mentions licensing changes of some sort. it also has pre-1.0 stuff as well.
  - karavelov 8y ago
    Previously the license was BSD (as my commit above shows). v1.0 moved it to BSD+GPL2+Patent clause that is not more open-source than just BSD. After concerns from the community the patent clause was dropped, a little bit after React.
- felixhandte 8y ago
  Given that it was Yann himself who wrote that sentence, I think that's a needlessly uncharitable interpretation. Maybe a better wording would have been "Two years ago, we released Zstandard v1.0, an open source ...". But I don't think we anticipated anyone would read that much into it.
  - karavelov 8y ago
    Yes, may be I read it the wrong way. I later noticed the authors. BTW, thank you for your part at making it the best compression library for wide variaty of cases.
golergka 8y ago
I used zstandard to compress mesages in P2P multiplayer game engine, and, taught on our real-life packets, it got us 2x-5x improvement. Awesome library, will use it in any similar project from now on.

139 points by felixhandte 8y ago

I find the first chart so hard to understand. The axes need labels, and the color scheme is not ideal. They should use different line styles, and add a caption below summarizing the findings. There's a reason journals often require graphs to be formatted this way. This is a resource I've found helpful: https://www3.nd.edu/~pkamat/pdf/graphs.pdf https://www3.nd.edu/~pkamat/pdf/graphs.pdf "Consider readers with color blindness or deficiencies" "Avoid colors that are difficult to distinguish"

jclay 8y ago
That being said, can anyone able to decode this share some cases in which this would be better suited than zlib?
- terrelln 8y ago
  The x axis is compression speed, and the y axis is compression ratio. Zstandard outperforms zlib in compression ratio, compression speed, and decompression speed (not shown). The only reason to stick with zlib is for compatibility with systems that expect zlib.
  - jclay 8y ago
    That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacement for zlib?
    - Cyan4973 8y ago
      There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper https://github.com/facebook/zstd/tree/master/zlibWrapper
    - megous 8y ago
      Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib: https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6b2538401ec41f0f19f881b06 https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
      - kstrauser 8y ago
        Awesome! What kind of metrics differences have you seen from the change?
        megous 8y ago
        Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case. But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones. It's certainly better than zlib in any case.
    - terrelln 8y ago
      Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper
      - jclay 8y ago
        Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
        terrelln 8y ago
        There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...
  - hinkley 8y ago
    The informatics dysfunction on this graph is just off the charts. Here's the problem. The graph is designed to make your conclusion sound right, but it doesn't actually prove that. Let's look at what the numbers really say: For the sample data, and the best case, gzip gets you a file that's about 31% of the original size. zstandard can get you a file that is 25% of the original size but it will take you four times as long to get it. If you allow it the same time as gzip, you can get 27% compression instead of 31%. That's only 13% improvement on the wire. That's nice, but it's not impressive at all. It's not a good enough reason to change your stack. The only thing that is impressive is that if you want the same compression ratio as gzip you can do it up to 20 times faster. On some hardware that's totally worth it, but not on all (because who is streaming at 700 MBps?)
    - jclay 8y ago
      There's so much that _feels_ misleading here. Paragraph before the chart: "The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed." In what cases? What methodology was used to evaluate this? I certainly expected a scientific (honest) treatment of how the performance was evaluated and the corresponding trade-offs to follow on at some point.
      - vjeux 8y ago
        If you read the full article there are many examples of real systems being migrated with their associated wins.
      - hinkley 8y ago
        I will say that we've been 'doing just fine' with zlib for longer than I'm comfortable with. I hung out on comp.compression when I was still in college and shared the dream of writing the next great compression library. Maybe the only thing I ever accomplished though was altering a Java minifier to improve compressibility of the class files. On a very deep level I'm pretty disappointed that zlib has been 'good enough' for almost 30 years. When it was Google proposing a change, I wasn't enthused about handing more control over HTTP to Google. They have too much already. The ways I'm concerned about Facebook have nothing to do with standards bodies or protocols. So maybe this is good enough. There are more options for files-in-motion and files-at-rest, and that could put it over the top. But you need to be filing high quality PRs on both HAProxy and Nginx if you want anybody to care.
- danielvf 8y ago
  zstd totally dominates zlib. There is nothing that zlib does that zstd does not do better. It’s just that simple. For a given compression speed, zstd will compress the file more that zlib would. For a given compression amount, zstd will compress the data much faster than zlib would.
  - jandrese 8y ago
    Except for compatibility. If you are compressing files for yourself it's no big deal, but when you're compressing them for someone else you have to consider if your bleeding edge compression software will be an issue, especially if they are on some corporate locked down machine and can't install new software.
    - Cyan4973 8y ago
      zstd availability is becoming more and more common nowadays. See for example this tracker : https://repology.org/metapackage/zstd/versions https://repology.org/metapackage/zstd/versions
      - jandrese 8y ago
        Not listed: Windows
        danielvf 8y ago
        I’ve used it for years on both Windows and OS X.
        Cyan4973 8y ago
        On Windows, I'm using Chocolatey package manager : https://chocolatey.org/ https://chocolatey.org/ zstd is among the available packages.
        tatersolid 8y ago
        To the GGP’s point, not a single machine at $dayjob (financial services) has chocolatey on the whitelist of software allowed. We couldn’t even use GoToMeeting in our industry for ages because it required an executable download that was mostly not on the whitelist of our 2000+ customer banks & credit unions. Windows has this thing called Software Restriction Policies, think of it as an Apple App Store curated by windows sysadmins. You can whitelist by code signer, SHA256, or even file path and name (which is insecure). https://docs.microsoft.com/en-us/windows-server/identity/software-restriction-policies/software-restriction-policies https://docs.microsoft.com/en-us/windows-server/identity/sof...
- cmurphycode 8y ago
  The short answer is that zstd is faster and better at compressing than zlib for most input data. For more details, here's a comment I made a couple years ago: There's a nice comparison of compression algorithms (including zlib, zstd, brotli, snappy, etc) here: https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ It's nice because it uses many datasets and machine platforms. Unfortunately the graphs leave a bit to be desired, especially when you're trying to compare two algorithms. They provide the raw data, but it needs a little munging to make it workable. For my day job I made the following graph to compare zlib level1 and zstd. This was using the "peltast" platform which is a Xeon based system, because that was most relevant for us. http://imgur.com/a/0a3kK http://imgur.com/a/0a3kK (the original comment: https://news.ycombinator.com/item?id=12400503 https://news.ycombinator.com/item?id=12400503 )
  - JyrkiAlakuijala 8y ago
    Note, that this benchmark uses an old version of zstd and doesn't use large window brotli.
kzrdude 8y ago
Sure, this is the chart they usually show so it should be: x-axis: compression speed (in input data) y-axis: compression ratio (1x is no compression).

facecrook 8y ago

Good to know their data pipeline was performant as they sent my personal data to all sorts of third party platforms without my knowledge https://www.nytimes.com/2018/12/18/technology/facebook-privacy.html https://www.nytimes.com/2018/12/18/technology/facebook-priva...

josephg 8y ago

How does zstd compare with brotli? Would it be a better compression standard for http responses?

stochastic_monk 8y ago
Both brotli and zstd support user-provided dictionaries, which helps with compressing files which contain items from the dictionary. This is particularly helpful for small files. Zstd is both faster and more effective at compressing.
- felixhandte 8y ago
  At the moment, Brotli doesn't accept a user-provided dictionary. I know they're working to re-introduce that functionality, but it's not currently present.
- JyrkiAlakuijala 8y ago
  Actually brotli compresses some more, often around 5 % more density. If you compress to same density, brotli tends to be significantly faster. Zstd decompresses faster. Brotli is naturally limited to 16 MB of memory use whereas ZStd makes less attempts to reduce memory use. If you want to use more memory with Brotli, you need to use another flag '--large_window' instead of '--window', to indicate that you really want it, whereas zstd uses a lot of memory more silently. When you compare compression levels, you should compare with same memory consumption -- and if you do that, brotli tends to win by 5 %. When in doubt in benchmarking, run brotli with --large_window 30 https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a... is a summary of a recent large-scale experiment of a large variety of compressors, including brotli, zstd and LZMA.
  - twitch_checksum 8y ago
    > ZStd makes less attempts to reduce memory use Ever heard of flag `--ultra` ? Without it, zstd memory usage will be limited to 8 MB. It has been there since, forever.

stochastic_monk 8y ago

The best thing about zstd is its zlibWrapper, which lets you write code as if you’re consuming zlib-compressed files while transparently working with zlib-, zstd-, or uncompressed files. I build several of my tools with zstd for this reason.

loeg 8y ago
zlibWrapper seems useful if you're specifically using the low level zlib APIs already and don't want to change your code very much. But if you just use zopen() or similar, or are willing to make minor changes, I don't see much benefit. (Especially given the performance gap vs native zstd APIs). I have seen some fopencookie(3)-based zstd/zlib/xz/etc FILE-object wrappers floating around that make it pretty easy to work with any compression library's streaming APIs.
- stochastic_monk 8y ago
  zlib is standard in my field, so being compatible is a big plus. I should do some testing with the zstd API, though. Thanks for the performance discrepancy heads-up!
hyc_symas 8y ago
Did someone say compression wrapper? https://github.com/hyc/polyZ https://github.com/hyc/polyZ Did that 4 years ago. > The snappy wrapper can serve as a frontend for bzip2, lz4, lz4hc, lzma, lzo, and zlib. > The zlib wrapper can serve as a frontend for bzip2, lz4, lz4hc, lzma, lzo, and snappy. Very handy for quickly benchmarking multiple compressors without having to write multiple implementations of the same test code. As illustrated here http://www.lmdb.tech/bench/inmem/compress/ http://www.lmdb.tech/bench/inmem/compress/ Feel free to PR support for brotli or zstd or anything else that comes along.

IvanK_net 8y ago

My browser loaded that website with a header: accept-encoding: gzip, deflate, br ("br" means Brotli by Google) The response had a header: content-encoding: gzip Zstandard looks like an improvement of DEFLATE (= gzip = zlib) and its specification is only 3x longer (even though it is introduced 22 years later): https://tools.ietf.org/html/rfc8478 https://tools.ietf.org/html/rfc8478 Since Zstandard is so simple and efficient, I thought it would get into browsers very quickly. Then, it could make sense to compress even PNG or JPG images, which are usually impossible to compress with DEFLATE.

felixhandte 8y ago
It's something we're actively working on!
wolf550e 8y ago
Brotli is a competitor of zstd from Google. Google integrated Brotli into Chrome. Facebook are trying to get zstd into Chrome.
JyrkiAlakuijala 8y ago
With internet speed significantly above the decompression speeds zstd is favorable. With internet speeds below the decompression speeds brotli is favorable because less bytes need to be transmitted. Usual users have internet speeds of a 10 MB/s or so, and brotli is more favorable up to around 200 MB/s (2 gbps) internet speeds. It is also not just about speed, but mobile users need to pay less with brotli as less bytes are transferred. Further (I'm not an expert on this, somewhat speculative) but likely the streaming properties of brotli are slightly better, i.e., less bytes are needed to hide in a buffer to be able to decode bytes. This may allow the browser to issue new fetches for urls in an html document earlier with brotli than with zstd.
- IvanK_net 8y ago
  From what you say, it sounds like you think, that Brotli has a better compression ratio, than Zstdandard. According to this chart, Zstandard has a better compression ratio, while compressing and decompressing faster than Brotli: https://facebook.github.io/zstd/ https://facebook.github.io/zstd/

m0zg 8y ago

I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of speed and memory savings. On the high end it'd be great to exceed XZ/LZMA. Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy). At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.

terrelln 8y ago
We recently added negative compression levels that extends the fast end of the spectrum significantly. We are also working on incrementally improving the strong end of the compression ratio spectrum. We don't expect plain zstd to compress stronger than xz, but we hope to close the gap some.
- m0zg 8y ago
  Ah, I didn't know, thanks! I'll have to re-test.
  - JyrkiAlakuijala 8y ago
    With zstd you end up on average 5-6 % worse than LZMA in density on a variety of large test corpora, but decodes 8x faster. Brotli with --large_window 30 can get within 0.6 % of LZMA, and decodes 5x faster than LZMA. https://encode.ru/threads/2947-large-window-brotli-results-available-for-a-several-corpora https://encode.ru/threads/2947-large-window-brotli-results-a...
praseodym 8y ago
It does depend on your dataset. In my experiments, Zstandard did outperform XZ/LZMA in terms of compression ratio, but not by much. However, the massive improvement in decompression speed is what won me over.

jzawodn 8y ago

Am I the only one getting sick of "at scale"?

erikb 8y ago
It's Enterprise slang for "we are big, so we assume everything we do works better at big scale, but please don't check if it's true, just trust us". (in this thread it might be true, usually it isn't though)

sammycdubs 8y ago

What's the Weissman score?

felixhandte 8y ago
Ha, that just came up: https://github.com/facebook/zstd/issues/1087 https://github.com/facebook/zstd/issues/1087
wolf550e 8y ago
Charles Bloom is a data compression expert working on the compression suite at RAD game tools (proprietary library for game developers that delivers better compression ratios and better decompression speed over the best available in open source, i.e. lz4, zstd and lzma): http://www.radgametools.com/oodlecompressors.htm http://www.radgametools.com/oodlecompressors.htm Here is a blog post of his about Weissman score for compressors, including his and Yann Collet's: https://cbloomrants.blogspot.com/2018/01/05-17-16-weissman-score.html https://cbloomrants.blogspot.com/2018/01/05-17-16-weissman-s...
- tpetry 8y ago
  But no third party not involved with RAD game tools has _ever_ verified these bold claims. Heck, everyone can construct benchmarks to make almost any compression algorithm outperform any other. But if it happens with real world data too is another story.
  - Legogris 8y ago
    Thing is, compression performance characteristics depend on data. You will have to run benchmarks on your own data to get real representative and conclusive results.
  - wolf550e 8y ago
    https://richg42.blogspot.com/2016/08/rads-ground-breaking-lossless.html https://richg42.blogspot.com/2016/08/rads-ground-breaking-lo...

valarauca1 8y ago

I'd really like to thank Cyan for their contributions. `zstd` and `lz4` are great. I'm pretty much exclusively using `zstd` for my tarball needs in the present day as it beats the pants off `gzip` and for plane text code (most of what I compress) it performs amazingly. (shameless self promotion) I wrote my own tar clone to make usage of it [1]. It is nice to have disk IO be the limiting factor on decompression even when you are using NVMe drives. [1] https://github.com/valarauca/car https://github.com/valarauca/car

koolba 8y ago

> And the zstd binary itself includes a dictionary trainer (zstd --train). Building a pipeline for handling compression dictionaries can therefore be reduced to being a matter of gluing these building blocks together in relatively straightforward ways. What happens if your user data trained dictionary ends up storing user data and you receive a GPDR destruction request?

bowmessage 8y ago
standard GDPR cop out disclaimer that's something about needing to keep the data for technical reasons I wish I knew more about it, but that's what I keep hearing.
- karavelov 8y ago
  The dictionary data in not personally identifiable, so no need to do anything. At least this is my understanding of GDRP, I am not a loyer.
Legogris 8y ago
It would not be covered by GDPR, just like e.g. private emails containing PII about a third party are not in scope from perspective of the e-mail provider.

karavelov 8y ago

> Two years ago, Facebook open-sourced Zstandard v1.0... Bullshit, Zstd was open-source from the very beginning, they just hired Yann and moved the project under facebook org. How do I know? I have written the JVM bindings [1] since v0.1 that are now used by Spark, Kafka, etc. EDIT: Actually, my initial bindings were against v0.0.2 [2] Kudos to FB for hiring him and helping Zstd getting production ready. This is just a PR false claim. [1] https://github.com/luben/zstd-jni https://github.com/luben/zstd-jni [2] https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da3268af6aa73dce6014298ef https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da32...

deleted 8y ago
[deleted]
hnbroseph 8y ago
on the github releases page, 1.0 mentions licensing changes of some sort. it also has pre-1.0 stuff as well.
- karavelov 8y ago
  Previously the license was BSD (as my commit above shows). v1.0 moved it to BSD+GPL2+Patent clause that is not more open-source than just BSD. After concerns from the community the patent clause was dropped, a little bit after React.
felixhandte 8y ago
Given that it was Yann himself who wrote that sentence, I think that's a needlessly uncharitable interpretation. Maybe a better wording would have been "Two years ago, we released Zstandard v1.0, an open source ...". But I don't think we anticipated anyone would read that much into it.
- karavelov 8y ago
  Yes, may be I read it the wrong way. I later noticed the authors. BTW, thank you for your part at making it the best compression library for wide variaty of cases.

golergka 8y ago

I used zstandard to compress mesages in P2P multiplayer game engine, and, taught on our real-life packets, it got us 2x-5x improvement. Awesome library, will use it in any similar project from now on.