4 ms·

Zstandard Compression and the application/zstd Media Type

87 points by okket 8y ago

twotwotwo 8y ago
I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like speeds, or have very cheap compression using its fastest modes (hundreds of MB/s/core; see --fast modes at https://github.com/facebook/zstd/releases/tag/v1.3.4 https://github.com/facebook/zstd/releases/tag/v1.3.4). brotli especially helped at the high end (static stuff you'll serve many times and can spend a long time compressing); zstd can be especially helpful nearer the low end (dynamic content you'll serve a single time). On Twitter a designer of Brotli raised the question of whether there should be a standardized limit on RAM required to decode in the HTTP Content-Encoding context, since zstd supports huge window sizes (GBs). I don't feel strongly--clients can refuse to handle responses that take too much RAM to decode, servers have an incentive to serve up things their clients can handle, and there tend to be sharply diminishing returns to increasing window size--but I can imagine a limit in the MBs being a reasonable compromise between allowing high compression and not breaking on phones with a couple hundred MB RAM.
- mraison 8y ago
  It looks like this was filed already: https://www.iana.org/assignments/media-types/application/zstd https://www.iana.org/assignments/media-types/application/zst...
  - twotwotwo 8y ago
    As best I can see that assigns a Content-Type ("MIME type"), distinct from a Content-Encoding supported by browsers. So it specifies what Content-Type header your browser should use for a .zst file, but doesn't represent a plan to let your HTML, JS, CSS etc. be delivered compressed using Zstd and transparently decompressed by the browser, as is possible with the gzip and br Content-Encodings.
    - twotwotwo 8y ago
      (But turns out pauldino's comment points out it is in the "content coding registry" at https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame...)
    - deleted 8y ago
      [deleted]
- faragon 8y ago
  For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/libsrt https://github.com/faragon/libsrt make ./enc -ezh <hugefile >output.zh For decompress: ./enc -dz <output.zh >hugefile.again
  - twotwotwo 8y ago
    That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.
    - faragon 8y ago
      For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
      - twotwotwo 8y ago
        SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
        the8472 8y ago
        Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?
- pauldino 8y ago
  Looks like zstd has been added to the HTTP Content Coding Registry, see https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame... (or section 6.2 of the linked RFC)
  - twotwotwo 8y ago
    Oh, that's great. Then a matter of browser support I guess! Mozilla has a bug here (about Transfer-Encoding): https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 Couldn't find any mentioning zstd in the public Chrome, WebKit, or MS Edge issue trackers.
    - felixhandte 8y ago
      Yeah, we're just getting the ball rolling. I recently added support to HHVM[1]. Pushing for community adoption generally is definitely on our agenda, but will take some time. [1] https://github.com/facebook/hhvm/commit/d830ac5b335cea8433abcda14a044d71da3bd0dc https://github.com/facebook/hhvm/commit/d830ac5b335cea8433ab...
- felixhandte 8y ago
  The RFC touches on that issue: > In order to protect the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame that requests a memory size beyond the decoder's authorized range. > For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation; each decoder is free to support higher or lower limits, depending on local limitations. As a corollary, and in the absence of an explicit negotiation process between client and server, servers should probably avoid using a window larger than 8 MB.
  - twotwotwo 8y ago
    Ah, great that there's an official number. And does seem like incentives are aligned for servers to try to do the reasonable thing.
- deleted 8y ago
  [deleted]
jepler 8y ago
So can someone help me untangle how this relates to possible Zstandard patents? Back some time ago, when all public facebook git repositories had a patent "grant" document, there was some angst about what exactly this meant: https://github.com/facebook/zstd/issues/335 https://github.com/facebook/zstd/issues/335 -- this was considered "resolved" when facebook decided to distribute the software under GPLv2, though after reading the relevant sections of GPLv2 I can't say I follow why this was supposed to resolve any uncertainty around potential FB patents of zstd. Under BCP79 it would be Facebook's duty to disclose any IPR it has in zstd; there are presently no such disclosures registered: https://datatracker.ietf.org/ipr/search/?draft=&rfc=8478&submit=rfc&doctitle=&group=&holder=&iprtitle=&patent= https://datatracker.ietf.org/ipr/search/?draft=&rfc=8478&sub... As far as I can tell, this is good news for people who would like to adopt zstd!
- electrum 8y ago
  The commit that dual-licensed under BSD and GPLv2 also "removed PATENTS clause": https://github.com/facebook/zstd/commit/4f73b3b55d83b082aad180816a5d336eff33a243 https://github.com/facebook/zstd/commit/4f73b3b55d83b082aad1...
  - tensor 8y ago
    BSD doesn't have a patent grant, so doesn't this mean you are only legally free of their patent if you use the GPL code? Also, any independent implementation of the algorithm would be subject to patent fees?
    - garmaine 8y ago
      GPLv2 doesn't have patent clauses either (that was the reason for GPLv3). So I'm extra confused.

87 points by okket 8y ago

I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like speeds, or have very cheap compression using its fastest modes (hundreds of MB/s/core; see --fast modes at https://github.com/facebook/zstd/releases/tag/v1.3.4 https://github.com/facebook/zstd/releases/tag/v1.3.4). brotli especially helped at the high end (static stuff you'll serve many times and can spend a long time compressing); zstd can be especially helpful nearer the low end (dynamic content you'll serve a single time). On Twitter a designer of Brotli raised the question of whether there should be a standardized limit on RAM required to decode in the HTTP Content-Encoding context, since zstd supports huge window sizes (GBs). I don't feel strongly--clients can refuse to handle responses that take too much RAM to decode, servers have an incentive to serve up things their clients can handle, and there tend to be sharply diminishing returns to increasing window size--but I can imagine a limit in the MBs being a reasonable compromise between allowing high compression and not breaking on phones with a couple hundred MB RAM.

mraison 8y ago
It looks like this was filed already: https://www.iana.org/assignments/media-types/application/zstd https://www.iana.org/assignments/media-types/application/zst...
- twotwotwo 8y ago
  As best I can see that assigns a Content-Type ("MIME type"), distinct from a Content-Encoding supported by browsers. So it specifies what Content-Type header your browser should use for a .zst file, but doesn't represent a plan to let your HTML, JS, CSS etc. be delivered compressed using Zstd and transparently decompressed by the browser, as is possible with the gzip and br Content-Encodings.
  - twotwotwo 8y ago
    (But turns out pauldino's comment points out it is in the "content coding registry" at https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame...)
  - deleted 8y ago
    [deleted]
faragon 8y ago
For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/libsrt https://github.com/faragon/libsrt make ./enc -ezh <hugefile >output.zh For decompress: ./enc -dz <output.zh >hugefile.again
- twotwotwo 8y ago
  That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.
  - faragon 8y ago
    For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
    - twotwotwo 8y ago
      SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
      - the8472 8y ago
        Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?
pauldino 8y ago
Looks like zstd has been added to the HTTP Content Coding Registry, see https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame... (or section 6.2 of the linked RFC)
- twotwotwo 8y ago
  Oh, that's great. Then a matter of browser support I guess! Mozilla has a bug here (about Transfer-Encoding): https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 Couldn't find any mentioning zstd in the public Chrome, WebKit, or MS Edge issue trackers.
  - felixhandte 8y ago
    Yeah, we're just getting the ball rolling. I recently added support to HHVM[1]. Pushing for community adoption generally is definitely on our agenda, but will take some time. [1] https://github.com/facebook/hhvm/commit/d830ac5b335cea8433abcda14a044d71da3bd0dc https://github.com/facebook/hhvm/commit/d830ac5b335cea8433ab...
felixhandte 8y ago
The RFC touches on that issue: > In order to protect the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame that requests a memory size beyond the decoder's authorized range. > For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation; each decoder is free to support higher or lower limits, depending on local limitations. As a corollary, and in the absence of an explicit negotiation process between client and server, servers should probably avoid using a window larger than 8 MB.
- twotwotwo 8y ago
  Ah, great that there's an official number. And does seem like incentives are aligned for servers to try to do the reasonable thing.
deleted 8y ago
[deleted]

jepler 8y ago

So can someone help me untangle how this relates to possible Zstandard patents? Back some time ago, when all public facebook git repositories had a patent "grant" document, there was some angst about what exactly this meant: https://github.com/facebook/zstd/issues/335 https://github.com/facebook/zstd/issues/335 -- this was considered "resolved" when facebook decided to distribute the software under GPLv2, though after reading the relevant sections of GPLv2 I can't say I follow why this was supposed to resolve any uncertainty around potential FB patents of zstd. Under BCP79 it would be Facebook's duty to disclose any IPR it has in zstd; there are presently no such disclosures registered: https://datatracker.ietf.org/ipr/search/?draft=&rfc=8478&submit=rfc&doctitle=&group=&holder=&iprtitle=&patent= https://datatracker.ietf.org/ipr/search/?draft=&rfc=8478&sub... As far as I can tell, this is good news for people who would like to adopt zstd!

electrum 8y ago
The commit that dual-licensed under BSD and GPLv2 also "removed PATENTS clause": https://github.com/facebook/zstd/commit/4f73b3b55d83b082aad180816a5d336eff33a243 https://github.com/facebook/zstd/commit/4f73b3b55d83b082aad1...
- tensor 8y ago
  BSD doesn't have a patent grant, so doesn't this mean you are only legally free of their patent if you use the GPL code? Also, any independent implementation of the algorithm would be subject to patent fees?
  - garmaine 8y ago
    GPLv2 doesn't have patent clauses either (that was the reason for GPLv3). So I'm extra confused.