4 ms·

I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like

I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like speeds, or have very cheap compression using its fastest modes (hundreds of MB/s/core; see --fast modes at https://github.com/facebook/zstd/releases/tag/v1.3.4 https://github.com/facebook/zstd/releases/tag/v1.3.4). brotli especially helped at the high end (static stuff you'll serve many times and can spend a long time compressing); zstd can be especially helpful nearer the low end (dynamic content you'll serve a single time). On Twitter a designer of Brotli raised the question of whether there should be a standardized limit on RAM required to decode in the HTTP Content-Encoding context, since zstd supports huge window sizes (GBs). I don't feel strongly--clients can refuse to handle responses that take too much RAM to decode, servers have an incentive to serve up things their clients can handle, and there tend to be sharply diminishing returns to increasing window size--but I can imagine a limit in the MBs being a reasonable compromise between allowing high compression and not breaking on phones with a couple hundred MB RAM.

mraison 8y ago
It looks like this was filed already: https://www.iana.org/assignments/media-types/application/zstd https://www.iana.org/assignments/media-types/application/zst...
- twotwotwo 8y ago
  As best I can see that assigns a Content-Type ("MIME type"), distinct from a Content-Encoding supported by browsers. So it specifies what Content-Type header your browser should use for a .zst file, but doesn't represent a plan to let your HTML, JS, CSS etc. be delivered compressed using Zstd and transparently decompressed by the browser, as is possible with the gzip and br Content-Encodings.
  - twotwotwo 8y ago
    (But turns out pauldino's comment points out it is in the "content coding registry" at https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame...)
  - deleted 8y ago
    [deleted]
faragon 8y ago
For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/libsrt https://github.com/faragon/libsrt make ./enc -ezh <hugefile >output.zh For decompress: ./enc -dz <output.zh >hugefile.again
- twotwotwo 8y ago
  That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.
  - faragon 8y ago
    For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
    - twotwotwo 8y ago
      SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
      - the8472 8y ago
        Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?
pauldino 8y ago
Looks like zstd has been added to the HTTP Content Coding Registry, see https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame... (or section 6.2 of the linked RFC)
- twotwotwo 8y ago
  Oh, that's great. Then a matter of browser support I guess! Mozilla has a bug here (about Transfer-Encoding): https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 Couldn't find any mentioning zstd in the public Chrome, WebKit, or MS Edge issue trackers.
  - felixhandte 8y ago
    Yeah, we're just getting the ball rolling. I recently added support to HHVM[1]. Pushing for community adoption generally is definitely on our agenda, but will take some time. [1] https://github.com/facebook/hhvm/commit/d830ac5b335cea8433abcda14a044d71da3bd0dc https://github.com/facebook/hhvm/commit/d830ac5b335cea8433ab...
felixhandte 8y ago
The RFC touches on that issue: > In order to protect the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame that requests a memory size beyond the decoder's authorized range. > For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation; each decoder is free to support higher or lower limits, depending on local limitations. As a corollary, and in the absence of an explicit negotiation process between client and server, servers should probably avoid using a window larger than 8 MB.
- twotwotwo 8y ago
  Ah, great that there's an official number. And does seem like incentives are aligned for servers to try to do the reasonable thing.
deleted 8y ago
[deleted]

mraison 8y ago

It looks like this was filed already: https://www.iana.org/assignments/media-types/application/zstd https://www.iana.org/assignments/media-types/application/zst...

twotwotwo 8y ago
As best I can see that assigns a Content-Type ("MIME type"), distinct from a Content-Encoding supported by browsers. So it specifies what Content-Type header your browser should use for a .zst file, but doesn't represent a plan to let your HTML, JS, CSS etc. be delivered compressed using Zstd and transparently decompressed by the browser, as is possible with the gzip and br Content-Encodings.
- twotwotwo 8y ago
  (But turns out pauldino's comment points out it is in the "content coding registry" at https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame...)
- deleted 8y ago
  [deleted]

faragon 8y ago

For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/libsrt https://github.com/faragon/libsrt make ./enc -ezh <hugefile >output.zh For decompress: ./enc -dz <output.zh >hugefile.again

twotwotwo 8y ago
That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.
- faragon 8y ago
  For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
  - twotwotwo 8y ago
    SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
    - the8472 8y ago
      Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?

pauldino 8y ago

Looks like zstd has been added to the HTTP Content Coding Registry, see https://www.iana.org/assignments/http-parameters/http-parameters.xhtml https://www.iana.org/assignments/http-parameters/http-parame... (or section 6.2 of the linked RFC)

twotwotwo 8y ago
Oh, that's great. Then a matter of browser support I guess! Mozilla has a bug here (about Transfer-Encoding): https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 https://bugzilla.mozilla.org/show_bug.cgi?id=1477516 Couldn't find any mentioning zstd in the public Chrome, WebKit, or MS Edge issue trackers.
- felixhandte 8y ago
  Yeah, we're just getting the ball rolling. I recently added support to HHVM[1]. Pushing for community adoption generally is definitely on our agenda, but will take some time. [1] https://github.com/facebook/hhvm/commit/d830ac5b335cea8433abcda14a044d71da3bd0dc https://github.com/facebook/hhvm/commit/d830ac5b335cea8433ab...

felixhandte 8y ago

The RFC touches on that issue: > In order to protect the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame that requests a memory size beyond the decoder's authorized range. > For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation; each decoder is free to support higher or lower limits, depending on local limitations. As a corollary, and in the absence of an explicit negotiation process between client and server, servers should probably avoid using a window larger than 8 MB.

twotwotwo 8y ago
Ah, great that there's an official number. And does seem like incentives are aligned for servers to try to do the reasonable thing.

deleted 8y ago

[deleted]