4 ms·

For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/

For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory): git clone https://github.com/faragon/libsrt https://github.com/faragon/libsrt make ./enc -ezh <hugefile >output.zh For decompress: ./enc -dz <output.zh >hugefile.again

twotwotwo 8y ago
That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.
- faragon 8y ago
  For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
  - twotwotwo 8y ago
    SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
    - the8472 8y ago
      Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?
      - twotwotwo 8y ago
        I bet some unpackers grab physical memory for their sliding window up front, but the balance between memory per connection and ratio, and the question of whether any standard here ought to specify a number or we should just let servers and clients do their best to work things out, would still be a thing even if the space taken were also bounded by the download's size or stream's length.

twotwotwo 8y ago

That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.

faragon 8y ago
For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libslz/blob/master/src/zenc.c https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...
- twotwotwo 8y ago
  SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.
  - the8472 8y ago
    Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?
    - twotwotwo 8y ago
      I bet some unpackers grab physical memory for their sliding window up front, but the balance between memory per connection and ratio, and the question of whether any standard here ought to specify a number or we should just let servers and clients do their best to work things out, would still be a thing even if the space taken were also bounded by the download's size or stream's length.