3 ms·

For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot mo

For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.

GoblinSlayer 4y ago
So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.
- nezirus 4y ago
  If you are concerned about RAM usage, there are some trade-offs available (slightly lower compression ratio), for more effective resource usage. As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz https://github.com/wtarreau/libslz
- lifthrasiir 4y ago
  Virtually every compression format utilizing LZ77 has a way to declare the window size used and reject any archive requiring too large window. This is even the case with the traditional DEFLATE.
- twotwotwo 4y ago
  When compressing, window size is a tunable option. I think the compression level sets it implicitly, but it can be set explicitly using brotli -w or windowLog in the advanced zstd API (https://facebook.github.io/zstd/zstd_manual.html https://facebook.github.io/zstd/zstd_manual.html). Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.
- semi 4y ago
  In the age of cloud computing, compression can easily be offloaded to another layer. CDNs can do brotli compression for you for example.

GoblinSlayer 4y ago

So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.

nezirus 4y ago
If you are concerned about RAM usage, there are some trade-offs available (slightly lower compression ratio), for more effective resource usage. As used by HAProxy for gzip filter, from the Will Tarreau, creator of HAProxy: https://github.com/wtarreau/libslz https://github.com/wtarreau/libslz
lifthrasiir 4y ago
Virtually every compression format utilizing LZ77 has a way to declare the window size used and reject any archive requiring too large window. This is even the case with the traditional DEFLATE.
twotwotwo 4y ago
When compressing, window size is a tunable option. I think the compression level sets it implicitly, but it can be set explicitly using brotli -w or windowLog in the advanced zstd API (https://facebook.github.io/zstd/zstd_manual.html https://facebook.github.io/zstd/zstd_manual.html). Decompressing many streams at once is more interesting. The decompression APIs do let you specify a max history-window size, but e.g. HTTP won't let you advertise a window size limit, so all you can do is error out if the response would take too much memory to decode. You can just not advertise support for new Content-Encodings at all if it's a concern.
semi 4y ago
In the age of cloud computing, compression can easily be offloaded to another layer. CDNs can do brotli compression for you for example.