4 ms·

> older applications where gzip is unavoidable despite its obsolescence Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhe

> older applications where gzip is unavoidable despite its obsolescence Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhere

jeffbee 3y ago
The zlib implementation is not optimal in any way that I am aware of. You can get better compression with the same compute time, and you can get the same compression with less time, in every application I have seen, by using some other thing. I usually want to look at zstd, lz4, and brotli in addition to zlib. That said, the igzip re-implementation is really good. If you can use it, igzip moves gzip closer to the optimal frontier.
- rincebrain 3y ago
  There's also the various zlib forks by cloudflare et al, if you want to go down that road. But yes, lz4 and zstd would be what I'd recommend people use.
  - dralley 3y ago
    The code quality of some of the cloudflare changes is questionable. Optimizations aren't cleanly separated by commits, some of the licensing is dicey, etc. zlib-ng tried to merge as much as they could from the cloudflare fork without getting into licensing issues.
    - rincebrain 3y ago
      Oh, sure, it's all exciting in that respect from some of those, but I was more pointing to the collection of forks as a whole as a thing to evaluate if you have to look at zlib. I would just go for lz4 and zstd, really, at this point, but if for some reason gzip is necessary or fits your use case better, ... (Though I'd really like to hear what that is.)
- jandrese 3y ago
  The only place gzip wins is in compatibility. Everything supports gzip. zstd, lz4, brotli, and others aren't included by default in most places yet, although the situation is starting to improve. Plus, if you want super compatibility there is older-than-dirt pkzip/infozip.
- benatkin 3y ago
  it's optimal in that it's a community project and not an extremely unethical BigCorp project
  - jeffbee 3y ago
    Are you suggesting that lz4 is unethical?
    - benatkin 3y ago
      just that zstd points to Facebook
wolf550e 3y ago
It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy https://github.com/Cyan4973/FiniteStateEntropy
- klauspost 3y ago
  Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.
  - terrelln 3y ago
    Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB. We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.

> older applications where gzip is unavoidable despite its obsolescence Is gzip actually obsolete or are there just newer alternatives? gzip is still everywhere

jeffbee 3y ago

The zlib implementation is not optimal in any way that I am aware of. You can get better compression with the same compute time, and you can get the same compression with less time, in every application I have seen, by using some other thing. I usually want to look at zstd, lz4, and brotli in addition to zlib. That said, the igzip re-implementation is really good. If you can use it, igzip moves gzip closer to the optimal frontier.

rincebrain 3y ago
There's also the various zlib forks by cloudflare et al, if you want to go down that road. But yes, lz4 and zstd would be what I'd recommend people use.
- dralley 3y ago
  The code quality of some of the cloudflare changes is questionable. Optimizations aren't cleanly separated by commits, some of the licensing is dicey, etc. zlib-ng tried to merge as much as they could from the cloudflare fork without getting into licensing issues.
  - rincebrain 3y ago
    Oh, sure, it's all exciting in that respect from some of those, but I was more pointing to the collection of forks as a whole as a thing to evaluate if you have to look at zlib. I would just go for lz4 and zstd, really, at this point, but if for some reason gzip is necessary or fits your use case better, ... (Though I'd really like to hear what that is.)
jandrese 3y ago
The only place gzip wins is in compatibility. Everything supports gzip. zstd, lz4, brotli, and others aren't included by default in most places yet, although the situation is starting to improve. Plus, if you want super compatibility there is older-than-dirt pkzip/infozip.
benatkin 3y ago
it's optimal in that it's a community project and not an extremely unethical BigCorp project
- jeffbee 3y ago
  Are you suggesting that lz4 is unethical?
  - benatkin 3y ago
    just that zstd points to Facebook

wolf550e 3y ago

It's obsolete. DEFLATE format is limited to 32KB LZ window with huffman coding. Zstd can use a much larger window (8MB recommended) and a much better entropy coder: https://github.com/Cyan4973/FiniteStateEntropy https://github.com/Cyan4973/FiniteStateEntropy

klauspost 3y ago
Last I checked QAT was limited to 64KB backreferences and independent 128KB blocks. Now you know why they are comparing 16KB payloads only.
- terrelln 3y ago
  Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compression in RocksDB. We also have some half baked ideas to combine a fast SW match finder that only looks for matches >64KB away, and supplements the matches that QAT finds.