3 ms·

Fascinating--I wonder to what extent the performance improvements are due to the better constraints Rust hands over to LLVM for optimization (specifically think

Fascinating--I wonder to what extent the performance improvements are due to the better constraints Rust hands over to LLVM for optimization (specifically thinking `noalias` rules) vs aggressiv code optimization by the developers to be to the point where they beat the C zlib. The latter could be more difficult to achieve given how long the original zlib has been getting incrementally improved, though.

cratermoon 2y ago
Reading the linked blog post: -Cllvm-args=-enable-dfa-jump-thread option, which recovers most of the performance here. It performs a kind of jump threading for deterministic finite automata, and our decompression logic matches this pattern. https://en.wikipedia.org/wiki/Jump_threading https://en.wikipedia.org/wiki/Jump_threading
TinkersW 2y ago
zlib is really old and out of date, no real reason to use it except legacy. I think most optimization effort goes into newer compression libs like zstd.
- 0cf8612b2e1e 2y ago
  Legacy is a bit much. Deflate is probably the only supported compression format in many existing/live standards. For a greenfield project where you can control all clients, sure, use whatever, but zlib is likely going to continue to be used for decades.
  - daveguy 2y ago
    EDIT: doh! Mixed up Zstd and Zlib. Nothing to see here. (Can't delete with a reply.)
    - dralley 2y ago
      I think you've confused Zstd (which they were not discussing) for Zlib (the primary implementation of DEFLATE, which is most commonly used by Gzip) Gzip/DEFLATE is mediocre on most metrics relevant to compression but it's widely used of course. Zstd is better all-around, but much newer and thus doesn't have the same level of adoption.
- dralley 2y ago
  The comparison is against zlib-ng, not the OG zlib. So this is still very impressive. I suspect the results won't be quite as good on Aarch64 or other architectures though. zlib-ng has a pretty wide range of hand-optimized intrinsics whereas zlib-rs seems to only really have x86_64.
  - sanxiyn 2y ago
    zlib-rs does use ARM CRC32 instructions though.

cratermoon 2y ago

Reading the linked blog post: -Cllvm-args=-enable-dfa-jump-thread option, which recovers most of the performance here. It performs a kind of jump threading for deterministic finite automata, and our decompression logic matches this pattern. https://en.wikipedia.org/wiki/Jump_threading https://en.wikipedia.org/wiki/Jump_threading

TinkersW 2y ago

zlib is really old and out of date, no real reason to use it except legacy. I think most optimization effort goes into newer compression libs like zstd.

0cf8612b2e1e 2y ago
Legacy is a bit much. Deflate is probably the only supported compression format in many existing/live standards. For a greenfield project where you can control all clients, sure, use whatever, but zlib is likely going to continue to be used for decades.
- daveguy 2y ago
  EDIT: doh! Mixed up Zstd and Zlib. Nothing to see here. (Can't delete with a reply.)
  - dralley 2y ago
    I think you've confused Zstd (which they were not discussing) for Zlib (the primary implementation of DEFLATE, which is most commonly used by Gzip) Gzip/DEFLATE is mediocre on most metrics relevant to compression but it's widely used of course. Zstd is better all-around, but much newer and thus doesn't have the same level of adoption.
dralley 2y ago
The comparison is against zlib-ng, not the OG zlib. So this is still very impressive. I suspect the results won't be quite as good on Aarch64 or other architectures though. zlib-ng has a pretty wide range of hand-optimized intrinsics whereas zlib-rs seems to only really have x86_64.
- sanxiyn 2y ago
  zlib-rs does use ARM CRC32 instructions though.