4 ms·

I think this may not be a very high bar. zippy in Nim claims to be about 1.5x to 2.0x faster than zlib: https://github.com/guzba/zippy https://github.com/guzba

by cb321 2y ago

I think this may not be a very high bar. zippy in Nim claims to be about 1.5x to 2.0x faster than zlib: https://github.com/guzba/zippy https://github.com/guzba/zippy I think there are also faster zlib's around in C than the standard install one, such as https://github.com/ebiggers/libdeflate https://github.com/ebiggers/libdeflate (EDIT: also mentioned elsethread https://news.ycombinator.com/item?id=43381768 https://news.ycombinator.com/item?id=43381768 by mananaysiempre) zlib itself seems pretty antiquated/outdated these days, but it does remain popular, even as a basis for newer parallel-friendly formats such as https://www.htslib.org/doc/bgzip.html https://www.htslib.org/doc/bgzip.html

hinkley 2y ago
Zlib is unapologetically written to be portable rather than fast. It is absolutely no wonder that a Rust implementation would be faster. It runs on a pathetically small number of systems by contrast. This is not a dig at Rust, it’s an acknowledgement of how many systems exist out there, once you include embedded, automotive, aerospace, telecom, industrial control systems, and mainframes. Richard Hipp denounces claims that SQLite is the widest-used piece of code in the world and offers zlib as a candidate for that title, which I believe he is entirely correct about. I’ve been consciously using it for almost thirty years, and for a few years before that without knowing I was.
- maccard 2y ago
  Except this comparison isn’t against zlib, it’s against zlib-ng [0]. The readme states: > The result is a better performing and easier to maintain zlib-ng. So they’re comparing a first pass rewrite against a variation of zlib designed for performance [0] https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng
lern_too_spel 2y ago
They're comparing against zlib-ng, not zlib. zlib-ng is more than twice as fast as zlib for decompression. https://github.com/zlib-ng/zlib-ng/discussions/871 https://github.com/zlib-ng/zlib-ng/discussions/871 libdeflate is not zlib compatible. It doesn't support streaming decompression.
- cb321 2y ago
  Thanks (to all correctors). FWIW, that zlib-ng discussion page you link to has way more information about what machine the benchmarks were run on than TFA. It's also a safe bet that Google timed their chromium lib (which seems really close) on a much larger diversity of core architectures than these 3..4 guys have with zlib-rs. So, you know, very early days in terms of perf claims, IMO. Also, FWIW, that zippy Nim library has essentially zero CPU-specific optimizations that I could find. Maybe one tiny one in some checksumming bit. Optimization is specialization. So, I'd guess it's probably a little slower than zlib-ng now that this is pointed out, but as @hinkley observed, portability can also be a meaningful goal/axis.
mastax 2y ago
The benchmarks in the parent post are comparing to zlib-ng, which is substantially faster than zlib. The zippy claims are against "zlib found on a fresh Linux install" which at least for Debian is classic zlib.
JoshTriplett 2y ago
The bar here is not zlib, it's zlib-ng, which aims primarily for performance. libdeflate is an impressive library, but it doesn't help if you need to stream data rather than having it all in memory at once.

by cb321 2y ago

hinkley 2y ago

Zlib is unapologetically written to be portable rather than fast. It is absolutely no wonder that a Rust implementation would be faster. It runs on a pathetically small number of systems by contrast. This is not a dig at Rust, it’s an acknowledgement of how many systems exist out there, once you include embedded, automotive, aerospace, telecom, industrial control systems, and mainframes. Richard Hipp denounces claims that SQLite is the widest-used piece of code in the world and offers zlib as a candidate for that title, which I believe he is entirely correct about. I’ve been consciously using it for almost thirty years, and for a few years before that without knowing I was.

maccard 2y ago
Except this comparison isn’t against zlib, it’s against zlib-ng [0]. The readme states: > The result is a better performing and easier to maintain zlib-ng. So they’re comparing a first pass rewrite against a variation of zlib designed for performance [0] https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng

lern_too_spel 2y ago

They're comparing against zlib-ng, not zlib. zlib-ng is more than twice as fast as zlib for decompression. https://github.com/zlib-ng/zlib-ng/discussions/871 https://github.com/zlib-ng/zlib-ng/discussions/871 libdeflate is not zlib compatible. It doesn't support streaming decompression.

cb321 2y ago
Thanks (to all correctors). FWIW, that zlib-ng discussion page you link to has way more information about what machine the benchmarks were run on than TFA. It's also a safe bet that Google timed their chromium lib (which seems really close) on a much larger diversity of core architectures than these 3..4 guys have with zlib-rs. So, you know, very early days in terms of perf claims, IMO. Also, FWIW, that zippy Nim library has essentially zero CPU-specific optimizations that I could find. Maybe one tiny one in some checksumming bit. Optimization is specialization. So, I'd guess it's probably a little slower than zlib-ng now that this is pointed out, but as @hinkley observed, portability can also be a meaningful goal/axis.

mastax 2y ago

The benchmarks in the parent post are comparing to zlib-ng, which is substantially faster than zlib. The zippy claims are against "zlib found on a fresh Linux install" which at least for Debian is classic zlib.

JoshTriplett 2y ago

The bar here is not zlib, it's zlib-ng, which aims primarily for performance. libdeflate is an impressive library, but it doesn't help if you need to stream data rather than having it all in memory at once.