4 ms·

One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap: git clone https://github.com/google/zopfli.git

One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap: git clone https://github.com/google/zopfli.git cc -O2 zopfli/src/zopfli/*.c -lm It just requires a C compiler and linker.

stabbles 2y ago
The main downside though, it's impressively slow. Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`. For example, gzipping llvm.tar (624MB) takes less than a second for me: $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar real 0m0.779s user 0m11.126s sys 0m0.460s At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular. It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
- Levitating 2y ago
  > 2700x slower That is impressively slow. In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package. Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged. > 2 gigabytes of bandwidth per year across all installations This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
  - bonzini 2y ago
    2 GB for the author's package which is neither extremely common nor large; it would be 2 TB/year just for react core.
    - Levitating 2y ago
      I am confused, how is this number calculated? Because the authors mentioned package, Helmet[1], is 103KB uncompressed and has had 132 versions in 13 years. Meaning downloading every Helmet version uncompressed would result in 132*103KB = 13.7MB. I feel like I must be missing something really obvious. Edit: Oh it's 2GB/year across all installations. [1]: https://www.npmjs.com/package/helmet?activeTab=versions https://www.npmjs.com/package/helmet?activeTab=versions

stabbles 2y ago

The main downside though, it's impressively slow. Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`. For example, gzipping llvm.tar (624MB) takes less than a second for me: $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar real 0m0.779s user 0m11.126s sys 0m0.460s At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular. It is almost 2700x slower than the state of the art for just 6.8% bytes saved.

Levitating 2y ago
> 2700x slower That is impressively slow. In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package. Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged. > 2 gigabytes of bandwidth per year across all installations This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
- bonzini 2y ago
  2 GB for the author's package which is neither extremely common nor large; it would be 2 TB/year just for react core.
  - Levitating 2y ago
    I am confused, how is this number calculated? Because the authors mentioned package, Helmet[1], is 103KB uncompressed and has had 132 versions in 13 years. Meaning downloading every Helmet version uncompressed would result in 132*103KB = 13.7MB. I feel like I must be missing something really obvious. Edit: Oh it's 2GB/year across all installations. [1]: https://www.npmjs.com/package/helmet?activeTab=versions https://www.npmjs.com/package/helmet?activeTab=versions