3 ms·
In parts (1) and (2) comparing the default setting of Zstd (level 3) against the default setting of Brotli (level 11) is a bit misleading. It shows Brotli compr
by terrelln 3y ago
In parts (1) and (2) comparing the default setting of Zstd (level 3) against the default setting of Brotli (level 11) is a bit misleading. It shows Brotli compressing ~30% better than Zstd, but Brotli's default level is >100x slower than Zstd's default level. Zstd level 3 is expected to run at hundreds of MB/s, and Brotli level 11 is expected to run at ~2 MB/s. The compression speed is only 30% slower because that benchmark includes the time to tar the directory, which is likely more expensive than the compression itself. As @sfink already suggested, just running lzbench on the npm-9.7.1.tar would be a better benchmark.
In part (3), because its running only on lib/npm.js which is 13KB, you are getting skewed results which aren't directly applicable to the compression of npm-9.7.1.tar. Brotli excels at compressing small Javascript files, as this is where its dictionary provides the most benefit. The benefits of the dictionary for a large tar file will be negligible.
However, in the npm-9.7.1.tar scenario we still expect Brotli level 11 to produce slightly smaller files than Zstd level 19. Likely ~5% smaller. But we do expect Zstd to provide significantly faster decompression speed.
- terrelln 3y agoThat said, this is an interesting article, and I love to see people experimenting with modern compression algorithms for package management! There are a lot of easy wins in this space.
- unilynx 3y agoI think the article points out that there are no easy wins at all. Sure, you could try to update the compression algorithm any time it gives a small, easy win (and as others have pointed out, his approach to comparing algorithms is debatable) The interesting part of the article to me is: okay, now that we have a better compression option, how do we deploy it to an existing ecosystem? And suddenly we're looking at a 4 year migration path! No easy win at all.
- tuatoru 3y ago> The compression speed is only 30% slower because that benchmark includes the time to tar the directory, which is likely more expensive than the compression itself. Amdahl's Law in action.