4 ms·

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with fa

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

bufferoverflow 2y ago
Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once. https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ There even more obscure options: https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html
vlovich123 2y ago
Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing. The package lock checksum is probably a more solvable issue with some coordination. The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
MBCook 2y ago
Thats a much higher hurdle to jump. I don’t blame the author for trying this first. If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think. So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.

bufferoverflow 2y ago

Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once. https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ There even more obscure options: https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html

vlovich123 2y ago

Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing. The package lock checksum is probably a more solvable issue with some coordination. The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).

MBCook 2y ago

Thats a much higher hurdle to jump. I don’t blame the author for trying this first. If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think. So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.