3 ms·

What to cut total NPM traffic by 50% or more overall? Easy: Create a shared Brotli dictionary (or zstandard or whatever) based on the top NPM packages by down

What to cut total NPM traffic by 50% or more overall? Easy: Create a shared Brotli dictionary (or zstandard or whatever) based on the top NPM packages by download bandwidth and then have all npm packages compressed using it. I think this can be done server side by npmjs.org, where NPM packages are recompressed in this fashion after upload using the shared dictionary, thus it is an optional feature and fully backwards compatible. Riffing on this idea because of this new chrome feature, which does this in a flexible fashion: https://chromestatus.com/feature/5124977788977152 https://chromestatus.com/feature/5124977788977152 EDIT: This recompression of packages may be insecure as the digital signature of the package no longer aligns, but then the trick is to sign the contents of the package rather than package itself.

hoten 3y ago
Signing the contents means you must unzip it to confirm its validity, which exposes users to the latent security bugs of their unarchival program (or a zip bomb).
- bhouston 3y ago
  If recompressing it on the server breaks the chain of trust of the client provided zip, then one would need to upgrade npm on the creator's side of things so that they create both uploads themselves (legacy zip, and the shared-dictionary brotli package) and sign both themselves. Wait, does npm have digital signatures at all? I sort of assumed it did, but does it really?
IshKebab 3y ago
50%? I doubt that, unless your dictionary is literally the top NPM packages, in which case you're not saving much bandwidth by sending them all in advance... Maybe a small dictionary would save you 20% though.
- bhouston 3y ago
  It is hard to judge ahead of time and yeah it depends on the dictionary size. It is hard to argue concretely about hypotheticals. It would be a fun experiment to figure out the compression ratio of say the top 1000 packages given a shared brotli dictionary of X size. Just keep increases the dictionary size until you see diminishing returns. My estimate is based on NPM packages contain a bunch of super stereotypical files that if they are used to create the dictionary likely result in amazing compression ratios: package.json, package-lock.json, CSS, Tailwind, Bootstrap .gitignore, LICENSE, .eslint, README, React/Vue/Angular code...
Cthulhu_ 3y ago
I don't know how compression works; would a common library for Javascript save bandwidth? I'm just thinking that JS is a lot of repeated characters and keywords, and any browser or npm or node installation could be shipped with a (partial) dictionary. Although the bandwidth savings would only be the size of that dictionary. Still, I can't help but think there's ways to improve size & bandwidth usage by a lot, even besides using a different compression algorithm; a non-HTTP transfer method, for example.
- infogulch 3y ago
  The bandwidth savings would be the size of the dictionary per downloaded package.
shanemhansen 3y ago
It would be a fun test to run. But I'm not encouraged by the fact that the existing brotli dictionary already contains a bunch of javascript specific stuff: https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e brotli literally already has a tokens for function/return/throw/indexOf(/.match/.length/etc. Also verify after decompress is not without tradeoffs. On one hand we have folks like github who can't change the version of zlib because people rely on identical .tar.gz. https://news.ycombinator.com/item?id=34586917 https://news.ycombinator.com/item?id=34586917 On the other hand we have a whole lot of iffy stuff you can do to make programs decompressing content use large amounts of resources https://en.wikipedia.org/wiki/Zip_bomb https://en.wikipedia.org/wiki/Zip_bomb which makes "decompress this potentially untrusted file so that I can validate it's safe to use" hard.
- Aeolun 3y ago
  > On one hand we have folks like github who can't change the version of zlib because people rely on identical .tar.gz. People rely on the same checksums for the same files. They’re perfectly fine with changing the method for new files.
  - orf 3y ago
    The files (repo tarballs) are generated on demand, there are no “new” or “old” files. I guess you could base it on the commit time, but this is user supplied.
- bhouston 3y ago
  > brotli literally already has a tokens for function/return/throw/indexOf(/.match/.length/etc. Yeah, I see it already has a lot of JavaScript, HTML and CSS content. Interesting. I didn't realize it had an existing web-focused token library, and figured it was more like zstd, 7z and zlib, which I believe have none. I would love to do the experiment if I had time. I wonder what is the laziest way to do it?
  - anonacct37 3y ago
corbezzoli 3y ago
You say that as packages never receive updates. Unless you’re doing a lot of installs every day (let’s say on CI), you’re suggesting it’s better to forcibly download a hundred megabytes of data daily just to potentially save a couple of megabytes once a week. There’s a smarter mechanism than that: npm already caches packages locally twice, both in node_modules and in a global location. In reality you’re probably never going to install every dependency from scratch every day, so you’re already using either cache on every install. What I’d like to see instead is a pre-resolution of packages done by the registry. I have a list of 30 dependencies, please resolve the tree and send it all over instead of forcing me to do a waterfall of fetches.
- bhouston 3y ago
  I was saying a dictionary trained on the top packages but not the size of them. I believe you can specify the size of the dictionary right? So you want to figure out when you have diminishing returns for average users. I figure an effective shared dictionary is anywhere from a few 100kb to a few MB. Where between those is optimal I don’t know. I am definitely not saying download 100MB to save a few MB per user. That is just dumb. I find NPM in the last year or so is faster to resolve that yarn. Although I think bun is even faster and may have better global caching.
kevincox 3y ago
If you are going to create a custom dictionary you may as well use zstd. IIRC the main "innovation" that Brotli brings to compression is a default dictionary trained on web content. So if you are replacing this you best use the better algorithm.

hoten 3y ago

Signing the contents means you must unzip it to confirm its validity, which exposes users to the latent security bugs of their unarchival program (or a zip bomb).

bhouston 3y ago
If recompressing it on the server breaks the chain of trust of the client provided zip, then one would need to upgrade npm on the creator's side of things so that they create both uploads themselves (legacy zip, and the shared-dictionary brotli package) and sign both themselves. Wait, does npm have digital signatures at all? I sort of assumed it did, but does it really?

IshKebab 3y ago

50%? I doubt that, unless your dictionary is literally the top NPM packages, in which case you're not saving much bandwidth by sending them all in advance... Maybe a small dictionary would save you 20% though.

bhouston 3y ago
It is hard to judge ahead of time and yeah it depends on the dictionary size. It is hard to argue concretely about hypotheticals. It would be a fun experiment to figure out the compression ratio of say the top 1000 packages given a shared brotli dictionary of X size. Just keep increases the dictionary size until you see diminishing returns. My estimate is based on NPM packages contain a bunch of super stereotypical files that if they are used to create the dictionary likely result in amazing compression ratios: package.json, package-lock.json, CSS, Tailwind, Bootstrap .gitignore, LICENSE, .eslint, README, React/Vue/Angular code...

Cthulhu_ 3y ago

I don't know how compression works; would a common library for Javascript save bandwidth? I'm just thinking that JS is a lot of repeated characters and keywords, and any browser or npm or node installation could be shipped with a (partial) dictionary. Although the bandwidth savings would only be the size of that dictionary. Still, I can't help but think there's ways to improve size & bandwidth usage by a lot, even besides using a different compression algorithm; a non-HTTP transfer method, for example.

infogulch 3y ago
The bandwidth savings would be the size of the dictionary per downloaded package.

shanemhansen 3y ago

It would be a fun test to run. But I'm not encouraged by the fact that the existing brotli dictionary already contains a bunch of javascript specific stuff: https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e brotli literally already has a tokens for function/return/throw/indexOf(/.match/.length/etc. Also verify after decompress is not without tradeoffs. On one hand we have folks like github who can't change the version of zlib because people rely on identical .tar.gz. https://news.ycombinator.com/item?id=34586917 https://news.ycombinator.com/item?id=34586917 On the other hand we have a whole lot of iffy stuff you can do to make programs decompressing content use large amounts of resources https://en.wikipedia.org/wiki/Zip_bomb https://en.wikipedia.org/wiki/Zip_bomb which makes "decompress this potentially untrusted file so that I can validate it's safe to use" hard.

Aeolun 3y ago
> On one hand we have folks like github who can't change the version of zlib because people rely on identical .tar.gz. People rely on the same checksums for the same files. They’re perfectly fine with changing the method for new files.
- orf 3y ago
  The files (repo tarballs) are generated on demand, there are no “new” or “old” files. I guess you could base it on the commit time, but this is user supplied.
bhouston 3y ago
> brotli literally already has a tokens for function/return/throw/indexOf(/.match/.length/etc. Yeah, I see it already has a lot of JavaScript, HTML and CSS content. Interesting. I didn't realize it had an existing web-focused token library, and figured it was more like zstd, 7z and zlib, which I believe have none. I would love to do the experiment if I had time. I wonder what is the laziest way to do it?
- anonacct37 3y ago

corbezzoli 3y ago

You say that as packages never receive updates. Unless you’re doing a lot of installs every day (let’s say on CI), you’re suggesting it’s better to forcibly download a hundred megabytes of data daily just to potentially save a couple of megabytes once a week. There’s a smarter mechanism than that: npm already caches packages locally twice, both in node_modules and in a global location. In reality you’re probably never going to install every dependency from scratch every day, so you’re already using either cache on every install. What I’d like to see instead is a pre-resolution of packages done by the registry. I have a list of 30 dependencies, please resolve the tree and send it all over instead of forcing me to do a waterfall of fetches.

bhouston 3y ago
I was saying a dictionary trained on the top packages but not the size of them. I believe you can specify the size of the dictionary right? So you want to figure out when you have diminishing returns for average users. I figure an effective shared dictionary is anywhere from a few 100kb to a few MB. Where between those is optimal I don’t know. I am definitely not saying download 100MB to save a few MB per user. That is just dumb. I find NPM in the last year or so is faster to resolve that yarn. Although I think bun is even faster and may have better global caching.

kevincox 3y ago

If you are going to create a custom dictionary you may as well use zstd. IIRC the main "innovation" that Brotli brings to compression is a default dictionary trained on web content. So if you are replacing this you best use the better algorithm.