4 ms·

Who cares about how large a NPM library is? the issue is what you send to your clients, not how monstrous is what you download once on the integration server.

by kimi 3y ago

bhouston 3y ago
> Who cares about how large a NPM library is? NPMJS.org probably cares. And having smaller downloads for everyone else would speed things up a bit. BTW I saw this recently about shared brotli dictionaries for delivering JS, which is nice: https://chromestatus.com/feature/5124977788977152 https://chromestatus.com/feature/5124977788977152
- fabiospampinato 3y ago
  I mean they care to some degree, if they _really_ cared presumably everything would be compressed with zstd and served to the more modern npm-cli installations, and npm-cli would refuse to upload binaries that are not explicitly allowlisted.
hoten 3y ago
Lots. See the "npm node_modules blackhole" meme. For one practical reason, confirming the quality of the code in node_modules is so impossible a task it just isn't even attempted. So people are shipping code they have 0 knowledge of. For a paranoia-fueled reason, even devDep packages run the risk of being harmful (malware in the build process).
- kimi 3y ago
  Of all the reasons why the NPM ecosystem sucks, the compression of archives is but a side note.
Karellen 3y ago
> In real terms that means a saving of around 1MB. That doesn’t sound like much, but at 4 million weekly downloads, that would save 4TB of bandwidth per week. Yeah, who cares about 4TB/week? What is this, the '90s?!
- kimi 3y ago
  That's 6 Mbytes per second.
  - taway1237 3y ago
    For a single package. And npm hosts a few more than that. They shrinked a package size by almost 40%. No way this isn't significant improvement. Hell, at their scale 4% improvement is big.
  - dpcx 3y ago
    That's a lot of data that you're not paying for...
- kimi 3y ago
  A dedicated 10 Gigabit internet link at a data center will cost you around $1,000 per month from any of a number of providers. A 10 gigabit link can transmit 3240 Terabytes of data in one month.
bdcravens 3y ago
This assumes third-party code is never served directly from the node_modules folder (or via proxies like unpkg)
justinsaccount 3y ago
If anyone really cared, they would fix the broken ecosystem that encourages people to redownload the same packages millions of times. Picked a popular package at random, webpack. npm says version 5.88.2 released 3 months ago has 5,992,398 downloads in the last 7 days. I don't know how anyone can look at that see it as anything other than a massive failure. Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago. npm could fix this by aggressively rate limiting clients that have already downloaded the same package multiple times, but I guess as long as the vc funding is paying the bandwidth bill it's not a problem, and those "millions of downloads" make you look good.
- paulddraper 3y ago
  Elaborate, please. The vast majority of those are from CI on ephemeral cloud instances. Do you think CI should not be run? Or CI should be run, but not on ephemeral cloud instances? Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?
  - justinsaccount 3y ago
    > Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable? Yes, you should vendor external dependencies. A build should ideally not require internet access to complete. People learned nothing from leftpad.
    - paulddraper 3y ago
      > A build should ideally not require internet access to complete. You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Technically possible, but I call BS.
Too 3y ago
> download once on the integration server. Have you seen most CI systems in the wild? Majority of projects just wipe everything and do a fresh install, on every single run. Maybe it will be counter-productive to shrink the packages, since it would encourage that behavior of not caring about local cache even more.

Who cares about how large a NPM library is? the issue is what you send to your clients, not how monstrous is what you download once on the integration server.

by kimi 3y ago

Who cares about how large a NPM library is? the issue is what you send to your clients, not how monstrous is what you download once on the integration server.

bhouston 3y ago

> Who cares about how large a NPM library is? NPMJS.org probably cares. And having smaller downloads for everyone else would speed things up a bit. BTW I saw this recently about shared brotli dictionaries for delivering JS, which is nice: https://chromestatus.com/feature/5124977788977152 https://chromestatus.com/feature/5124977788977152

fabiospampinato 3y ago
I mean they care to some degree, if they _really_ cared presumably everything would be compressed with zstd and served to the more modern npm-cli installations, and npm-cli would refuse to upload binaries that are not explicitly allowlisted.

hoten 3y ago

Lots. See the "npm node_modules blackhole" meme. For one practical reason, confirming the quality of the code in node_modules is so impossible a task it just isn't even attempted. So people are shipping code they have 0 knowledge of. For a paranoia-fueled reason, even devDep packages run the risk of being harmful (malware in the build process).

kimi 3y ago
Of all the reasons why the NPM ecosystem sucks, the compression of archives is but a side note.

Karellen 3y ago

> In real terms that means a saving of around 1MB. That doesn’t sound like much, but at 4 million weekly downloads, that would save 4TB of bandwidth per week. Yeah, who cares about 4TB/week? What is this, the '90s?!

kimi 3y ago
That's 6 Mbytes per second.
- taway1237 3y ago
  For a single package. And npm hosts a few more than that. They shrinked a package size by almost 40%. No way this isn't significant improvement. Hell, at their scale 4% improvement is big.
- dpcx 3y ago
  That's a lot of data that you're not paying for...
kimi 3y ago
A dedicated 10 Gigabit internet link at a data center will cost you around $1,000 per month from any of a number of providers. A 10 gigabit link can transmit 3240 Terabytes of data in one month.

bdcravens 3y ago

This assumes third-party code is never served directly from the node_modules folder (or via proxies like unpkg)

justinsaccount 3y ago

If anyone really cared, they would fix the broken ecosystem that encourages people to redownload the same packages millions of times. Picked a popular package at random, webpack. npm says version 5.88.2 released 3 months ago has 5,992,398 downloads in the last 7 days. I don't know how anyone can look at that see it as anything other than a massive failure. Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago. npm could fix this by aggressively rate limiting clients that have already downloaded the same package multiple times, but I guess as long as the vc funding is paying the bandwidth bill it's not a problem, and those "millions of downloads" make you look good.

paulddraper 3y ago
Elaborate, please. The vast majority of those are from CI on ephemeral cloud instances. Do you think CI should not be run? Or CI should be run, but not on ephemeral cloud instances? Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?
- justinsaccount 3y ago
  > Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable? Yes, you should vendor external dependencies. A build should ideally not require internet access to complete. People learned nothing from leftpad.
  - paulddraper 3y ago
    > A build should ideally not require internet access to complete. You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Technically possible, but I call BS.

Too 3y ago

> download once on the integration server. Have you seen most CI systems in the wild? Majority of projects just wipe everything and do a fresh install, on every single run. Maybe it will be counter-productive to shrink the packages, since it would encourage that behavior of not caring about local cache even more.