3 ms·

If anyone really cared, they would fix the broken ecosystem that encourages people to redownload the same packages millions of times. Picked a popular package

If anyone really cared, they would fix the broken ecosystem that encourages people to redownload the same packages millions of times. Picked a popular package at random, webpack. npm says version 5.88.2 released 3 months ago has 5,992,398 downloads in the last 7 days. I don't know how anyone can look at that see it as anything other than a massive failure. Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago. npm could fix this by aggressively rate limiting clients that have already downloaded the same package multiple times, but I guess as long as the vc funding is paying the bandwidth bill it's not a problem, and those "millions of downloads" make you look good.

paulddraper 3y ago
Elaborate, please. The vast majority of those are from CI on ephemeral cloud instances. Do you think CI should not be run? Or CI should be run, but not on ephemeral cloud instances? Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?
- justinsaccount 3y ago
  > Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable? Yes, you should vendor external dependencies. A build should ideally not require internet access to complete. People learned nothing from leftpad.
  - paulddraper 3y ago
    > A build should ideally not require internet access to complete. You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Technically possible, but I call BS.
    - justinsaccount 3y ago
      > You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Vendored dependencies are pulled down from an internal s3 bucket (and cached locally) before the build starts, the rest of the build runs with no internet access. Look how nix does this, it's basically the same.
    - mixmastamyk 3y ago
      Woosh.
      - wierd-eye-loser 3y ago
        [dead]
      - wierd-eye-loser 3y ago
        [dead]
CharlieDigital 3y ago
> Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago. Maybe it's just me, but I've always thought it was well known best practice to cache your deps[0]. I'm pretty certain that this can be achieved with most CI/CD tools. https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows https://docs.github.com/en/actions/using-workflows/caching-d...
- justinsaccount 3y ago
  I've seen a lot of pipelines that simply don't bother. Or maybe they tried, but the caching isn't working and since the build works in the end no one notices an extra 30 seconds.

paulddraper 3y ago

Elaborate, please. The vast majority of those are from CI on ephemeral cloud instances. Do you think CI should not be run? Or CI should be run, but not on ephemeral cloud instances? Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable?

justinsaccount 3y ago
> Or CI should be run on ephemeral cloud instances, but the packages should be cached using a separate service from npmjs.com (e.g. S3)? If so, what makes this other service preferable? Yes, you should vendor external dependencies. A build should ideally not require internet access to complete. People learned nothing from leftpad.
- paulddraper 3y ago
  > A build should ideally not require internet access to complete. You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Technically possible, but I call BS.
  - justinsaccount 3y ago
    > You've got a non-internet CI with non-internet source code repository with non-internet vendored dependencies?? Vendored dependencies are pulled down from an internal s3 bucket (and cached locally) before the build starts, the rest of the build runs with no internet access. Look how nix does this, it's basically the same.
  - mixmastamyk 3y ago
    Woosh.
    - wierd-eye-loser 3y ago
      [dead]
    - wierd-eye-loser 3y ago
      [dead]

CharlieDigital 3y ago

> Fast connections and free bandwidth have caused people to completely ignore the fact that every time some CI pipeline runs, npm goes off and downloads 100MB of dependencies. Dependencies that haven't changed since the pipeline last ran 30 seconds ago. Maybe it's just me, but I've always thought it was well known best practice to cache your deps[0]. I'm pretty certain that this can be achieved with most CI/CD tools. https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows https://docs.github.com/en/actions/using-workflows/caching-d...

justinsaccount 3y ago
I've seen a lot of pipelines that simply don't bother. Or maybe they tried, but the caching isn't working and since the build works in the end no one notices an extra 30 seconds.