4 ms·

How are the download rates of these things so high? Are there 1000 people running 100 CI pipelines/day where downloads aren’t cached?

GenerocUsername 1y ago
Yes
kingstnap 1y ago
Yep. True webscale is astronomical levels of inefficient and wasteful usage of bandwidth and CPU.
nerdponx 1y ago
It doesn't help that container images typically require you to "opt in" to sharing a cache with the host system. Docker even has a special file mount type for it. I find that people on average don't have a lot of interest in making such things work, they see that it's not available right in front of them and shrug and say "oh well" because ultimately it doesn't hurt them much.
- phito 1y ago
  Yeah I'd say most people view it as such, until they hit (docker hub) rate limits.
- BobbyTables2 1y ago
  Local caching is also a royal pain when one has multiple CI runners, besides the hassle of Docker related configuration. Do they cache to shared storage and worry about concurrency? Do they only have local storage in various differing states?
zahlman 1y ago
It seems to happen in the Python world, too. Check out the stats for the most commonly downloaded packages from PyPI (https://pypistats.org/top https://pypistats.org/top), and also consider which packages are on the list. (Pip used to be right up there next to setuptools. It's still pretty popular — https://pypistats.org/packages/pip https://pypistats.org/packages/pip — but uv has clearly changed the equation.)
swiftcoder 1y ago
Yep, every single transitive downstream consumer pulls a fresh copy on every CI run. It's kind of ridiculous, and it's like this in every modern ecosystem. For example, this is a completely empty rust crate, which I started years ago and never released, and it's still downloaded multiple times per day... https://crates.io/crates/ruble https://crates.io/crates/ruble
- fainpul 1y ago
  I'm confused. Why did you publish this? Who "uses" it?
  - swiftcoder 1y ago
    I published it purely to placeholder the name (and then life got in the way, and I never released the actual library). Nobody uses it at all, but some CI system is still polling at regular intervals...
    - qludes 1y ago
      So someone could reserve all the pronounceable crate names?
      - swiftcoder 1y ago
        Yeah, it’s a known problem in the (non-namespaced) crates.io ecosystem. If someone else wants this name for their Bluetooth LE crate, I’ll happily hand over the keys, but obviously not every crate-squatter is so principled
    - fainpul 1y ago
      But if no other crate depends on your crate and nobody directly uses it, why would any CI system download it?
      - swiftcoder 1y ago
        I think some of the CI systems just monitor all published crates. For example, docs.io potentially pulls it to see if there is any new documentation
mrweasel 1y ago
Have you noticed how much stuff breaks when Github or NPM is down? People do not cache, they do not mirror, modern development for large parts are just people YOLO-ing it.

GenerocUsername 1y ago

Yes

kingstnap 1y ago

Yep. True webscale is astronomical levels of inefficient and wasteful usage of bandwidth and CPU.

nerdponx 1y ago

It doesn't help that container images typically require you to "opt in" to sharing a cache with the host system. Docker even has a special file mount type for it. I find that people on average don't have a lot of interest in making such things work, they see that it's not available right in front of them and shrug and say "oh well" because ultimately it doesn't hurt them much.

phito 1y ago
Yeah I'd say most people view it as such, until they hit (docker hub) rate limits.
BobbyTables2 1y ago
Local caching is also a royal pain when one has multiple CI runners, besides the hassle of Docker related configuration. Do they cache to shared storage and worry about concurrency? Do they only have local storage in various differing states?

zahlman 1y ago

It seems to happen in the Python world, too. Check out the stats for the most commonly downloaded packages from PyPI (https://pypistats.org/top https://pypistats.org/top), and also consider which packages are on the list. (Pip used to be right up there next to setuptools. It's still pretty popular — https://pypistats.org/packages/pip https://pypistats.org/packages/pip — but uv has clearly changed the equation.)

swiftcoder 1y ago

Yep, every single transitive downstream consumer pulls a fresh copy on every CI run. It's kind of ridiculous, and it's like this in every modern ecosystem. For example, this is a completely empty rust crate, which I started years ago and never released, and it's still downloaded multiple times per day... https://crates.io/crates/ruble https://crates.io/crates/ruble

fainpul 1y ago
I'm confused. Why did you publish this? Who "uses" it?
- swiftcoder 1y ago
  I published it purely to placeholder the name (and then life got in the way, and I never released the actual library). Nobody uses it at all, but some CI system is still polling at regular intervals...
  - qludes 1y ago
    So someone could reserve all the pronounceable crate names?
    - swiftcoder 1y ago
      Yeah, it’s a known problem in the (non-namespaced) crates.io ecosystem. If someone else wants this name for their Bluetooth LE crate, I’ll happily hand over the keys, but obviously not every crate-squatter is so principled
  - fainpul 1y ago
    But if no other crate depends on your crate and nobody directly uses it, why would any CI system download it?
    - swiftcoder 1y ago
      I think some of the CI systems just monitor all published crates. For example, docs.io potentially pulls it to see if there is any new documentation

mrweasel 1y ago

Have you noticed how much stuff breaks when Github or NPM is down? People do not cache, they do not mirror, modern development for large parts are just people YOLO-ing it.