3 ms·
We wouldn't have to worry about over-the-wire package size if the modern DevOps approach wasn't "nuke everything, download from the Internet" every build. Back
by franga2000 2y ago
We wouldn't have to worry about over-the-wire package size if the modern DevOps approach wasn't "nuke everything, download from the Internet" every build.
Back in my Java days, most even small-time dev shops had a local Maven registry that would pass through and cache the big ones. A CI job, even if the "container" was nuked before each build, would create maybe a few kilobytes of Internet traffic, possibly none at all.
Now your average CI job spins up a fresh VM or container, pulls a Docker base image, apt installs a bunch of system dependencies, pip/npm/... installs a bunch of project dependencies, packages things up and pushes the image to the Docker registry. No Docker layer caching because it's fresh VM, no package manager caching because it's a fresh container, no object caching because...you get the idea....
Even if we accept that the benefits of the "clean slate every time" approach outweigh the gross inefficiency, why aren't we at least doing basic HTTP caching? I guess ingress is cheap and the egress on the other side is "someone else's money".
- Terr_ 2y agoYeah, I would also note that in addition to speed/transfer-costs, having an organizational package proxy is useful for reproducibility and security.
- robertlagrant 2y agoLots of places use a cache like Artifactory so they don't get slammed with costs, and are resilient to network outages and dependency builds vanishing.
- alimbada 2y agoAfter reading the article, this comment and the comment thread further down on pnpm[1], it feels to me like the NPM team are doing everyone a disservice by ignoring the inefficiencies in the packaging system. It may not be deliberate or malicious but they could easily have provided better solutions than the one proposed in the article which, in my opinion is a band-aid solution at best. The real fix would be to implement what you mention here: local registry and caching, and/or symlinking a la pnpm. [1] https://news.ycombinator.com/item?id=42841658 https://news.ycombinator.com/item?id=42841658
- selfmodruntime 2y agoIn every org I've worked with, we had a local dependency mirror in the GitOps architecture.
- Spivak 2y agoI really don't want to go back to the old world where every part of your build is secretly stateful and fails in mysterious hard to reproduce ways. You can and should have your own caching proxy for all your builds but local caches are evil.