3 ms·

I'm late to the left-pad discussion. I thought it was considered a bad practice to depend on an external repo as part of your build process. At my company we us

I'm late to the left-pad discussion. I thought it was considered a bad practice to depend on an external repo as part of your build process. At my company we use Artefactory to host our maven libs. Even if one were removed maven central our builds would continue to work fine (in theory).

gefh 11y ago
Terrible, terrible practice. But I think a lot of people got caught out as left pad was usually an indirect dependency, and using npmjs in your build is the quickest way to satisfy your dependency graph.
mikeash 11y ago
I'm always surprised at how common this is. You check out some repository, and it ends up having to fetch more stuff from all over creation. Whether it's just git submodules or some more sophisticated dependency manager, it seems like an obviously bad idea. It's OK if they're pulling from other repositories you own, but requiring external repositories as part of the checkout process seems like an obvious point of failure. There's all sorts of possibilities for malicious activities, and just plain downtime will affect you too. Storage is cheap, so it seems like there ought to be no reason not to pull in external dependencies and save them locally. Dependency managers should support and encourage this rather than defaulting to just referencing some random stuff off in space.
- toyg 11y ago
  The reason is maintenance. Once you set up yet-another-repo-mirror (after your apt, rpm, nuget, pypi etc mirrors), someone needs to keep it up, back it up, secure it, refresh the packages, etc etc. It's "cloud culture": rely on some else's maintenance effort and just work on your own problems. Like all things, it has drawbacks.
  - mikeash 11y ago
    I don't buy it. You need a place to keep your own repository, and once you have that, keeping copies of other repositories is pretty much free. Keeping up, backing up, and securing N repositories is no more work than doing that for one repository. You can still outsource this, even. There's nothing wrong with using GitHub or Bitbucket or whatever for your repositories, just make sure that you're using your repositories, not relying on other people's. The only thing that potentially gets harder is refreshing the packages, and I'd argue that's actually a good thing because that's really just saying that packages don't randomly change out from under you. In any case, your package manager should make it easy to refresh them when you want to, just say "go update that dependency to the latest available from the official source" and let it do its thing. If package managers don't offer this, they need to start.
    - toyg 11y ago
      You need a place and a person who knows all this stuff (or likely more than one). That's expensive and hard, as knowledge of all these setvices has to be correctly handed over across people and over time. That's on top of org-specific processes that are likely more important and more byzantine, which means they have priority. So the day your awesome_pkg_server breaks, nobody remembers how and why it was set up, and the whole thing is scrapped. Back to square 1. This assuming you actually have one or two people with enough free time in the first place... Whereas standard "fetch package" scripts are trivial, and each individual stack is known well enough by developers working on it every day that there is zero administration and little need for extensive knowledge transfer... At least until left-pad disappears from under your feet; but that's rare enough that the trade-off is worth it overall (or people cannot even imagine it ever happening). I'm not saying this is how it should be, I'm just describing why a lot of people do what they do. Maybe the solution is an overall simplification and harmonization of all these services, so that the responsible option and the lazy option won't differ so much. You could have something like a prebuilt "repo server" image that can be downloaded and configured quickly, and which will then auto-cache all requested packages across different services. Cache invalidation would still be tricky but could be triggered selectively from an admin panel. Maybe all this should be packed into CI servers by default...

gefh 11y ago

Terrible, terrible practice. But I think a lot of people got caught out as left pad was usually an indirect dependency, and using npmjs in your build is the quickest way to satisfy your dependency graph.

mikeash 11y ago

I'm always surprised at how common this is. You check out some repository, and it ends up having to fetch more stuff from all over creation. Whether it's just git submodules or some more sophisticated dependency manager, it seems like an obviously bad idea. It's OK if they're pulling from other repositories you own, but requiring external repositories as part of the checkout process seems like an obvious point of failure. There's all sorts of possibilities for malicious activities, and just plain downtime will affect you too. Storage is cheap, so it seems like there ought to be no reason not to pull in external dependencies and save them locally. Dependency managers should support and encourage this rather than defaulting to just referencing some random stuff off in space.

toyg 11y ago
The reason is maintenance. Once you set up yet-another-repo-mirror (after your apt, rpm, nuget, pypi etc mirrors), someone needs to keep it up, back it up, secure it, refresh the packages, etc etc. It's "cloud culture": rely on some else's maintenance effort and just work on your own problems. Like all things, it has drawbacks.
- mikeash 11y ago
  I don't buy it. You need a place to keep your own repository, and once you have that, keeping copies of other repositories is pretty much free. Keeping up, backing up, and securing N repositories is no more work than doing that for one repository. You can still outsource this, even. There's nothing wrong with using GitHub or Bitbucket or whatever for your repositories, just make sure that you're using your repositories, not relying on other people's. The only thing that potentially gets harder is refreshing the packages, and I'd argue that's actually a good thing because that's really just saying that packages don't randomly change out from under you. In any case, your package manager should make it easy to refresh them when you want to, just say "go update that dependency to the latest available from the official source" and let it do its thing. If package managers don't offer this, they need to start.
  - toyg 11y ago
    You need a place and a person who knows all this stuff (or likely more than one). That's expensive and hard, as knowledge of all these setvices has to be correctly handed over across people and over time. That's on top of org-specific processes that are likely more important and more byzantine, which means they have priority. So the day your awesome_pkg_server breaks, nobody remembers how and why it was set up, and the whole thing is scrapped. Back to square 1. This assuming you actually have one or two people with enough free time in the first place... Whereas standard "fetch package" scripts are trivial, and each individual stack is known well enough by developers working on it every day that there is zero administration and little need for extensive knowledge transfer... At least until left-pad disappears from under your feet; but that's rare enough that the trade-off is worth it overall (or people cannot even imagine it ever happening). I'm not saying this is how it should be, I'm just describing why a lot of people do what they do. Maybe the solution is an overall simplification and harmonization of all these services, so that the responsible option and the lazy option won't differ so much. You could have something like a prebuilt "repo server" image that can be downloaded and configured quickly, and which will then auto-cache all requested packages across different services. Cache invalidation would still be tricky but could be triggered selectively from an admin panel. Maybe all this should be packed into CI servers by default...