4 ms·

Pulling packages down at build time seems ludicrous to me, I can understand it in a development environment, but I don't understand how "Pull packages from the

by ehnto 6y ago

Pulling packages down at build time seems ludicrous to me, I can understand it in a development environment, but I don't understand how "Pull packages from the public internet and put them into our production codebase" past any kind of robustness scrutiny. I guess it's a case of the ease of use proving too great, so convenient in fact that we just kind of swept the implications under the rug.

uncledave 6y ago
Yes. The way we develop software quite frankly scares the shit out of me on a daily basis.
- alistairSH 6y ago
  Even worse - if you consider yourself average, half the software out there was built on top of even weaker foundations. The web really is held together by duct tape and bubble gum.
- RGamma 6y ago
  Well it's mostly held together by trust and (in the commercial case) warranty. That said there's so many potential entry points for malicious actors it's not even funny anymore (esp. in desktop computing)... I try not to think about it too much and have faith in the powers that be
  - tremon 6y ago
    Can you point me to a documented instance where a warranty claim on a software product yielded a useful outcome for the claimant? The only one I can remember was against Microsoft for forcing an upgrade to Windows 10, which wasn't (IIRC) a warranty claim but a bait-and-switch issue.
    - RGamma 6y ago
      Now that you mention it, I can't remember anything off the top of my head, but it's gotta stand for something, right?
  - lazide 6y ago
    Every software package and SaaS provider always includes an explicit lack of warranty somewhere in their TOS - and if that means we’re just working off trust, we probably need to move to more ‘trust but verify’ instead of the current ‘fool me once, well just fool me as many times as you want because I won’t use something else anyway and you know it’ model we seem to be using now.
TeMPOraL 6y ago
> I can understand it in a development environment I can't. It's incredibly wasteful time and resource-wise, and ties your development process to third-party providers (and your ISP), which fall over often enough in practice. It's a good practice to have a local cache of all the third-party dependencies you use, available to both developers and CI infrastructure.
- golergka 6y ago
  > local cache For a distributed company with developers from all over the globe the "local" here doesn't really make much sense. But from my experience with NPM, you download packages on your developer machine once you set up a project, and then only when something really messes up node_modules, which happens once in three months, on average. You do re-download packages for every build in CI pipeline as you build a docker image from scratch though, and that's when NPM mirror is usually set up.
  - 076ae80a-3c97-4 6y ago
    You're lucky. I feel like I have to delete node_modules at least once a week.
- SamuelAdams 6y ago
  > It's a good practice to have a local cache of all the third-party dependencies you use, available to both developers and CI infrastructure. We have that, it's called Java and .NET, but apparently solved problems aren't interesting anymore.
- ehnto 6y ago
  Well I agree, I don't personally do it either. My stack is comprised of tools that are pretty comprehensive on their own, so they get committed to the repository. A backend framework, a SASS compiler binary and a frontend framework if needed. It all gets put in the repo and any tasks are run by a makefile.
golergka 6y ago
Migrating from public NPM to a privately-hosted, your own mirror of NPM is not a very complicated process, and if you already have a CI pipeline in place, it can be implemented completely transparently to developers. But as many other things that an organisation has to change as it grows from a single-founder startup to a real company, it's something many people just forget to do until they face the consequences.
- anchochilis 6y ago
  Mirrors are great for speed and protecting you from dependencies getting randomly deleted off the public repo, but I don't think they can protect from malicious packages. They'll just get pulled into the mirror. At my last gig (Java), developers reviewed all third-party libraries + dependencies and manually uploaded them to a private Ivy server. I don't think that could work in the Node ecosystem, where every module seems to have 100+ dependencies. EDIT: There's a real security vs accessibility trade-off here. You can't be a productive web developer, according to modern standards, and review every single transitive dependency that gets pulled into your application. And it's very inefficient to have individual developers at different orgs separately reviewing the same libraries over and over again. One would naturally turn to repository administrators to enforce stricter security standards. Maybe RubyGems could review all source code for every new version of a package and build it themselves instead of accepting uploads of pre-built artifacts. But these repositories are run by smallish groups of volunteers, and they don't have the resources to conduct those kinds of reviews. And no open-source developer wants to have to go through an App Store-like review process to upload their silly McWidget library.
  - marcus0x62 6y ago
    Right — the only way to really identify malicious packages is a line by line audit... All the way down the dependency chain. On top of that, does a typical org using these packages have developers who can conduct these audits and identify obvious back doors? What about less obvious bug doors?
  - ehnto 6y ago
    > You can't be a productive web developer, according to modern standards, and review every single transitive dependency that gets pulled into your application. Applications I encounter use a ridiculous amount of outside tooling to do relatively simple work, and that's where I see dependencies explode most often. When I work on package managed, env managed projects, every other day is a new environment issue, configuration issue or version mismatch. It's all a colossal waste of time. Had the project just chosen a handful of comprehensive tools and committed them to the repo, there would be no dark rituals of configuration and package management to perform. The code is in the repo, the code works with itself, life is good.
WrtCdEvrydy 6y ago
If you are using `npm i` instead of `npm ci`... you are also guilty of this.
- ehnto 6y ago
  The secret is to not use NPM at all. It's the worst offender for exploding dependency trees.
nijave 6y ago
Some things are like that but there is a decent amount of package managers now a days that at least pin package hashes so they'll fail if the package has been tampered with. I'm not aware of many places that audit dependencies to a greater extent than "the license is compatible and it has reasonable maintenance". Pulling packages from the internet is fine and that's how all Linux distros work but the more important thing is signature verification, imo
- HereBeBeasties 6y ago
  This really doesn't help when devs just upgrade everything. Or if they simply install the latest (p0wned) version of something. Pinning hashes really isn't the answer here.
thrower123 6y ago
99% of everyone does exactly this, though. Partly because nobody has any fucking idea what they are doing, and because this is what all the documentation everywhere tells you to do, so that's what the guy who gets tasked with setting up the CI build does...

by ehnto 6y ago

uncledave 6y ago

Yes. The way we develop software quite frankly scares the shit out of me on a daily basis.

alistairSH 6y ago
Even worse - if you consider yourself average, half the software out there was built on top of even weaker foundations. The web really is held together by duct tape and bubble gum.
RGamma 6y ago
Well it's mostly held together by trust and (in the commercial case) warranty. That said there's so many potential entry points for malicious actors it's not even funny anymore (esp. in desktop computing)... I try not to think about it too much and have faith in the powers that be
- tremon 6y ago
  Can you point me to a documented instance where a warranty claim on a software product yielded a useful outcome for the claimant? The only one I can remember was against Microsoft for forcing an upgrade to Windows 10, which wasn't (IIRC) a warranty claim but a bait-and-switch issue.
  - RGamma 6y ago
    Now that you mention it, I can't remember anything off the top of my head, but it's gotta stand for something, right?
- lazide 6y ago
  Every software package and SaaS provider always includes an explicit lack of warranty somewhere in their TOS - and if that means we’re just working off trust, we probably need to move to more ‘trust but verify’ instead of the current ‘fool me once, well just fool me as many times as you want because I won’t use something else anyway and you know it’ model we seem to be using now.

TeMPOraL 6y ago

> I can understand it in a development environment I can't. It's incredibly wasteful time and resource-wise, and ties your development process to third-party providers (and your ISP), which fall over often enough in practice. It's a good practice to have a local cache of all the third-party dependencies you use, available to both developers and CI infrastructure.

golergka 6y ago
> local cache For a distributed company with developers from all over the globe the "local" here doesn't really make much sense. But from my experience with NPM, you download packages on your developer machine once you set up a project, and then only when something really messes up node_modules, which happens once in three months, on average. You do re-download packages for every build in CI pipeline as you build a docker image from scratch though, and that's when NPM mirror is usually set up.
- 076ae80a-3c97-4 6y ago
  You're lucky. I feel like I have to delete node_modules at least once a week.
SamuelAdams 6y ago
> It's a good practice to have a local cache of all the third-party dependencies you use, available to both developers and CI infrastructure. We have that, it's called Java and .NET, but apparently solved problems aren't interesting anymore.
ehnto 6y ago
Well I agree, I don't personally do it either. My stack is comprised of tools that are pretty comprehensive on their own, so they get committed to the repository. A backend framework, a SASS compiler binary and a frontend framework if needed. It all gets put in the repo and any tasks are run by a makefile.

golergka 6y ago

Migrating from public NPM to a privately-hosted, your own mirror of NPM is not a very complicated process, and if you already have a CI pipeline in place, it can be implemented completely transparently to developers. But as many other things that an organisation has to change as it grows from a single-founder startup to a real company, it's something many people just forget to do until they face the consequences.

anchochilis 6y ago
Mirrors are great for speed and protecting you from dependencies getting randomly deleted off the public repo, but I don't think they can protect from malicious packages. They'll just get pulled into the mirror. At my last gig (Java), developers reviewed all third-party libraries + dependencies and manually uploaded them to a private Ivy server. I don't think that could work in the Node ecosystem, where every module seems to have 100+ dependencies. EDIT: There's a real security vs accessibility trade-off here. You can't be a productive web developer, according to modern standards, and review every single transitive dependency that gets pulled into your application. And it's very inefficient to have individual developers at different orgs separately reviewing the same libraries over and over again. One would naturally turn to repository administrators to enforce stricter security standards. Maybe RubyGems could review all source code for every new version of a package and build it themselves instead of accepting uploads of pre-built artifacts. But these repositories are run by smallish groups of volunteers, and they don't have the resources to conduct those kinds of reviews. And no open-source developer wants to have to go through an App Store-like review process to upload their silly McWidget library.
- marcus0x62 6y ago
  Right — the only way to really identify malicious packages is a line by line audit... All the way down the dependency chain. On top of that, does a typical org using these packages have developers who can conduct these audits and identify obvious back doors? What about less obvious bug doors?
- ehnto 6y ago
  > You can't be a productive web developer, according to modern standards, and review every single transitive dependency that gets pulled into your application. Applications I encounter use a ridiculous amount of outside tooling to do relatively simple work, and that's where I see dependencies explode most often. When I work on package managed, env managed projects, every other day is a new environment issue, configuration issue or version mismatch. It's all a colossal waste of time. Had the project just chosen a handful of comprehensive tools and committed them to the repo, there would be no dark rituals of configuration and package management to perform. The code is in the repo, the code works with itself, life is good.

WrtCdEvrydy 6y ago

If you are using `npm i` instead of `npm ci`... you are also guilty of this.

ehnto 6y ago
The secret is to not use NPM at all. It's the worst offender for exploding dependency trees.

nijave 6y ago

Some things are like that but there is a decent amount of package managers now a days that at least pin package hashes so they'll fail if the package has been tampered with. I'm not aware of many places that audit dependencies to a greater extent than "the license is compatible and it has reasonable maintenance". Pulling packages from the internet is fine and that's how all Linux distros work but the more important thing is signature verification, imo

HereBeBeasties 6y ago
This really doesn't help when devs just upgrade everything. Or if they simply install the latest (p0wned) version of something. Pinning hashes really isn't the answer here.

thrower123 6y ago

99% of everyone does exactly this, though. Partly because nobody has any fucking idea what they are doing, and because this is what all the documentation everywhere tells you to do, so that's what the guy who gets tasked with setting up the CI build does...