8 ms·

The whole "grab the latest semver compatible version" was designed for a reason, and just pinning to a specific version would bring us to a world where very dif

The whole "grab the latest semver compatible version" was designed for a reason, and just pinning to a specific version would bring us to a world where very different articles need to be written. Specifically, when there's a vulnerability in any dependency (a thing that happens much more often than a rogue OSS dev) the upgrade process would be a nightmare for everyone involved. You have to chose one of the two evils; automatic bug/vulnerability fixes, or protection against rogue OSS devs you depend on. I'd say the former is an order of magnitude more common and important, and so the npm/JS world works this way.

wbl 5y ago
Have you used Go? Updating a module is easy and it uses exactly the solution described above.
zorr 5y ago
I wonder if both are possible with a trust system where releases can be approved or signed by community members. This way we can have automatic semver updates for releases approved by people we trust. Something like "I trust NPM so accept all releases that NPM has approved". Large companies could have their own internal approval keys etc.
- remram 5y ago
  This exists, it's called crev: https://github.com/crev-dev/crev https://github.com/crev-dev/crev It is really good, allowing for distributed reviews, not necessarily from the people authoring your dependencies. The OP suggests that if you -> A -> B, the package manager should only install versions of B vetted by A (or close); I think this doesn't scale well in practice (all the more so if the only way A can vet a new version of B is by doing a new release). Having the possibility to rely on other people to vet releases (possibly in your company) opens a lot of doors, such as the ability to not trust the author of A at all.
- boris 5y ago
  That's in a sense what Debian is doing with its experimental/unstable/testing/stable repositories. We also adopted a similar approach in build2 where we have queue/testing/stable sections of the repository. A newly submitted package version first lands in the queue where it is not yet visible to anyone. After it has been tested and reviewed, it is migrated to testing where it becomes available for testing by the larger community. After some time, if no issues are found in the package, it is migrated to stable. You as a user of build2 have a choice from which section(s) of the repository you want your dependencies.
foobiekr 5y ago
It's... almost like pulling random code from the internet is a liability and the people doing so need to eat the costs of doing so safely and not pull code that cannot be pulled safety. Like, for example, evidence that the author actually takes security issues seriously and has a formal way to announce them. The reality here is that just grabbing and running random shit on clients or servers has _always_ been unsafe. The people not pinning and mirroring have gigantic exposure. Pinning and mirroring people have some exposure. None of them should be doing what they are doing now.
- worik 5y ago
  >The reality here is that just grabbing and running random shit on clients or servers has _always_ been unsafe That is what this whole catastrophic mess has been based on from day one. Do not use NPM for anything new. Let it die a natural death.
  - foobiekr 5y ago
    It's not just NPM. It's no better to pull some github python repo into your server side unvetted and run it. The issue is that a broad swath of tech companies have thousands of instances of software that they've never even casually inspected running in user web browsers and in their back end servers, in contexts that are almost never locked down (network controls and segmentation, privilege reduction/separation, CSPs, etc.). We are one malicious party from a broad total disaster.
    - oauea 5y ago
      The difference is that your tooling doesn't suddenly decide to pull in a new version than the one you've tested with.
      - foobiekr 5y ago
        I know people who, in their jenkins jobs, just run git clones from the repos. They're not even managing versions. This is a really broad problem. Focusing on NPM, which I agree is a trashfire, is kind of missing the problem.
oauea 5y ago
Yes, and that reason is no good. There's a reason maven3 moved away from it.
luhn 5y ago
But the npm/JS doesn't work that way. NPM will by default create a lockfile that pins the dependencies. `npm install` will install dependencies from the lockfile as long as package.json hasn't been updated, or update the lockfile if it has. `npm ci` will install from the lockfile and fail hard if it can't satisfy the version constraints in package.json. imo this gives the best of both worlds: Easy, controlled updates to the latest compatible versions and stable CI/CD. The reason color.js was an issue is because people use `npm install` in their CI and either don't commit their lockfiles or have their lockfile inadvertently out of date—It's easy to do and I'm not sure how many people actually know about `npm ci`.
- judge2020 5y ago
  iirc the biggest issue with `npm ci` before was that it cleared node_modules, which used to be the primary way of caching deps in CI builds, but now most suggest caching ~/.npm[0]. 0: https://github.com/actions/cache/blob/main/examples.md#node---npm https://github.com/actions/cache/blob/main/examples.md#node-...
  - toastal 5y ago
    Aside: would be great if they actually used XDG directories despite many open bugs.
- m1keil 5y ago
  > as long as package.json hasn't been updated, or update the lockfile if it has There is a problem here: it will update the __entire__ package-lock.json file, regardless of the actual changes in packages.json. Pinning doesn't help here if colors dependency is not your direct dependency. So you can have a pinned library that doesn't pin colors.js. Now you make a change in packages.json (say adding or removing another unrelated package). You will end up with colors being bumped to latest.
  - dathinab 5y ago
    > There is a problem here: it will update the __entire__ package-lock.json file, regardless of the actual changes in packages.json. I'm so happy I don't need to fight npm anymore.
jka 5y ago
Hrm. Generally what you're saying sounds reasonable, although I wonder if there's a chance of a false dichotomy there. What about human errors introduced by non-rogue developers? (a reasonably common occurrence, probably?)
armchairhacker 5y ago
IMO what should be done is dependency pinning, but package repositories can send a "critical update" to essentially replace a pinned dependency. Maintainers have to explicitly request that a critical update be applied, and it's ultimately up to the repository owner to approve - generally these critical security fixes shouldn't happen often. This would handle both Log4Shell and colors/left-pad issues. Additionally, a repository owner could force a mandatory package update without the author's consent, in case someone hides malicious code in their package and it becomes popular. Of course the repository owner could go rogue, but I trust NPM/Maven/Cargo/OPAM maintainers much more than a random package developer.
kristjansson 5y ago
Sure, but there are points in the design space between “everyone specifies exact version requirements” and “everyone specifies lower bounds and pulls the absolute latest”.

wbl 5y ago

Have you used Go? Updating a module is easy and it uses exactly the solution described above.

zorr 5y ago

I wonder if both are possible with a trust system where releases can be approved or signed by community members. This way we can have automatic semver updates for releases approved by people we trust. Something like "I trust NPM so accept all releases that NPM has approved". Large companies could have their own internal approval keys etc.

remram 5y ago
This exists, it's called crev: https://github.com/crev-dev/crev https://github.com/crev-dev/crev It is really good, allowing for distributed reviews, not necessarily from the people authoring your dependencies. The OP suggests that if you -> A -> B, the package manager should only install versions of B vetted by A (or close); I think this doesn't scale well in practice (all the more so if the only way A can vet a new version of B is by doing a new release). Having the possibility to rely on other people to vet releases (possibly in your company) opens a lot of doors, such as the ability to not trust the author of A at all.
boris 5y ago
That's in a sense what Debian is doing with its experimental/unstable/testing/stable repositories. We also adopted a similar approach in build2 where we have queue/testing/stable sections of the repository. A newly submitted package version first lands in the queue where it is not yet visible to anyone. After it has been tested and reviewed, it is migrated to testing where it becomes available for testing by the larger community. After some time, if no issues are found in the package, it is migrated to stable. You as a user of build2 have a choice from which section(s) of the repository you want your dependencies.

foobiekr 5y ago

It's... almost like pulling random code from the internet is a liability and the people doing so need to eat the costs of doing so safely and not pull code that cannot be pulled safety. Like, for example, evidence that the author actually takes security issues seriously and has a formal way to announce them. The reality here is that just grabbing and running random shit on clients or servers has _always_ been unsafe. The people not pinning and mirroring have gigantic exposure. Pinning and mirroring people have some exposure. None of them should be doing what they are doing now.

worik 5y ago
>The reality here is that just grabbing and running random shit on clients or servers has _always_ been unsafe That is what this whole catastrophic mess has been based on from day one. Do not use NPM for anything new. Let it die a natural death.
- foobiekr 5y ago
  It's not just NPM. It's no better to pull some github python repo into your server side unvetted and run it. The issue is that a broad swath of tech companies have thousands of instances of software that they've never even casually inspected running in user web browsers and in their back end servers, in contexts that are almost never locked down (network controls and segmentation, privilege reduction/separation, CSPs, etc.). We are one malicious party from a broad total disaster.
  - oauea 5y ago
    The difference is that your tooling doesn't suddenly decide to pull in a new version than the one you've tested with.
    - foobiekr 5y ago
      I know people who, in their jenkins jobs, just run git clones from the repos. They're not even managing versions. This is a really broad problem. Focusing on NPM, which I agree is a trashfire, is kind of missing the problem.

oauea 5y ago

Yes, and that reason is no good. There's a reason maven3 moved away from it.

luhn 5y ago

But the npm/JS doesn't work that way. NPM will by default create a lockfile that pins the dependencies. `npm install` will install dependencies from the lockfile as long as package.json hasn't been updated, or update the lockfile if it has. `npm ci` will install from the lockfile and fail hard if it can't satisfy the version constraints in package.json. imo this gives the best of both worlds: Easy, controlled updates to the latest compatible versions and stable CI/CD. The reason color.js was an issue is because people use `npm install` in their CI and either don't commit their lockfiles or have their lockfile inadvertently out of date—It's easy to do and I'm not sure how many people actually know about `npm ci`.

judge2020 5y ago
iirc the biggest issue with `npm ci` before was that it cleared node_modules, which used to be the primary way of caching deps in CI builds, but now most suggest caching ~/.npm[0]. 0: https://github.com/actions/cache/blob/main/examples.md#node---npm https://github.com/actions/cache/blob/main/examples.md#node-...
- toastal 5y ago
  Aside: would be great if they actually used XDG directories despite many open bugs.
m1keil 5y ago
> as long as package.json hasn't been updated, or update the lockfile if it has There is a problem here: it will update the __entire__ package-lock.json file, regardless of the actual changes in packages.json. Pinning doesn't help here if colors dependency is not your direct dependency. So you can have a pinned library that doesn't pin colors.js. Now you make a change in packages.json (say adding or removing another unrelated package). You will end up with colors being bumped to latest.
- dathinab 5y ago
  > There is a problem here: it will update the __entire__ package-lock.json file, regardless of the actual changes in packages.json. I'm so happy I don't need to fight npm anymore.

jka 5y ago

Hrm. Generally what you're saying sounds reasonable, although I wonder if there's a chance of a false dichotomy there. What about human errors introduced by non-rogue developers? (a reasonably common occurrence, probably?)

armchairhacker 5y ago

IMO what should be done is dependency pinning, but package repositories can send a "critical update" to essentially replace a pinned dependency. Maintainers have to explicitly request that a critical update be applied, and it's ultimately up to the repository owner to approve - generally these critical security fixes shouldn't happen often. This would handle both Log4Shell and colors/left-pad issues. Additionally, a repository owner could force a mandatory package update without the author's consent, in case someone hides malicious code in their package and it becomes popular. Of course the repository owner could go rogue, but I trust NPM/Maven/Cargo/OPAM maintainers much more than a random package developer.

kristjansson 5y ago

Sure, but there are points in the design space between “everyone specifies exact version requirements” and “everyone specifies lower bounds and pulls the absolute latest”.