8 ms·

We pin all of our npm dependencies and upgrade them via dependabot. Dependabot links to the GitHub or GitLab release for each dependency bump, and I typically s

by eemax 5y ago

We pin all of our npm dependencies and upgrade them via dependabot. Dependabot links to the GitHub or GitLab release for each dependency bump, and I typically skim / scan every single commit to each dependency. But there's no guarantee that what's on GH matches what is uploaded to npm (which is what happened in this case; there are no malicious commits). Does anyone know of a good way to verify that a npm release matches what's on GH? Version controlling the entirety of node_modules/ and running untrusted updates in a sandbox would work in theory, but in practice many packages contain minified js which makes the diffs between version bumps unreadable.

pwdisswordfish0 5y ago
Skip the nonsense and just check your dependencies in directly to your repo. The separation has no real world gains for developers and doesn't serve anyone except the host of your source repo. As it turns out most people's repo host is also the operator of the package registry they're using, so there aren't even theoretical gains for them, either. Doing it this way doesn't preclude the ability to upgrade your dependencies, it _completely_ sidesteps the intentional or unintentional desync between a dependency's source and its releases, it means people have to go out of their way to get a deployment that isn't reproducible, and in 4 years when your project has rotted and someone tries to stand it up again even if just temporarily to effect some long-term migration, then they aren't going to run into problems because the packages and package manager changed out from beneath them. I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying.
- ncallaway 5y ago
  > I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying. I don't think that's right. Just because someone denies a problem exists—a problem that you know for a fact, with 100% certainty exists—doesn't mean they're lying. It may mean you know they are wrong, but wrong != lying, and it's a good thing to keep in mind. If you have external reasons to believe that the person you're talking to should or does know better, then it's fair to say they are lying. But, in general, if you accuse someone who is simply wrong to be lying, you're going to immediately shut down any productive conversation that you could otherwise have.
- jasonpeacock 5y ago
  And what happens when you need to update those dependencies? Software is a living beast, you can't keep it alive on 4yr-old dependencies. In fact, you've cursed it with unpatched bugs and security issues. Yes, keep a separate repo, but also keep it updated. The best approach is to maintain a lag between your packages and upstream so issues like these are hopefully detected & corrected before you update.
  - pwdisswordfish0 5y ago
    > And what happens when you need to update those dependencies? Then you update them just like you do otherwise, like I already said is possible. > you can't keep it alive on 4yr-old dependencies. In fact, you've cursed it with unpatched bugs and security issues This is misdirection. No one is arguing for the bad thing you're trying to bring up. Commit your dependencies.
    - jasonpeacock 5y ago
      Let's say that the day you update your dependencies is after this malware was injected but before it was noticed. Now you have malware in your local repo :( Having a local repo does not prevent malware. Your exposure to risk is less because you update your dependencies less frequently, but the risk still exists and needs to be managed. There's no silver bullet.
      - pwdisswordfish0 5y ago
        This is more misdirection. By no means am I arguing that if you're doing a thousand stupid things and then start checking in a copy of your dependencies, that you're magically good. _Yes_ you're still gonna need to sort yourself out re the 999 other dumb things.
- lhorie 5y ago
  Committing node_modules and reproducibility are somewhat not orthogonal though. You can get reasonable degrees of reproducibility by choosing reasonable tools: Yarn lets you commit their binary and run that in the specified repo regardless of which version you have installed globally. Rush also allows you to enforce package manager versions. Bazel/rules_nodejs goes a step further and lets you pin node version per repo in addition to the package manager. Bazel+Bazelisk for version management of Bazel itself provides a very hermetic setup. Packages themselves are immutable as long as you don't blow away your lockfile. I used to occasionally run into very nasty non-reproducibility issues with ancient packages using npm shrinkwrap (or worse, nothing at all), but since npm/yarn got lockfiles, these problems largely went away. These days, the non-hermeticity stuff that really grinds my gears is the very low plumbing stuff. On Mac, Node-GYP uses xcode tooling to compile C++ modules, so stuff breaks with MacOS upgrades. I'm hoping someone can come up with some zig-based sanity here. As for committing node_modules, there are pros and cons. Google famously does this at scale and my understanding is that they had to invest in custom tooling because upgrades and auditing were a nightmare otherwise. We briefly considered it at some point at work too but the version control noise was too much. At work, we've looked into committing tarballs (we're using yarn 3 now) but that also poses some challenges (our setup isn't quite able to deal w/ a large number of large blobs, and there are legal/auditing vs git performance trade-off concerns surrounding deletion of files from version control history)
  - pwdisswordfish0 5y ago
    Ill-advised tool adoption is exactly the problem I'm aiming to get people to wake up and say "no" to. You need only one version control system, not one reliable one plus one flaky one. Use the reliable one, and stop with the buzzword bandwagon, which is going to be a completely different landscape in 4 years. > Packages themselves are immutable as long as you don't blow away your lockfile Lockfiles mean nothing if it's not my project. "I just cloned a 4 year old repo and `npm install` is failing" is a ridiculous problem to have to deal with (which to repeat, is something that happens all the time whether people are willing to acknowledge it or not). This has to be addressed by making it part of the culture, which is where me telling you to commit your dependencies comes from.
- eyelidlessness 5y ago
  Now anything with native bindings is broken if you so much as sneeze.
- hk1337 5y ago
  > Skip the nonsense and just check your dependencies in directly to your repo. Haha, no. That would increase the size of the repository greatly. Ideally, you would want a local proxy where the dependencies are downloaded and managed or tarball the node_modules and save it in some artifacts manager, server, or s3 bucket
  - Too 5y ago
    What's the problem with a big repository? The files still need to be downloaded from somewhere. It's mostly just text anyway so no big blobs which is usually what causes git to choke. For that one-off occasion when you are on 3G, have a new computer without an older clone, and need to edit files without compiling the project (which would have required npm install anyway), there is git partial clone. Does npm have a shared cache if you have several projects using the same dependencies?
    - 5e92cb50239222b 5y ago
      >Does npm have a shared cache if you have several projects using the same dependencies? pnpm does, that's why I'm using it for everything. It's saving me many gigabytes of precious SSD space. https://github.com/pnpm/pnpm https://github.com/pnpm/pnpm
- Osiris 5y ago
  People don't do this because `node_modules` can be absolutely massive (hundreds of megabytes or more), and a lot of people don't like (for various reasons) such large repositories. There is a deprecated project at my work that committed the entire yarn offline cache to the repo. At least those were gzipped, but the repo still had a copy of every version of every dependency. It isn't a good long term solution unless you really don't care at all about disk space or bandwidth (which you may or may not).
  - rsj_hn 5y ago
    A middle ground that I've seen deployed is corporate node mirrors with whitelisted modules. Then individual repos can just point to the corporate repo. Same thing for jars, python packages, etc.
  - pacifika 5y ago
    And build pipelines that fail due to the size of the repo.
- kortex 5y ago
  Sounds like a great way to end up with what $lastco called "super builds" - massive repos with ridiculous amounts of cmake code to get the thing to compile somewhat reliably. It was a rite of passage for new hires to have them spend a week just trying to get it to compile. All this does is concentrate what would be occasionally pruning and tending to dependencies to periodic massive deprecation "parties" when stuff flat out no longer works and deadlines are looming.
- seer 5y ago
  That’s the whole deal with yarn 2 isn’t it? With their plug’n’play it becomes feasible to actually vendor your npm deps, since instead of thousands upon thousands (upon thousands) of files you only check in a hundred or so zip files, which git handles much more gracefully. I was skeptical at first as it all seemed like way too much hoops to jump through, but the more I think about it the more it feels that it’s worth it.
dmitryminkovsky 5y ago
> Does anyone know of a good way to verify that a npm release matches what's on GH? I'm not aware of any way to do this, and it's a huge problem. It would be great if they introduced a Docker Hub verified/automated builds[0]-type thing for open source projects. I think that would be the only way we could be certain what we're seeing on GitHub is what we're running. Honestly it’s hard to believe we all just run unverifiable, untrustable code. At the very least NPM they could require package signing, so we'd know the package came from the developer. But really NPM needs to build the package from GitHub source. Node is not a toy anymore, and hasn't been for some time—or is it? [0] https://docs.docker.com/docker-hub/builds/ https://docs.docker.com/docker-hub/builds/
- eyelidlessness 5y ago
  This is ~solvable at a third party level. Nearly everything on NPM (the host) is MIT licensed or similar. When packages are published, run their publish lifecycle and compare to the package that’s actually published. I don’t have the resources or bandwidth to do this, but it’s pretty straightforward +- weird publishing setups. Edit: of course this doesn’t apply to private repositories but… you’re in a whole different world of trust at that point.
  - hoten 5y ago
    I started working on this exact problem a few years ago. Didn't get far, though, I think I stopped because I assumed there just wouldn't be any real interest.
  - hoten 5y ago
    I couldn't find the code, so I just started over. Haven't hosted it anywhere yet. https://github.com/connorjclark/npm-package-repro https://github.com/connorjclark/npm-package-repro
    - eyelidlessness 5y ago
      Awesome! Thank you.
brundolf 5y ago
Technically you could pin directly to a git commit instead of an NPM release
- paraknight 5y ago
  This is the right answer. Everyone else replying is patently wrong.
- hoten 5y ago
  Although npm supports lifecycle methods that run before publish / on install, many packages fail to use those correctly (or at all) yet still require a build step, so using the GH repo directly very often does not work.
eyelidlessness 5y ago
Edit: this was intended for a child comment sorry
taion 5y ago
Renovate gives you links to diffs on the published npm package: https://app.renovatebot.com/package-diff?name=hookem&from=1.0.8&to=2.0.0 https://app.renovatebot.com/package-diff?name=hookem&from=1.... It's also great at doing bulk updates, so you get a lot less spam than you do from Dependabot.

by eemax 5y ago

pwdisswordfish0 5y ago

Skip the nonsense and just check your dependencies in directly to your repo. The separation has no real world gains for developers and doesn't serve anyone except the host of your source repo. As it turns out most people's repo host is also the operator of the package registry they're using, so there aren't even theoretical gains for them, either. Doing it this way doesn't preclude the ability to upgrade your dependencies, it _completely_ sidesteps the intentional or unintentional desync between a dependency's source and its releases, it means people have to go out of their way to get a deployment that isn't reproducible, and in 4 years when your project has rotted and someone tries to stand it up again even if just temporarily to effect some long-term migration, then they aren't going to run into problems because the packages and package manager changed out from beneath them. I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying.

ncallaway 5y ago
> I run into this crap all the time to the point that people who claim it isn't a problem I know have to be lying. I don't think that's right. Just because someone denies a problem exists—a problem that you know for a fact, with 100% certainty exists—doesn't mean they're lying. It may mean you know they are wrong, but wrong != lying, and it's a good thing to keep in mind. If you have external reasons to believe that the person you're talking to should or does know better, then it's fair to say they are lying. But, in general, if you accuse someone who is simply wrong to be lying, you're going to immediately shut down any productive conversation that you could otherwise have.
jasonpeacock 5y ago
And what happens when you need to update those dependencies? Software is a living beast, you can't keep it alive on 4yr-old dependencies. In fact, you've cursed it with unpatched bugs and security issues. Yes, keep a separate repo, but also keep it updated. The best approach is to maintain a lag between your packages and upstream so issues like these are hopefully detected & corrected before you update.
- pwdisswordfish0 5y ago
  > And what happens when you need to update those dependencies? Then you update them just like you do otherwise, like I already said is possible. > you can't keep it alive on 4yr-old dependencies. In fact, you've cursed it with unpatched bugs and security issues This is misdirection. No one is arguing for the bad thing you're trying to bring up. Commit your dependencies.
  - jasonpeacock 5y ago
    Let's say that the day you update your dependencies is after this malware was injected but before it was noticed. Now you have malware in your local repo :( Having a local repo does not prevent malware. Your exposure to risk is less because you update your dependencies less frequently, but the risk still exists and needs to be managed. There's no silver bullet.
    - pwdisswordfish0 5y ago
      This is more misdirection. By no means am I arguing that if you're doing a thousand stupid things and then start checking in a copy of your dependencies, that you're magically good. _Yes_ you're still gonna need to sort yourself out re the 999 other dumb things.
lhorie 5y ago
Committing node_modules and reproducibility are somewhat not orthogonal though. You can get reasonable degrees of reproducibility by choosing reasonable tools: Yarn lets you commit their binary and run that in the specified repo regardless of which version you have installed globally. Rush also allows you to enforce package manager versions. Bazel/rules_nodejs goes a step further and lets you pin node version per repo in addition to the package manager. Bazel+Bazelisk for version management of Bazel itself provides a very hermetic setup. Packages themselves are immutable as long as you don't blow away your lockfile. I used to occasionally run into very nasty non-reproducibility issues with ancient packages using npm shrinkwrap (or worse, nothing at all), but since npm/yarn got lockfiles, these problems largely went away. These days, the non-hermeticity stuff that really grinds my gears is the very low plumbing stuff. On Mac, Node-GYP uses xcode tooling to compile C++ modules, so stuff breaks with MacOS upgrades. I'm hoping someone can come up with some zig-based sanity here. As for committing node_modules, there are pros and cons. Google famously does this at scale and my understanding is that they had to invest in custom tooling because upgrades and auditing were a nightmare otherwise. We briefly considered it at some point at work too but the version control noise was too much. At work, we've looked into committing tarballs (we're using yarn 3 now) but that also poses some challenges (our setup isn't quite able to deal w/ a large number of large blobs, and there are legal/auditing vs git performance trade-off concerns surrounding deletion of files from version control history)
- pwdisswordfish0 5y ago
  Ill-advised tool adoption is exactly the problem I'm aiming to get people to wake up and say "no" to. You need only one version control system, not one reliable one plus one flaky one. Use the reliable one, and stop with the buzzword bandwagon, which is going to be a completely different landscape in 4 years. > Packages themselves are immutable as long as you don't blow away your lockfile Lockfiles mean nothing if it's not my project. "I just cloned a 4 year old repo and `npm install` is failing" is a ridiculous problem to have to deal with (which to repeat, is something that happens all the time whether people are willing to acknowledge it or not). This has to be addressed by making it part of the culture, which is where me telling you to commit your dependencies comes from.
eyelidlessness 5y ago
Now anything with native bindings is broken if you so much as sneeze.
hk1337 5y ago
> Skip the nonsense and just check your dependencies in directly to your repo. Haha, no. That would increase the size of the repository greatly. Ideally, you would want a local proxy where the dependencies are downloaded and managed or tarball the node_modules and save it in some artifacts manager, server, or s3 bucket
- Too 5y ago
  What's the problem with a big repository? The files still need to be downloaded from somewhere. It's mostly just text anyway so no big blobs which is usually what causes git to choke. For that one-off occasion when you are on 3G, have a new computer without an older clone, and need to edit files without compiling the project (which would have required npm install anyway), there is git partial clone. Does npm have a shared cache if you have several projects using the same dependencies?
  - 5e92cb50239222b 5y ago
    >Does npm have a shared cache if you have several projects using the same dependencies? pnpm does, that's why I'm using it for everything. It's saving me many gigabytes of precious SSD space. https://github.com/pnpm/pnpm https://github.com/pnpm/pnpm
Osiris 5y ago
People don't do this because `node_modules` can be absolutely massive (hundreds of megabytes or more), and a lot of people don't like (for various reasons) such large repositories. There is a deprecated project at my work that committed the entire yarn offline cache to the repo. At least those were gzipped, but the repo still had a copy of every version of every dependency. It isn't a good long term solution unless you really don't care at all about disk space or bandwidth (which you may or may not).
- rsj_hn 5y ago
  A middle ground that I've seen deployed is corporate node mirrors with whitelisted modules. Then individual repos can just point to the corporate repo. Same thing for jars, python packages, etc.
- pacifika 5y ago
  And build pipelines that fail due to the size of the repo.
kortex 5y ago
Sounds like a great way to end up with what $lastco called "super builds" - massive repos with ridiculous amounts of cmake code to get the thing to compile somewhat reliably. It was a rite of passage for new hires to have them spend a week just trying to get it to compile. All this does is concentrate what would be occasionally pruning and tending to dependencies to periodic massive deprecation "parties" when stuff flat out no longer works and deadlines are looming.
seer 5y ago
That’s the whole deal with yarn 2 isn’t it? With their plug’n’play it becomes feasible to actually vendor your npm deps, since instead of thousands upon thousands (upon thousands) of files you only check in a hundred or so zip files, which git handles much more gracefully. I was skeptical at first as it all seemed like way too much hoops to jump through, but the more I think about it the more it feels that it’s worth it.

dmitryminkovsky 5y ago

> Does anyone know of a good way to verify that a npm release matches what's on GH? I'm not aware of any way to do this, and it's a huge problem. It would be great if they introduced a Docker Hub verified/automated builds[0]-type thing for open source projects. I think that would be the only way we could be certain what we're seeing on GitHub is what we're running. Honestly it’s hard to believe we all just run unverifiable, untrustable code. At the very least NPM they could require package signing, so we'd know the package came from the developer. But really NPM needs to build the package from GitHub source. Node is not a toy anymore, and hasn't been for some time—or is it? [0] https://docs.docker.com/docker-hub/builds/ https://docs.docker.com/docker-hub/builds/

eyelidlessness 5y ago
This is ~solvable at a third party level. Nearly everything on NPM (the host) is MIT licensed or similar. When packages are published, run their publish lifecycle and compare to the package that’s actually published. I don’t have the resources or bandwidth to do this, but it’s pretty straightforward +- weird publishing setups. Edit: of course this doesn’t apply to private repositories but… you’re in a whole different world of trust at that point.
- hoten 5y ago
  I started working on this exact problem a few years ago. Didn't get far, though, I think I stopped because I assumed there just wouldn't be any real interest.
- hoten 5y ago
  I couldn't find the code, so I just started over. Haven't hosted it anywhere yet. https://github.com/connorjclark/npm-package-repro https://github.com/connorjclark/npm-package-repro
  - eyelidlessness 5y ago
    Awesome! Thank you.

brundolf 5y ago

Technically you could pin directly to a git commit instead of an NPM release

paraknight 5y ago
This is the right answer. Everyone else replying is patently wrong.
hoten 5y ago
Although npm supports lifecycle methods that run before publish / on install, many packages fail to use those correctly (or at all) yet still require a build step, so using the GH repo directly very often does not work.

eyelidlessness 5y ago

Edit: this was intended for a child comment sorry

taion 5y ago

Renovate gives you links to diffs on the published npm package: https://app.renovatebot.com/package-diff?name=hookem&from=1.0.8&to=2.0.0 https://app.renovatebot.com/package-diff?name=hookem&from=1.... It's also great at doing bulk updates, so you get a lot less spam than you do from Dependabot.