7 ms·

What really sucks is people committing node_modules, that's just plain wrong.

by coding123 9y ago

bpicolo 9y ago
> that's just plain wrong there was at least one point in time where that was the recommended strategy
- aalleavitch 9y ago
  I'm not sure I could understanding the reasoning behind that. Does it have to do with dependency versions or the assumption npm might not be available or what?
  - bpicolo 9y ago
    There were/are a few factors. NPM availability is definitely one - before caching, and without the overhead of running your own npm replica. It also didn't used to have things like lock files. Vendoring gives you a deterministic build and removes availability concerns. In that aspect, it's not the worst thing ever, mostly just leads to noisy diffs (and maybe c extension issues if your team works on a variety of OSs?) This is pretty much what the golang world does (though now there are some tools that do a better job).
  - deleted 9y ago
    [deleted]
  - yoz-y 9y ago
    Both actually. Even with yarn and lock files, npm servers can (and eventually will) pull a rug from under you. I have already been in a situation where a dependency version that I was locked to was simply removed from the official registry. The right solution is to have your own registry or backup the archives of dependencies that you are using. I think it is better to commit the archives rather than the whole node_modules as it does not produce a mess.
    - yeukhon 9y ago
      Yeah running a registry is the better option, although another solution is forking / committing the dependencies in a separate repository, and use branch + shallow clone to keep the download size small.
      - yoz-y 9y ago
        What I ended up doing at work is to have a docker image of a registry with package zips committed in the repo (with a small script to publish all packages in). That way I can rebuild a running registry in a few moments.
  - deleted 9y ago
    [deleted]
  - c_shu 9y ago
    like this one? https://evertpot.com/npm-revoke-breaks-the-build/ https://evertpot.com/npm-revoke-breaks-the-build/
hinkley 9y ago
No what sucks is they made a clickbaity article by not removing checked in dependencies right at the beginning. There are even scenarios where I’ve seen Java projects check in their dependencies (in conservative industries) so this erodes the value of the numbers greatly. We have a pretty big issue with duplication of effort. I was hoping for an article that would be a wake up call but instead it’s just measurement artifacts.
- keiyakins 9y ago
  Copy-and-pasted libraries are a direct cause of duplicated effort. If you all grab it from the same repo (possibly through some sort of 'repo link' mechanism that keeps a local clone but is clearly marked as a clone, of course) then when a bug is fixed, it's fixed. If you copy and paste it, suddenly every fix has to be done once per project using the library.
  - hinkley 9y ago
    No they aren’t. They’re just a snapshot that works when the internet is broken, or when you can’t trust the internet. Nobody is supposed to modify those files. They’re just a local replica.
yeukhon 9y ago
This reminds me to hate AWS Lambda. I am not sure if anything has changed since my last use (around spring of 2017). I wrote my code in Python, and wanted to use psycopg2 to connect to my PG RDS instance. Luck out, yike! I was told to commit some psycopg2 files in my repository. I don't remember whether s3 was a feasible option, I think it wasn't; even then I would hate having a separate place for storing dependencies just because AWS couldn't solve customer problems. And no, I refuse to believe such requirement is hard to solve at all. I refuse to believe AWS doesn't create some container-like environment behind the scene for launching and running a function. See https://forums.aws.amazon.com/thread.jspa?messageID=791221&tstart=0 https://forums.aws.amazon.com/thread.jspa?messageID=791221&t...
linkregister 9y ago
... until LeftPad 2.0 [1] happens, then everyone will be mocking those who didn't commit node_modules. 1. http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
- granos 9y ago
  Seriously. I've been on a deployment that failed because China blocked access to Github for a little while over some piece of code in some repository. Make sure your dependencies are available.
- keiyakins 9y ago
  What's wrong is that the package manager and repo manager aren't talking to one another. "Keeping a copy of the dependency with your code" and "copy and pasting the code" should not be the same thing.
  - linkregister 9y ago
    I agree, ideally artifacts should be stored and cached, e.g. in a stored container or as a tarball. Other than a local npm mirror, what do you think is the best way to store artifacts for new builds? Vendoring code is not a bad solution if it permits new builds to occur even in the event of a package manager outage. I see vendoring code as the simplest solution for small companies who don't want the management overhead of running a local mirror of package managers. That said, I'm very much open to learning a new technique for handling this problem.

What really sucks is people committing node_modules, that's just plain wrong.

bpicolo 9y ago

> that's just plain wrong there was at least one point in time where that was the recommended strategy

aalleavitch 9y ago
I'm not sure I could understanding the reasoning behind that. Does it have to do with dependency versions or the assumption npm might not be available or what?
- bpicolo 9y ago
  There were/are a few factors. NPM availability is definitely one - before caching, and without the overhead of running your own npm replica. It also didn't used to have things like lock files. Vendoring gives you a deterministic build and removes availability concerns. In that aspect, it's not the worst thing ever, mostly just leads to noisy diffs (and maybe c extension issues if your team works on a variety of OSs?) This is pretty much what the golang world does (though now there are some tools that do a better job).
- deleted 9y ago
  [deleted]
- yoz-y 9y ago
  Both actually. Even with yarn and lock files, npm servers can (and eventually will) pull a rug from under you. I have already been in a situation where a dependency version that I was locked to was simply removed from the official registry. The right solution is to have your own registry or backup the archives of dependencies that you are using. I think it is better to commit the archives rather than the whole node_modules as it does not produce a mess.
  - yeukhon 9y ago
    Yeah running a registry is the better option, although another solution is forking / committing the dependencies in a separate repository, and use branch + shallow clone to keep the download size small.
    - yoz-y 9y ago
      What I ended up doing at work is to have a docker image of a registry with package zips committed in the repo (with a small script to publish all packages in). That way I can rebuild a running registry in a few moments.
- deleted 9y ago
  [deleted]
- c_shu 9y ago
  like this one? https://evertpot.com/npm-revoke-breaks-the-build/ https://evertpot.com/npm-revoke-breaks-the-build/

hinkley 9y ago

No what sucks is they made a clickbaity article by not removing checked in dependencies right at the beginning. There are even scenarios where I’ve seen Java projects check in their dependencies (in conservative industries) so this erodes the value of the numbers greatly. We have a pretty big issue with duplication of effort. I was hoping for an article that would be a wake up call but instead it’s just measurement artifacts.

keiyakins 9y ago
Copy-and-pasted libraries are a direct cause of duplicated effort. If you all grab it from the same repo (possibly through some sort of 'repo link' mechanism that keeps a local clone but is clearly marked as a clone, of course) then when a bug is fixed, it's fixed. If you copy and paste it, suddenly every fix has to be done once per project using the library.
- hinkley 9y ago
  No they aren’t. They’re just a snapshot that works when the internet is broken, or when you can’t trust the internet. Nobody is supposed to modify those files. They’re just a local replica.

yeukhon 9y ago

This reminds me to hate AWS Lambda. I am not sure if anything has changed since my last use (around spring of 2017). I wrote my code in Python, and wanted to use psycopg2 to connect to my PG RDS instance. Luck out, yike! I was told to commit some psycopg2 files in my repository. I don't remember whether s3 was a feasible option, I think it wasn't; even then I would hate having a separate place for storing dependencies just because AWS couldn't solve customer problems. And no, I refuse to believe such requirement is hard to solve at all. I refuse to believe AWS doesn't create some container-like environment behind the scene for launching and running a function. See https://forums.aws.amazon.com/thread.jspa?messageID=791221&tstart=0 https://forums.aws.amazon.com/thread.jspa?messageID=791221&t...

linkregister 9y ago

... until LeftPad 2.0 [1] happens, then everyone will be mocking those who didn't commit node_modules. 1. http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm

granos 9y ago
Seriously. I've been on a deployment that failed because China blocked access to Github for a little while over some piece of code in some repository. Make sure your dependencies are available.
keiyakins 9y ago
What's wrong is that the package manager and repo manager aren't talking to one another. "Keeping a copy of the dependency with your code" and "copy and pasting the code" should not be the same thing.
- linkregister 9y ago
  I agree, ideally artifacts should be stored and cached, e.g. in a stored container or as a tarball. Other than a local npm mirror, what do you think is the best way to store artifacts for new builds? Vendoring code is not a bad solution if it permits new builds to occur even in the event of a package manager outage. I see vendoring code as the simplest solution for small companies who don't want the management overhead of running a local mirror of package managers. That said, I'm very much open to learning a new technique for handling this problem.