6 ms·

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commi

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

klodolph 5y ago
I’d say that the main link in that reasoning is that “Git can’t handle this without making the repo a gazillion GBs” which, can of course, can be solved if you weren’t using Git in the first place. Certain other SCMs, like Perforce, allow you to trim history and don’t require you to clone the whole history in the first place.
- sofixa 5y ago
  With git you can specify the clone depth and only get the latest X versions. And there are ways to trim history with external plugins ( git filter repo).
  - toxik 5y ago
    You have to remake every commit of the repository which basically means you have a new repository, and new commits. In order to do the latter thing you said, you cannot have done the former thing you said.
- avgcorrection 5y ago
  The author thinks that Git is so obiquitous that they just use “Git” as a stand-in for “VCS” throughout after the first paragraph. So the author definitely thought that Git would be suitable for this.
umvi 5y ago
At my previous company we committed a zipped node modules to git LFS so we could have easy reproducible offline builds without hosting an internal npm instance. Seemed to work well enough.
- unqueued 5y ago
  If you are going to be tracking binaries, you should take a look at git-annex. It is so much more flexible and powerful. The thing that I don't like about git-lfs is how limiting it is with backend serving, and how you essentially can't remove something from your repo history after it has been checked in.
- TAForObvReasons 5y ago
  Building an internal npm instance is not hard at all. verdaccio (https://verdaccio.org/ https://verdaccio.org/) is magical
  - manicdee 5y ago
    Building an internal repository of any kind is not hard at all for you today. For me, five years after you left the company it’s a pain the arse because all your code refers to this repository that doesn’t exist, the one with the custom packages with no source control, the dependencies which are no longer even in LTS versions of any extant OS distribution, and the Vagrantfile won’t work because it used undocumented perimeters for both Vagrant and that homebuilt hyper visor that you and your team built as a lark (that doesn’t exist outside your personal laptop). So for me, it’s all the dependencies get checked into the repository, all the tests run before we merge to master, and we do not use any custom in-house infrastructure of any kind.
    - winrid 5y ago
      Running a proxy for dependencies, whether it be NPM or maven, is pretty common. This way you can build if those services go down. Also, performance.
      - manicdee 5y ago
        Just another service that needs to be maintained but isn’t on the books as something that needs to be maintained leading to a wonderful day a couple of years in the future when a license cull of abandoned VMs means all your code stops building successfully on the same day.
unqueued 5y ago
I think that having rapid access to node_modules can be very helpful sometimes. The solution I came up with was this: https://github.com/unqueued/git-cache-tag https://github.com/unqueued/git-cache-tag It copies all untracked stuff (including node_modules) into a leaf tag. It is fairly easy to manage them, or find the latest one. And because they are leaves, they can be pruned and completely garbage collected when they aren't useful anymore. I have been burnt many times by npm, and I use this script to guarantee that I have a stash of my node_modules, while also keeping my project small. And I have diffed different snapshot tags to see which module changed that broke something. And by leaving everything in unaltered text, it exposes it to git which does a great job at compression stuff, especially highly differential revisions of my node_modules. A 500M node_modules from one of my projects only weighed about 100M extra, even with several snapshots. And I can just delete them anyway. I need to work on it a lot more, it was just a quick and dirty solution when I had to work with React Native a few years ago. It doesn't handle submodules at all, and there's plenty more I'd like to do with it.
- andreineculau 5y ago
  Funny but I've also done this. Thanks for sharing! Depending on the context, if you don't want this in git history, and want to handle git submodules, there's also git-archive-all https://github.com/roehling/git-archive-all https://github.com/roehling/git-archive-all (if you like shell scripts, it is using bats for testing - it was the first time I heard of it)
andrewmcwatters 5y ago
Reread what you just wrote for a moment and reflect on that. Also, you clearly have not written software using Node.js on a long enough time horizon. Pinned versions don't mean anything when sub-dependencies can have transient versioning resolution occur. The reality is that unless you can fully byte-for-byte assure what you have deployed today is what you can retrieve from an old tag, let's say weeks, months, or years from now, you don't have a replicable build. Most people will never need to do this, sure, but serious operations who will choose Node.js to build some software and then plan to walk away from it later should not only commit their node_modules directory, but also keep a copy of the designated Node.js engine version as well. It's not likely you'll need a backup of an LTS version when you can just go retrieve it, but that's not the point: You will encounter a scenario in your professional career where retrieval is not an option for some piece of software. Edit: There are industries where committing prebuilts is normal and has absolute strengths, and having experienced it myself, it certainly is desirable sometimes.
- framecowbird 5y ago
  > Reread what you just wrote for a moment and reflect on that. This comes across as very patronising. I would assume the author is perfectly aware of what they just wrote.
  - manicdee 5y ago
    I guarantee they are not aware of the ramifications of what they wrote. They live in the now, following the contemporary paradigm. I live in the five years future where the propeller head rock star programmer has moved to greener pastures (the ones where he doesn’t have to write project planning documents of any kind). They are different worlds.
    - 0des 5y ago
      Hey it's me, 10-15yrs in the future guy. Pass me a stack of punch cards and let me babble a little bit about legacy code. I found some dusty bourbon in rockstar guy's old desk. Don't let it get to you, I'll happily cash checks to work on whatever legacy spaghetti tech is in play. Hours are hours, dollars are dollars, and as long as I'm maintaining a happy ratio of those two, I don't mind what code I'm working on. I'll sharpen pencils and sweep floors for 30 hours a week, I'll even listen to your life's struggles if that's what you want me to do for that money. Don't let me give the impression that this is because all I think about is money or that I don't care, it's quite the contrary. I think of myself as a developer, I like to think I do a good job, when I get the opportunity to straighten the edges of a sagging beam or create a structure from scratch I take pride in that, as it is my purpose. There's no point in getting worked up about the practices of my peers because that does not serve me. This is my craft, and the person who's creations I am now steward of was also a craftsman, who had different experiences, motives, and contexts that led them to expressing their intent through the code now entrusted to me. Imagine a television show, or movie franchise; it may have different writers over time commanding the dialogue of the main character, developing their mannerisms, polishing their pearl. These businesses and legacy products we work on are just like those characters who get passed on to new writers. Think of yourself as one of that team, carrying on a legacy, adding your flair and support. Never stop working on your pearl and use every project as an opportunity to fulfill your own desires and express yourself, while honing your skill.
shric 5y ago
I remember an article a few years ago about Google's 86TB google3 source code repo. Someone on HN asked "I know Google is big but how on earth do they have 86TB?". Someone then quipped "someone accidentally checked in node_modules"
- batoure 5y ago
  Basically came here to say this. The whole article felt very silly until I was like “oh wait OP stated early on he works at google… yeah seems about right ~closes tab~”
  - batoure 5y ago
    Insomnia lead me to go dig into the repo… the average age of 98% of files in node_modules is 10 months old, attached to the commit when most of these files were added to the repo in the first place… so the entire argument is predicated on the changes to 2% of the dependencies
locallost 5y ago
Whether he's right or wrong can be debated, but reducing the argument to "follow industry practice" is a perfect example of cargo culting. Industry practice needs to be based on something today, not something from ten years ago that might or might not be valid anymore. He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling - e.g. we already want reproducible builds and use package-lock, but why not skip this step altogether? Why not skip setting up cache on your CI if you don't need to? What if knowing the details of your package manager and CI is useless because there are simpler ways of doing things. I'm not that convinced that a few commits would create a gazillion GB repo, so his arguments seem stronger. But "not best practice", qed.
- fivea 5y ago
  > He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling (...) Are they, really? The less debatable point is arguing that making CICD pipelines slightly faster, but this feels like an appeal to microoptimization. Any free tier CICD system out there let's you do a single npm install and move these dependencies as far as you'd like into the pipeline as artifacts. Is a git checkout really faster that a npm install? For example, is adding a npm dependency really invisible if you already track package.json and even package-lock.json? Those files show up in diffs, and it's hard to miss them. Also, if the goal is to get replicated builds, isn't this handled by pinning versions and tracking package-lock? The left_pad example is particularly ridiculous as I highly doubt that a company like Google, like any company that cares about auditing and vending dependencies, does not run its own npm proxy with cherry-picked packages.
  - xtian 5y ago
    The fact that there’s no mention of the `npm ci` command (or yarn or pnpm) makes me wonder how deeply this problem was investigated before using it as a justification for this hacky workaround. One wonders why they aren’t checking in the binaries for their database and language runtimes. Surely this would save crucial seconds in project setup.
danShumway 5y ago
This is a fun shift in culture because for a long time checking in node_modules was the official advice of the early Node documentation unless you were building a library for npm. It's already been long enough since that time that people seem to have forgotten that it even existed. Vendoring dependencies is one of those things where every once and a while a language will reintroduce the concept, and it always seems to catch people off guard. Go is a good example, although it seems to have varying advice about whether vendored dependencies should be checked in to version control. That might not be surprising considering that Go is also coming out of Google, just like this article. It feels a little bit weird to say that this is just "industry practice" when you have the Chrome DevTools team telling you they don't do it, but :shrug:. Google does tend to be a bit of a rarity in how it treats monorepos. I'm just always interested to see how opinions on this have evolved; it's rare for me to see analysis that says, "we used to do this, and here's why we found out that it didn't work." Usually the opinions end up seeming more universalist, like the very idea of vendoring dependencies is somehow weird and unexpected, and not something that the industry was largely on board with for a decent amount of time.
StreamBright 5y ago
Lets address some points instead of cargo culting. How would you go about the left pad issue?
bennyp101 5y ago
I've just checked a largish repo, and the node_modules folder is just under 500mb - I could check that in, and be done with it. Updates aren't constant, so it would be every now and then. That's really not a lot for knowing the code that is being used in your codebase hasn't changed, and the bonus of having everything available should the registry go down, or something. So I don't think it's 'horrible advice' - it's do what suits your needs best. Some people want to have everything they need to build their application in their control, on the off chance everything hits the fan. (Also, this: "Don't break the industry practice and check-in your node_modules" - does not necessarily mean it is the best way, it just happens to be the advice from the start.
caymanjim 5y ago
You're drastically overestimating the storage requirements. Time is far, far more valuable than storage savings. The point about CI builds running faster is enough to sell the idea all by itself. Oddly not mentioned in the article is that it allows you to build when npmjs.org is down or unreachable, which happens often enough to be frustrating, and if it happens when you're trying to deal with an emergency, it's downright infuriating.
- stavros 5y ago
  We cache the directory on CI and get the same benefits without needing to make our repo massive.
- mohanmcgeek 5y ago
  We run an npm server that proxies request to npmjs.org and caches what it got.
- Goz3rr 5y ago
  Git is good at storing text diffs, any binary file in your node_modules (images, natives etc) are permanently stored in your repo, including old versions or deleted files. I've seen times where this had meaningful impact on both disk use and the speed of Git itself

klodolph 5y ago

I’d say that the main link in that reasoning is that “Git can’t handle this without making the repo a gazillion GBs” which, can of course, can be solved if you weren’t using Git in the first place. Certain other SCMs, like Perforce, allow you to trim history and don’t require you to clone the whole history in the first place.

sofixa 5y ago
With git you can specify the clone depth and only get the latest X versions. And there are ways to trim history with external plugins ( git filter repo).
- toxik 5y ago
  You have to remake every commit of the repository which basically means you have a new repository, and new commits. In order to do the latter thing you said, you cannot have done the former thing you said.
avgcorrection 5y ago
The author thinks that Git is so obiquitous that they just use “Git” as a stand-in for “VCS” throughout after the first paragraph. So the author definitely thought that Git would be suitable for this.

umvi 5y ago

At my previous company we committed a zipped node modules to git LFS so we could have easy reproducible offline builds without hosting an internal npm instance. Seemed to work well enough.

unqueued 5y ago
If you are going to be tracking binaries, you should take a look at git-annex. It is so much more flexible and powerful. The thing that I don't like about git-lfs is how limiting it is with backend serving, and how you essentially can't remove something from your repo history after it has been checked in.
TAForObvReasons 5y ago
Building an internal npm instance is not hard at all. verdaccio (https://verdaccio.org/ https://verdaccio.org/) is magical
- manicdee 5y ago
  Building an internal repository of any kind is not hard at all for you today. For me, five years after you left the company it’s a pain the arse because all your code refers to this repository that doesn’t exist, the one with the custom packages with no source control, the dependencies which are no longer even in LTS versions of any extant OS distribution, and the Vagrantfile won’t work because it used undocumented perimeters for both Vagrant and that homebuilt hyper visor that you and your team built as a lark (that doesn’t exist outside your personal laptop). So for me, it’s all the dependencies get checked into the repository, all the tests run before we merge to master, and we do not use any custom in-house infrastructure of any kind.
  - winrid 5y ago
    Running a proxy for dependencies, whether it be NPM or maven, is pretty common. This way you can build if those services go down. Also, performance.
    - manicdee 5y ago
      Just another service that needs to be maintained but isn’t on the books as something that needs to be maintained leading to a wonderful day a couple of years in the future when a license cull of abandoned VMs means all your code stops building successfully on the same day.

unqueued 5y ago

I think that having rapid access to node_modules can be very helpful sometimes. The solution I came up with was this: https://github.com/unqueued/git-cache-tag https://github.com/unqueued/git-cache-tag It copies all untracked stuff (including node_modules) into a leaf tag. It is fairly easy to manage them, or find the latest one. And because they are leaves, they can be pruned and completely garbage collected when they aren't useful anymore. I have been burnt many times by npm, and I use this script to guarantee that I have a stash of my node_modules, while also keeping my project small. And I have diffed different snapshot tags to see which module changed that broke something. And by leaving everything in unaltered text, it exposes it to git which does a great job at compression stuff, especially highly differential revisions of my node_modules. A 500M node_modules from one of my projects only weighed about 100M extra, even with several snapshots. And I can just delete them anyway. I need to work on it a lot more, it was just a quick and dirty solution when I had to work with React Native a few years ago. It doesn't handle submodules at all, and there's plenty more I'd like to do with it.

andreineculau 5y ago
Funny but I've also done this. Thanks for sharing! Depending on the context, if you don't want this in git history, and want to handle git submodules, there's also git-archive-all https://github.com/roehling/git-archive-all https://github.com/roehling/git-archive-all (if you like shell scripts, it is using bats for testing - it was the first time I heard of it)

andrewmcwatters 5y ago

Reread what you just wrote for a moment and reflect on that. Also, you clearly have not written software using Node.js on a long enough time horizon. Pinned versions don't mean anything when sub-dependencies can have transient versioning resolution occur. The reality is that unless you can fully byte-for-byte assure what you have deployed today is what you can retrieve from an old tag, let's say weeks, months, or years from now, you don't have a replicable build. Most people will never need to do this, sure, but serious operations who will choose Node.js to build some software and then plan to walk away from it later should not only commit their node_modules directory, but also keep a copy of the designated Node.js engine version as well. It's not likely you'll need a backup of an LTS version when you can just go retrieve it, but that's not the point: You will encounter a scenario in your professional career where retrieval is not an option for some piece of software. Edit: There are industries where committing prebuilts is normal and has absolute strengths, and having experienced it myself, it certainly is desirable sometimes.

framecowbird 5y ago
> Reread what you just wrote for a moment and reflect on that. This comes across as very patronising. I would assume the author is perfectly aware of what they just wrote.
- manicdee 5y ago
  I guarantee they are not aware of the ramifications of what they wrote. They live in the now, following the contemporary paradigm. I live in the five years future where the propeller head rock star programmer has moved to greener pastures (the ones where he doesn’t have to write project planning documents of any kind). They are different worlds.
  - 0des 5y ago
    Hey it's me, 10-15yrs in the future guy. Pass me a stack of punch cards and let me babble a little bit about legacy code. I found some dusty bourbon in rockstar guy's old desk. Don't let it get to you, I'll happily cash checks to work on whatever legacy spaghetti tech is in play. Hours are hours, dollars are dollars, and as long as I'm maintaining a happy ratio of those two, I don't mind what code I'm working on. I'll sharpen pencils and sweep floors for 30 hours a week, I'll even listen to your life's struggles if that's what you want me to do for that money. Don't let me give the impression that this is because all I think about is money or that I don't care, it's quite the contrary. I think of myself as a developer, I like to think I do a good job, when I get the opportunity to straighten the edges of a sagging beam or create a structure from scratch I take pride in that, as it is my purpose. There's no point in getting worked up about the practices of my peers because that does not serve me. This is my craft, and the person who's creations I am now steward of was also a craftsman, who had different experiences, motives, and contexts that led them to expressing their intent through the code now entrusted to me. Imagine a television show, or movie franchise; it may have different writers over time commanding the dialogue of the main character, developing their mannerisms, polishing their pearl. These businesses and legacy products we work on are just like those characters who get passed on to new writers. Think of yourself as one of that team, carrying on a legacy, adding your flair and support. Never stop working on your pearl and use every project as an opportunity to fulfill your own desires and express yourself, while honing your skill.

shric 5y ago

I remember an article a few years ago about Google's 86TB google3 source code repo. Someone on HN asked "I know Google is big but how on earth do they have 86TB?". Someone then quipped "someone accidentally checked in node_modules"

batoure 5y ago
Basically came here to say this. The whole article felt very silly until I was like “oh wait OP stated early on he works at google… yeah seems about right ~closes tab~”
- batoure 5y ago
  Insomnia lead me to go dig into the repo… the average age of 98% of files in node_modules is 10 months old, attached to the commit when most of these files were added to the repo in the first place… so the entire argument is predicated on the changes to 2% of the dependencies

locallost 5y ago

Whether he's right or wrong can be debated, but reducing the argument to "follow industry practice" is a perfect example of cargo culting. Industry practice needs to be based on something today, not something from ten years ago that might or might not be valid anymore. He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling - e.g. we already want reproducible builds and use package-lock, but why not skip this step altogether? Why not skip setting up cache on your CI if you don't need to? What if knowing the details of your package manager and CI is useless because there are simpler ways of doing things. I'm not that convinced that a few commits would create a gazillion GB repo, so his arguments seem stronger. But "not best practice", qed.

fivea 5y ago
> He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling (...) Are they, really? The less debatable point is arguing that making CICD pipelines slightly faster, but this feels like an appeal to microoptimization. Any free tier CICD system out there let's you do a single npm install and move these dependencies as far as you'd like into the pipeline as artifacts. Is a git checkout really faster that a npm install? For example, is adding a npm dependency really invisible if you already track package.json and even package-lock.json? Those files show up in diffs, and it's hard to miss them. Also, if the goal is to get replicated builds, isn't this handled by pinning versions and tracking package-lock? The left_pad example is particularly ridiculous as I highly doubt that a company like Google, like any company that cares about auditing and vending dependencies, does not run its own npm proxy with cherry-picked packages.
- xtian 5y ago
  The fact that there’s no mention of the `npm ci` command (or yarn or pnpm) makes me wonder how deeply this problem was investigated before using it as a justification for this hacky workaround. One wonders why they aren’t checking in the binaries for their database and language runtimes. Surely this would save crucial seconds in project setup.

danShumway 5y ago

This is a fun shift in culture because for a long time checking in node_modules was the official advice of the early Node documentation unless you were building a library for npm. It's already been long enough since that time that people seem to have forgotten that it even existed. Vendoring dependencies is one of those things where every once and a while a language will reintroduce the concept, and it always seems to catch people off guard. Go is a good example, although it seems to have varying advice about whether vendored dependencies should be checked in to version control. That might not be surprising considering that Go is also coming out of Google, just like this article. It feels a little bit weird to say that this is just "industry practice" when you have the Chrome DevTools team telling you they don't do it, but :shrug:. Google does tend to be a bit of a rarity in how it treats monorepos. I'm just always interested to see how opinions on this have evolved; it's rare for me to see analysis that says, "we used to do this, and here's why we found out that it didn't work." Usually the opinions end up seeming more universalist, like the very idea of vendoring dependencies is somehow weird and unexpected, and not something that the industry was largely on board with for a decent amount of time.

StreamBright 5y ago

Lets address some points instead of cargo culting. How would you go about the left pad issue?

bennyp101 5y ago

I've just checked a largish repo, and the node_modules folder is just under 500mb - I could check that in, and be done with it. Updates aren't constant, so it would be every now and then. That's really not a lot for knowing the code that is being used in your codebase hasn't changed, and the bonus of having everything available should the registry go down, or something. So I don't think it's 'horrible advice' - it's do what suits your needs best. Some people want to have everything they need to build their application in their control, on the off chance everything hits the fan. (Also, this: "Don't break the industry practice and check-in your node_modules" - does not necessarily mean it is the best way, it just happens to be the advice from the start.

caymanjim 5y ago

You're drastically overestimating the storage requirements. Time is far, far more valuable than storage savings. The point about CI builds running faster is enough to sell the idea all by itself. Oddly not mentioned in the article is that it allows you to build when npmjs.org is down or unreachable, which happens often enough to be frustrating, and if it happens when you're trying to deal with an emergency, it's downright infuriating.

stavros 5y ago
We cache the directory on CI and get the same benefits without needing to make our repo massive.
mohanmcgeek 5y ago
We run an npm server that proxies request to npmjs.org and caches what it got.
Goz3rr 5y ago
Git is good at storing text diffs, any binary file in your node_modules (images, natives etc) are permanently stored in your repo, including old versions or deleted files. I've seen times where this had meaningful impact on both disk use and the speed of Git itself