18 ms·

We check our node_modules folder into source control

• nhoughto 5y ago
  I like the benefits but our node_modules is 1.9GB, def not checking that in. A Git bot that parses changes to yarn lock and comments with size/loc/files of all the deps etc, kinda like how coverage bots work. get all the observability benefits without the checkin cost, also wouldn’t have to split npm changed and code changes..
  • xdennis 5y ago
    What do you have in there that it's that big? I assume there are quite a few binary files which probably shouldn't be in Git.
    • nhoughto 5y ago
      Tbh haven’t gone deep into it, now you ask maybe I should. It compresses ok, zstd to 240MB. React + react native, we lack etc tons of stuff. My guess/hope is mostly dev dependencies.
      • nhoughto 5y ago
        mostly dev dependencies, i guess you really pay the price for not having a compressed intermediate format like a JAR: - storybook is 420MB (! will look into this one!) - babel is 110MB - sentry sdk is 92MB (?) - aws-sdk is 63MB - typescript 60MB - react-native 51MB .. on and on, 40MB..30MB..20MB.. hundreds of them just adds up! 147 dependencies over 1M. pretty incredible/wasteful really when you look at it.
• simonw 5y ago
  In the past for a large Python project I've handled this using a separate repository for all of the dependencies - that way you can still get work done even if PyPI is unavailable for some reason, but you don't bloat your main repository with an extra few hundred MBs of stuff.
• kavaruka 5y ago
  for this use case is better to use yarn 2+ with Plug'n'Play enabled
• maxyurk 5y ago
  Why not use fixed version dependencies and an internal npm registery such as JFrog Artifactory or Sonatype Nexus?
• uranusjr 5y ago
  Adding my voice to say this is not a bad idea at all in practice. Before anyone trying to apply the same approach to other ecosystems, however, be aware that (frontend) JavaScript is kind of unique in this. In most languages you’d likely have at least one package somewhere in the dependency chain that is in binary form, and those tend to need a bit more vendoring mechanism to handle correctly, unless everyone in your team have exactly the same setup.
• codevark 5y ago
  Note to self: don't have dependencies. oh, wait, I don't ha
• paulryanrogers 5y ago
  This very week I was dealing with my artifacts exploding in size because AWS got 429s from GitHub. Then Composer pulled from source and there were so many extra files and SCM folders we exceeded the max artifact size. Another idea is to host your own package cache. That would be my preference where SCM size prohibits checking in dependencies themselves.
  • antonvs 5y ago
    > Another idea is to host your own package cache. This makes most sense, and is the common solution at larger companies. Right tool for the job and all that.
• wtfrmyinitials 5y ago
  Perhaps you could achieve the best of both worlds by checking node_modules into a git submodule
  • saghm 5y ago
    I don't disagree with the general idea, but in practice, using git submodules always ends up feeling like I'm in the worst of most worlds
    • DaiPlusPlus 5y ago
      Subtrees, instead of submodules, are meant to be super-cool - and I'd love to try them, but I'm still far too wedded to existing git tooling (namely GitKraken) where there's still no support for subtrees.
• deleted 5y ago
  [deleted]
• cj 5y ago
  > Once you check your node_modules in, there's no need to run an install step before you can get up and running on the codebase Is this really true for packages with pre/post install scripts?
  • Bellend 5y ago
    No it's not. They are talking about their personal preference and it has nothing to do with real life employers.
    • DaiPlusPlus 5y ago
      RTFA. It says, quite succinctly: > I currently work at Google on the Chrome DevTools team and we check our node_modules folder into source control
      • Bellend 5y ago
        Yeah I read that and I disagree because at google if I checked in my node_modules I would fail my commit.
        • DaiPlusPlus 5y ago
          The author says they work on Chrome. Chrome doesn't use Google's much-feared monorepo, so maybe they made it work for them? Can't you ping him internally (don't you guys still use WebEx? lol) to ask him to clarify, and find out how it works for their team?
      • dzhiurgis 5y ago
        tbf Google is so far ahead that it’s kinda not a real company. Aka you don’t have to adopt every best practice google does since you aren’t that big, mature, secure or rich.
        • DaiPlusPlus 5y ago
          Now we just need to convince all those startups burning cycles trying to get Kubernetes working instead of working on their actual product...
• danlugo92 5y ago
  No, this is what the lock file is for.
  • spankalee 5y ago
    Indeed. What's probably needed here is a way to review a diff of the contents of the updated packages. Checking them is is just a brute-force way to do that.
    • pas 5y ago
      not to mention that unless someone is very familiar with the code of dependencies it's very hard to review hundreds of small near meaningless changes unrelated to your actual functional/business requirements. something like cargo-crev for npm might be a long term solution https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev
  • Bellend 5y ago
    "I delete my lock file" is so common that I would say it has no value.
    • bitlax 5y ago
      Check in your lock file.
      • 01acheru 5y ago
        I think that checking in your lock file is an actual best practice
      • Bellend 5y ago
        Done. Then I delete it, and `npm install`. Then commit. The majority of people I have worked with do that. On Friday some dude was saying "shrinkwrap v2 is not shrinkwrap v1" and the advice was "delete it and npm install" then commit. (Payment company software manager).
• DylanSp 5y ago
  Does the Chrome DevTools team use Google's big monorepo and all the tooling around it? If so, that puts the author in a different situation than the vast majority of devs.
  • DaiPlusPlus 5y ago
    Chrome DevTools has their own separate repo * https://developer.chrome.com/docs/devtools/ https://developer.chrome.com/docs/devtools/ * It's mirrored on GitHub: https://github.com/ChromeDevTools/devtools-frontend https://github.com/ChromeDevTools/devtools-frontend * And look, THAR SHE BLOWS: https://github.com/ChromeDevTools/devtools-frontend/tree/main/node_modules https://github.com/ChromeDevTools/devtools-frontend/tree/mai...
• aaronyarborough 5y ago
  This is insane. The only pro I'll even entertain is the potential to cut a few minutes off your CI/CD build time, but for the vast majority, even that brings no actual benefits.
• ilumanty 5y ago
  Yarn offers "Plug'n'play" mode since v2, which basically promotes what the author says. It takes the idea further: dependencies are stored as zip archives instead of thousands of small files, which reduces the "git noise" and actually makes this viable as a performant workflow. https://yarnpkg.com/features/pnp https://yarnpkg.com/features/pnp
  • maxpert 5y ago
    Exactly reading article I for a second felt like author was either not fully aware of tooling, or doesn't care about the noise it will cause (potentially complex conflicts) checking node_modules will cause.
  • Bellend 5y ago
    It's awesome apart from the vast majority don't use yarn.
  • Jarred 5y ago
    Unfortunately, reading files from .zip can be considerably slower than the filesystem. On an Ubuntu x86_64 machine with an SSD, require("react") with Yarn PnP ends up 4x slower than when installed from npm (Node 14.17.6).
    • DecoPerson 5y ago
      I might be misremembering but at runtime, the Node process loads .pnp.js, which is a kind of monolithic “compiled” modules file containing all the modules installed. No reading of the .zip files occurs at runtime. (Again, I might be misremembering or have misunderstood. Please confirm/deny this if you know.)
      • matharmin 5y ago
        The pnp.js file has a list of modules, not the modules themselves. The modules are still read from zip files at runtime. Yarn doesn't do any kind of compilation itself - that would be done by build tools like webpack.
• kello 5y ago
  This seems like an unnecessary space hog. If you are that concerned about reproducibility isn't that what npm shrinkwrap tries to solve?
  • orliesaurus 5y ago
    yeah dependency fixing is the way to do it with NPM shrinkwrap you lock down the version of the package you installed and their deps. That way you can use the same package on all envs. Helps with testing and debugging as you're removing a variable (bad deps, outdated deps, newer deps etc)
  • kleton 5y ago
    The google3 repo is probably terabytes at this point
  • davidjytang 5y ago
    I don't get why we are concerned about space. Space at this scale (never reached GB level) is so cheap and easy, to me at least, yet reproducibility is not.
• tombh 5y ago
  Whilst I fully sympathise with the points, none of them are convincing nor helpful to me. I'm all for a balanced helping of "just do whatever works", but that's my only takeaway from this article, namely: compromising on best practices can have its place. Or is there actually a fundamental design problem with lock files in general? My sinking feeling is that it's rather just the Node ecosystem's implementation of them :/
• dzhiurgis 5y ago
  Does it work across x86, 64, arm, linux, mac and windows? Some modules straight up download binaries so I don’t see being so straightforward.
  • uranusjr 5y ago
    Depending on the use case, especially if they’re developing a frontend project, this may be a non-issue. It’s very much a YMMV situation.
  • coding123 5y ago
    No, and that detail is dangerously missing from the advice. lol Anyone that does this and has teams on Mac, Windows and Linux will find out how crappy this is very quickly.
    • eyelidlessness 5y ago
      It’s even worse than that. If you support all LTS/current Node versions—as is, and should be, very common for libraries—it'll break even on a single platform. I’m sure it’s solvable, but the solution would be so complex it’s tantamount to building a new package manager.
  • davidjytang 5y ago
    We can thus infer that author being working in "Google Chrome DevTools team", they must have a way to unifying their dev env.
  • tossaway9000 5y ago
    I came to mention this exact issue. I've seen this trip up people in the past, in one case a CI/CD system running Linux was used to produce a project deployed to a mostly windows environment this didn't cause any issues until the day a developer added a binary module. Honestly, I'm surprised it worked as long as it did, it took a little over a year before anyone hit that issue.
  • zffr 5y ago
    Would this still be an issue if the whole team uses docker to run the code?
• DaiPlusPlus 5y ago
  The GitHub mirror of the Chrome DevTools repo has the node_modules folder here: https://github.com/ChromeDevTools/devtools-frontend/blob/main/node_modules/ https://github.com/ChromeDevTools/devtools-frontend/blob/mai... But there is nuance (there always is...), the README file in node_modules is here: https://github.com/ChromeDevTools/devtools-frontend/blob/main/node_modules/README.chromium https://github.com/ChromeDevTools/devtools-frontend/blob/mai... - and it makes it clear the only NPM dependencies used by the build-system or infrastructure is meant to be checked-in. Other NPM packages should not. ---------- In conclusion: the linked blog-article is clickbait that misrepresents how the Chrome team manages their dependencies.
  • malkia 5y ago
    Maybe I'm wrong, but I see quite a lot of node_modules here - https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/node_modules/ https://source.chromium.org/chromium/chromium/src/+/main:thi... Here are all "node_modules" folders I can find (sorry I'm not much familiar with npm/js to know more) https://source.chromium.org/search?q=file:node_modules%2F$%20pcre:yes https://source.chromium.org/search?q=file:node_modules%2F$%2...
    • malkia 5y ago
      This file seems to be controlling what gets ("checked-in") - https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/scripts/deps/manage_node_deps.py;bpv=1 https://source.chromium.org/chromium/chromium/src/+/main:thi... In our industry (games) we often do that - checkin prebuilt code in the depots (typically "p4"). I'm not saying it's wrong/right, it's just what we do (not 100% fully, but almost, though people in IT/infra tend to do otherwise).
      • DaiPlusPlus 5y ago
        Ah Perforce... you have my sympathy. Does your team/company store art and asset blobs (even rendered FMVs?) in Perforce too, or do you have a separate asset-management system for that? If you do have two separate systems, how do you keep the asset-store and source-control in-sync? It's been a long time (easily 20 years now) since I dabbled in any high-end creative software (like 3ds, etc), but I remember they generally all had large binary singleton project files (like Flash .fla and Photoshop .psd files) that couldn't be deconstructed and effectively diffed by any source-control system (though Flash eventually supported external ActionScript source-files), I'm curious how that affects your org's asset storage needs.
• miganga 5y ago
  I mean come on now. You have internet? Good. Then u will be able to both run npm & git commands. Plus npm has cache. Why burn internet payload for no reason?
• tlhunter 5y ago
  This really only works if you avoid native modules. I'm not sure why the author didn't mention native modules as it's a pretty big deal in this context.
• AtlasBarfed 5y ago
  Maven, Gradle and the like always download dependencies as part of the build, and if a server is down (or flaky like jitpack) good lord is it annoying. What I don't understand is why dependency download isn't a separate task you do before compilation. FIRST you download all the dependencies to stabilize that, then you build the code. The reproducibility is the big one. IT drives me nuts that the dependencies are hidden away in maven and gradle. I have to lookup obscure "download dependencies to a lib" task configuration. Still the obsession with massive jars/wars/whatever when all you should have to update in a deploy is the difference in the libs and the main code jar. The reason for this obscurement is pretty much "uh, it saves disk space?" which is a laughable consideration given the bloat in war files, docker images, and the like. I agree with the poster.
  • gravypod 5y ago
    > What I don't understand is why dependency download isn't a separate task you do before compilation. FIRST you download all the dependencies to stabilize that, then you build the code. This is what bazel does. It also offers a `bazel fetch` to pre-download things before going offline (ex: flight).
  • oftenwrong 5y ago
    For Maven, you can use this plugin to move downloading dependencies into a separate step: https://github.com/qaware/go-offline-maven-plugin https://github.com/qaware/go-offline-maven-plugin It's not perfect, but it can be useful.
• mohanmcgeek 5y ago
  The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules
  • klodolph 5y ago
    I’d say that the main link in that reasoning is that “Git can’t handle this without making the repo a gazillion GBs” which, can of course, can be solved if you weren’t using Git in the first place. Certain other SCMs, like Perforce, allow you to trim history and don’t require you to clone the whole history in the first place.
    • sofixa 5y ago
      With git you can specify the clone depth and only get the latest X versions. And there are ways to trim history with external plugins ( git filter repo).
      • toxik 5y ago
        You have to remake every commit of the repository which basically means you have a new repository, and new commits. In order to do the latter thing you said, you cannot have done the former thing you said.
    • avgcorrection 5y ago
      The author thinks that Git is so obiquitous that they just use “Git” as a stand-in for “VCS” throughout after the first paragraph. So the author definitely thought that Git would be suitable for this.
  • umvi 5y ago
    At my previous company we committed a zipped node modules to git LFS so we could have easy reproducible offline builds without hosting an internal npm instance. Seemed to work well enough.
• revskill 5y ago
  In my experience, it's about the inconsistency between Node.js versions (each version could produce different package-lock.json). So, for example, you install the dependencies with Node 12, but have to run locally the system with Node 14. So, i think just checkout the node_modules folder into git is not complete solution. You're avoiding the need to commit inconsistent package-lock.json, which is not hard to solve though.
  • deanc 5y ago
    Nobody should be running different versions in a team. Unless you’re selling a product where this is a likely scenario you should be running the same environment either through some kind of virtualization or at worst nvm.
  • Tajnymag 5y ago
    Isn't that really a problem with npm and package maintainers? Node is really just a runtime.
    • revskill 5y ago
      When you install nodejs, you have npm installed. So in this case, there's not much you can do though. So it's not just npm issue, it's NodeJS issue, too.
      • Tajnymag 5y ago
        Not necessarily. On Arch, npm isn't bundled with the node package. Also, according to the official npm documentation, npm should be installed through nvm, thus having the ability to specify its version separately from node. https://docs.npmjs.com/downloading-and-installing-node-js-and-npm https://docs.npmjs.com/downloading-and-installing-node-js-an...
• chx 5y ago
  Totally agreed with every step, that's what we do too -- except it's composer and composer managed packages here not npm but the reasoning is similar. On top, we too often need to actually patch composer managed packages and rolling patches without the code being version controlled is a PITA. It's git diff --relative if it's already in git otherwise it's .... I dunno, check out the package somewhere else, hope you get a close enough version (because what's released can differ a bit from version control), copy over the files , roll a patch, clean up the patch... what an unnecessary nightmare. And composer patches makes life quite easy compared to maintaining a fork. If I were to fork something I would need to handle merging every time they have a new release, run the build etc. With composer patch, a new released version is installed and the patch on top. Sure, if there's a conflict that needs to manually resolved but that's usually minimal effort since most patches are absolutely tiny, a few kilobytes at most. I never even understood the arguments for keeping the packages out of git. Trying to save disk space these days is pointless. Maybe npm is different but composer handles about 160MB of code here. Maybe I missed the memo but these days that's nothing. My laptop shipped with a 500 000MB SSD so it's like, what, half a percent? The speed advantage , on the other hand, is absolutely undeniable, git won on speed in the first place, these script language tools can't possibly compete with a git pull on speed. git diff, as the author notes, is not at all a problem, just separate the vendor commits from your commits. And as I noted: they are useful for vendor packages.
• embazeller 5y ago
  Source code is source code, it’s the input into a build process, it doesn’t determine the output: the build process itself must be designed for reproducible builds. Committing all of your dependencies doesn’t give you reproducible builds. I don’t think committing ‘node_modules’ is egregious, and it has benefits as the post describes, but unfortunately it’s far from _the_ solution to the problems described. Something like Google’s own Bazel is a better option.
• dools 5y ago
  I agree, and to go a step further I don’t even understand why everyone is so obsessed with package management systems when sub modules do the same thing only better
• abraxas 5y ago
  This is basically a way of saying: to hell with those "package managers". It's a sentiment that I'm actually in agreement with. I've been coding mostly in Java for the past 22 years. Somewhere around 2010 Maven became the prevalent build tool quickly displacing the venerable Ant. With Ant we had builds that used checked in jar file dependencies. It was obvious what your builds consisted of and they were very fast once you cloned the repo. Then came Maven and the conflict resolution hell quickly followed (esp with unpinned dependencies). Now every time I type mvn install it feels like a small adventure in its own right. I'm absolutely flabbergasted. We sacrificed simplicity and reliability for a bit of instant gratification. Bad tradeoff.
  • kevin_thibedeau 5y ago
    Ant has shite syntax that "clever" coworkwers can turn into a morass.
  • smarx007 5y ago
    It’s been a long while since I saw a POM file with unpinned dependencies… ca 12 years of Java experience. Had to deal with an Ant project recently that didn’t check in the JARs into Git, I thought my hair would become gray by the time I figure out how to pull the right dependency versions transitively to get the Ant build to work. But I share the overall sentiment. What if Maven Central goes down one day? It’s just a web server like any other.
    • oftenwrong 5y ago
      >What if Maven Central goes down one day? Most Java shops run their own artefact repository acting as a pull-through repository to Maven Central and other 3rd party repositories. In addition to acting as insurance against losing access to critical dependencies, it also can provide a performance boost when downloading dependencies, and helps to offload your organisation's traffic from Maven Central.
  • oftenwrong 5y ago
    You can save yourself some headaches by not using transitive dependencies in your Maven builds. This can be enforced with Maven Enforcer. The trade-off is that you will have to clean unused dependencies. This can be tool-assisted with something like: https://github.com/castor-software/depclean/ https://github.com/castor-software/depclean/ . You can also enforce dependency version convergence with Maven Enforcer. None of this saves you from "jar hell" directly, but it helps prevent you from unknowingly creating a disaster-in-waiting.
• awill88 5y ago
  Can’t you like, cache the node_modules folder on CI builds? I dunno, seems gross, unless you’re on a project with minimal deps or very meticulous about which deps you leverage. I am just one of those people who are constantly trying to upgrade dependencies anyways (cautiously of course) as to avoid vulnerabilities. That said, I see the point, it’s interesting..
  • 20after4 5y ago
    Upgrading could introduce vulnerabilities just as easily as fixing them.
    • awill88 5y ago
      it’s possible, kind of a moot point as doing nothing induces a similar level of risk in my experience. Don’t touch your code for two weeks, I guarantee you that ‘npm audit’ will complain about some new issue.
      • trog 5y ago
        Also worth noting the longer you wait to upgrade, the harder it can be to do so when you finally need to. If someone discovers a critical fault in the version you're running but you're several years out of date, upgrading can be a huge pain.
• jakear 5y ago
  So instead of downloading the current set of dependencies after cloning, you download every dependency ever used while cloning? And you do this to "save bandwidth"? Doesn't make much sense. (Yes I know about shallow clones, but it's often nice to have the full history around) Any reasonable CI tool will have a way to cache generated assets based on file contents, that's the way to go here IMO.
  • cxr 5y ago
    Consider if you had written, "So instead of downloading the current project source tree, you download every version..?" That is what happens in a DVCS, after all—in fact, it's sort of the whole point. If you're so uncomfortable with this, it might be worth asking yourself whether it was ever really the case that you agreed that DVCSes were the right approach. (Even then, still no reason to embrace package managers like some kind of paramilitary force that's subject to its own rules—better to just improve your version control system to handle things the right way, right?)
    • jakear 5y ago
      Theres cost-benefit tradeoffs for sure. It'd be nice to be able to do a big bisect without needing to flash the dependencies every iteration, but if it comes at the cost of making everything else slower (more bandwidth, more disk I/O, larger diffs to parse, etc.), it just isn't worth it. Sure you can say "well why not just make git better then it will handle any operation in any size repo imperceptibly quickly", but I think you and I both know that isn't anywhere near as easy to implement as it is to type.
      • cxr 5y ago
        > you can say "well why not just make git better then it will handle any operation in any size repo imperceptibly quickly", but I think you and I both know that isn't anywhere near as easy to implement as it is to type Good thing I didn't type that, then. That's not what the shape of my argument looks like at all. There's a massive leap between, "we don't want every clone of our repo per se to carry all the baggage of our dependencies' histories, so it would by nice to have some scheme for handling lightweight, shallow copies" and "... so we decided the right way to do that, rather than making that a first class feature of the version control system we're using, is to create a hack in the form of a new set of unrelated tools meant to circumvent our VCS's fundamentals completely—so from our its point of view, these controlled objects and the scheme we use for managing them are invisible and might as well not even exist."
• ng12 5y ago
  > We've come up with a rule that helps us here: a change that updates node_modules may not touch any other code in the codebase What? What if the dependency upgrade requires code changes? Master is just broker until the second MR merges? This all sounds like terrible advice.
  • emteycz 5y ago
    You can have multiple commits in a PR
    • lf-non 5y ago
      Yes, but a lot of people prefer to have every commit in master in a buildable state. Otherwise things like git bisect become a lot more tedious.
      • xboxnolifes 5y ago
        > have every commit in master Yes, but a PR commit is not in master.
        • pjerem 5y ago
          I see you, little squasher :)
        • ng12 5y ago
          But then you lose the benefit. Hiding node_modules during code review is the easy part.
      • clintonb 5y ago
        Some teams force all pull requests to be squashed to a single commit.
  • deleted 5y ago
    [deleted]
  • NavinF 5y ago
• throwaway984393 5y ago
  Why are people still writing server-side code in Javascript? It was a cute idea a decade ago. But now you have to twist yourself into knots to prevent a ton of problems. Nobody will use C today because "memory safety", but everybody will use Node.js despite the dependencies being both a security and usability nightmare. Imagine if I suggested using Bashscript for a web application backend.
  • WA 5y ago
    What do you use for server-side code?
    • throwaway984393 5y ago
      Visual Basic
• scrollaway 5y ago
  This is all horrible advice and other commenters have rightfully pointed this out already so I won't repeat it more… (okay, once more: This is all horrible advice, don't do that) But, there is one thing I like from this, which is git diffs showing the actual final code diff when you upgrade dependencies. Of course, this being horrible advice, it ignores how many JS packages ship minified which would make the diff as useful as binary noise. But I like it in theory. This could be a good opportunity to write a tool that replicates this specifically (and what's more, for other languages as well)
  • johnnypangs 5y ago
    Most node modules (and I’m pretty sure is best practice) do not minify published node module code. The user should decide if they want to/how to minify. I often go in an read node module code and although it might have been transpiled for compatibility, it is not minified.
• nickkell 5y ago
  I know pulling in lots of dependencies from various anonymous authors is a security risk. Can you be sure that all of the code has been vetted? This seems exacerbate this. You’re allowing developers to check in anything without oversight and it will be ignored just because it’s in this particular folder
  • Gigachad 5y ago
    That’s already exactly the same risk almost all web developers take currently. Yes it is a real threat, but it’s too hard to deal with and not often exploited.
    • nickkell 5y ago
      We all hope and rely on the fact that popular OSS projects have enough eyeballs on them to make sure nothing malicious slips through. What is proposed here allows a load of changes to be made that completely bypass the normal review process
      • Gigachad 5y ago
        This doesn't bypass the review process because no ones review process includes auditing the code of all things in the package lock files. This is no more or less secure than the current way of doing things.
• jcun4128 5y ago
  I remember doing this on accident noticing 20,000+ files starting to get uploaded, was waiting like "something's not right".
• StopHammoTime 5y ago
  I don't agree that it should be checked into source control. I do believe it should be cached somewhere. How much bandwidth of popular sites are used by redundant actions. A single request for a 2GB archive is much better than a 1,000,000 small requests that are all a few KBs or MBs in size.
  • cookiengineer 5y ago
    Nodejs has a node-gyp problem. Every node_module that somewhere down the dependency tree requires a "native" module will require recompilation on the target machine (or in worst case: the user machine). I really would have hoped that the NaN module related problems will be fixed over time, but here we are in 2021 and nothing's been fixed. As long as npm doesn't use binaries and headers, those things will stay broken. The thing that they argue with to use "always source" is kinda ridiculous when considering that probably the most of all npm packages are using webpack or another bundler before pushing their own package to npm - because npm itself has become impossible to use as a package manager alone. I mean, a couple MB of libraries with the wrong dependencies can lead to multiple phantomjs installation, which is an inactive, deprecated, and unsecure project for years already... just because of some unit tests that have no place in a production npm package. My hopes are that more sane developers come together, switch to ESM and implement better policies for evaluating their dependencies (e.g. blocking sources from people that have more than 1000 npm packages and brag about it). Pikapkg was a great idea in my opinion, and I was using it before they moved the project to building astro as a platform :-/
• jdlshore 5y ago
  Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Granted, I’m fairly restrained in my use of dependencies. But I do update them regularly.) But even if it does take a lot of space… so what? If your dependencies are genuinely so huge that this is a problem, then vendoring may not be right for you. But you could also use one of the many techniques for managing the size of your repo. Or just acknowledge that practices are contextual, and there’s no such thing as “best practice”—just a bunch of trade-offs. 2) “It doesn’t work well with platform-specific code” This can cause some pain if you’re in a multi-platform environment. The way I deal with it (in Node) is by installing modules with --ignore-scripts, comitting the files, running “npm rebuild”, and then adding whatever shows up to .gitignore. I have a little shell script that makes this easier. This is only an issue for modules that have a platform-specific build, which I try to avoid anyway. But when it comes up, it can be a pain in the butt. I find its pain to be less frequent and more predictable than the pain that comes from not vendoring modules, though, so I put up with it. Bonus) “It’s not best practice” Sez who? Dogma is for juniors. “Best practices” are all situational, and the only way to know if a practice is a good idea is to examine its tradeoffs in the context of your situation.
  • jayflux 5y ago
    > Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.
    • dtgriscom 5y ago
      npm has long had a problem respecting lock files. The concept is easy: have a fixed lock file, get a reproducible build. But no: npm will change your lock file (I believe it's framed as "optimizing") without notice. (Perhaps they've solved this in the last couple of years. I've been staying away from that ecosystem... too much growing in it...)
• po1nt 5y ago
  None of the benefits mentioned in article won't benefit well maintained codebase. If you're forced into such solution you're doing something wrong. Just use locking, cache and reduce your dependencies to bare minimum you actually need. In JS you can often find yourself pulling a massive dependency tree just for simple usecase that would be replicable in your codebase under 3 hours.
• z3t4 5y ago
  Suggestion how to manage node_modules: When testing different modules use npm/yarn, but once you have decided which module to use, fork the module on Github (most node_modules are on Github) then link directly to your fork in package.json like this: https://github.com/{user}/{module}/tarball/master https://github.com/{user}/{module}/tarball/master Now you can use git/Github instead of npm to manage package updates.
• maccard 5y ago
  People here complaining about the size of the node modules folder - this is one of the reasons people use other solutions like perforce. Checking a 2GB folder into p4 is an absolute no brainer, and for all the flexibility people talk about with git, it's inability to handle this is pretty damning after so many years.
  • sktrdie 5y ago
    Can you expand on this? Why is the size smaller on p4 compared to git?
    • masklinn 5y ago
      P4 does not store the entire history locally, so your only limit is the amount of space on the server. Git will pack objects after a while (initially they’re stored as-is, just compressed), but large files or enormous amounts of changes can make the repository grow to unwieldy amounts, and while git is able to perform “shallow” clones (clones which only store part of the history), not everything handles them well
      • oftenwrong 5y ago
        Git also has "partial" clones, which can avoid some of the downsides of shallow clones. You can, for example, filter out all historical objects (--filter=blob:none). Historical objects are fetched lazily as needed, such as checking out an old revision. The same can be done with unreachable trees. This article has excellent diagrams that depict the different types of clones: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...
    • maccard 5y ago
      As the other commenter replied p4 doesn't download the entire history locally, so storage is only a concern on the server. Depending on what you're storing in p4, you can use p4 archive and/or p4 obliterate to manage the actual storage used on the server. E.g. you might use p4 obliterate to only keep the latest version of your node modules folder in p4 if you're trying to optimize for CI performance, or if you're using p4 for build artifacts, you might use p4 archive and store the older revsions on slower bulk storage.
• sam_goody 5y ago
  Submodules. We had left-pad type incident where the package we depended on disappeared from npm, and didn't even realize it till the next time we setup from Git. Since then, we have node_modules as a submodule tracking latest-commit, and it works.
• skytreader 5y ago
  > Better awareness of the code you're shipping. Then somewhere down the line... > One of our dependencies that we check in is TypeScript, and every time we update that, the git diff is huge and frankly not worth looking at (beyond the CHANGELOG) Which I assume is the official attitude towards any dependency of the same magnitude. How are you more aware of the code you are shipping? Okay you managed to give yourself a visual on how much LOC your dependencies are but is that a relevant awareness? Do I not get the same thing with a `du -h node_modules`, with a matching pretty GUI on top? The one thing I haven't seen addressed so far is: doesn't this make them susceptible to poisoned dependencies? Say they have a dependency to a large well-known library, what's stopping a malicious contributor from adding an HTTP call, thinly disguised to prevent grep, to some server in MiddleOfFucking, Nowhere? Even if they manage to flag this from blackbox testing, they now have a problem that only they have. I can try to answer my own question: they're Google, they can afford a team scanning for vulnerabilities like this, a team dedicated to analyzing codebases that are found to be compromised, a legal and PR team to handle the fallout if this kind of vulnerability makes it to the public. In short: horrible advice to follow if you are not Google.
• Chyzwar 5y ago
  This is idiotic and could only be written by someone that have no idea about node package managements. 1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds. 2. Once you start upgrading packages repo size will continue to grow. At some point you will be forced to git filter out node_modules. You will lose the ability to run locally older commits. 3. You will need to pin version of npm/yarn because structure of node_modules depends on hoisting algorithm. Every upgrade of node will be extra painful because you also potentially need to upgrade all yours packages. 4. Platform dependent modules like fsevents, node-sass can be broken if you use different OS. You will be forced to only support a single platform(linux). 5. Impossible to resolve node_modules conflicts. Modern package manager have git conflict resolution build in. If two people update the same module to the same version they can still create a merge conflict when node_modules are checked. 6. Currently, you have plenty of good options that can achieve the same with smaller effort. You can use yarn2 with node_modules linker and local cache. This would create .yarn folder in the repo that have all modules as zip files. During install it would use these to hydrate node_modules. Alternatively, You can use pnp and have zero install but with proper support[3] 7. You lose automatic audit and dependency management. Current best practice is to use something like dependabot or renovate bot. Once you commit your package, you will no longer be able to use this effectively. 8. Most people commenting on left-pad are maybe not aware, but today npm is immutable, and you simply cannot unpublish public package[4]. Because npm.com is such vital infrastructure, it unlikely that it would ever stop working. [1] https://docs.github.com/en/actions/advanced-guides/caching-dependencies-to-speed-up-workflows https://docs.github.com/en/actions/advanced-guides/caching-d... [2] https://circleci.com/docs/2.0/caching/ https://circleci.com/docs/2.0/caching/ [3] https://yarnpkg.com/features/pnp https://yarnpkg.com/features/pnp [4] https://docs.npmjs.com/policies/unpublish https://docs.npmjs.com/policies/unpublish
  • tentacleuno 5y ago
    > This is idiotic and could only be written by someone that have no idea about node package managements. Some of your points are valid, but the first sentence seems really hand-wavy and rude. > 1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds. I was wondering about this too[0], mostly because it's the setup I tended to gravitate towards for my projects. Gitlab CI and Travis can do this AFAIK, although I'm not sure how long the cache folders would be kept (especially on the Free tier). [0]: https://news.ycombinator.com/item?id=29529154 https://news.ycombinator.com/item?id=29529154
• jetsetgo 5y ago
  > Better awareness of the code you're shipping Ships free open source code and complains. Why don't they write it themselves. That way one can have full control.
• tiotempestade 5y ago
  I know there’s no other way, but I wish JS could just die!
  • Tajnymag 5y ago
    Then you might be happy to hear that a pure JavasScript has been dead for quite some time. Nowadays, most new large codebases run on TypeScript or some recent version of EcmaScript. The rant the article is based on is not really about language itself but the tooling and ecosystem around it.
• tentacleuno 5y ago
  > Once you check your node_modules in, there's no need to run an install step before you can get up and running on the codebase. This isn't just useful for developers locally, but a big boost for any bots you might have running on a Continuous Integration platform (e.g. CircleCI, GitHub Actions, and so on). That's now a step that the bots can miss out entirely. I've seen projects easily need at least 1-2 minutes to run a complete npm install from scratch [...] Couldn't this issue be solved by using caching? If I remember correctly, Travis CI has the option to cache certain folders between builds, meaning an npm install doesn't have to start from scratch, and can just incrementally update the cached node_modules folder (any changes are then copied to the next CI build).
• paulbjensen 5y ago
  I understand what he's getting at, but if you are trying to have an exact copy of code in order to guarantee an exact behaviour, then you would want to extend that to the operating system used to run the software. One example was having code where the test suite ran fine on my local MacBook, but would fail on CI (Linux). It turned out that on Linux finding files by name is case sensitive, whereas on Mac OS it isn't, and a require statement in Node.js was referencing a file path with casing that was different to the file name's spelling.
  • smarx007 5y ago
    Isn’t it where we are going with Nix and co?
    • notthathardbro 5y ago
      This entire comment thread is just torture for anyone who uses real dep management tools like Nix and Guix. Sorry, but it's just exhausting watching the same asinine conversations play out over and over. I guarantee there are multiple people who have spent more time pontificating over pointless ways to microscopically improve the node tooling situation who could've picked up Nix in 1/10th the amount of time. But then again, I know there's a bunch of Nixers that pick their head up slightly, shake it, and then just go back to work on actually interesting problems instead of a millionth discussion about dealing with npm. Christ, the stuff people put up with. EDIT: Ironic, this being here along with the CISA/log4j post where everyone is yammering about SBOM (software bill of matearials). Again, I just glance over at Nix and go, "sure, what do you want to know, I can tell you instantly if log4j is anywhere and if it's a vulnerable version (excluding non-source-built packages in nixpkgs)".
• HelloNurse 5y ago
  I currently have a more concrete problem than speed with leaving node_modules out of version control: buggy libraries that, as downloaded, need small corrections. Doing the right thing and fixing libraries for everyone on npm requires significantly more effort and it could be impossible.
  • williamdclt 5y ago
    patch-package (yet another dependency) does a decent job for that sort of thing
• hardwaresofton 5y ago
  see also: yarn offline caching[0]. Also great for dealing with CI environments where you can only cache stuff in certain folders (you can control where the cache is stored) [0] https://yarnpkg.com/features/offline-cache/ https://yarnpkg.com/features/offline-cache/
• somehnacct3757 5y ago
  Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI will pay the time penalty during git clone instead of npm ci. In fact, the node_modules folder will be bigger than your source folder almost immediately. And over time you won't be cloning just the head files you'll also be cloning every npm package binary ever committed. You can't undo this without investing in smarter git tooling. Which is time spent not writing features. NPM packages which install arch-specific binaries will constantly flip flop from commits by devs on different OS's. Nobody is safe from left-pad, not even Google, and committing your node_modules folder doesn't change that. Eventually someone is going to have to run npm i. Running npm ci on everyone's machine is reproduceable, I don't know what OP is warning about. Package lock pins all the versions. If you have a large enough team to invest in dev experience, there's way better ways to get the advantages of the article without the downsides. You can cache the npm ci result in a container layer for your CI/CD or use middleware like artifactory
  • mustyoshi 5y ago
    Isn't --depth=1 all you need to avoid downloading every binary?
    • somehnacct3757 5y ago
      Maybe, but everyone's CI situation is so variable that it may not be that easy. For instance if you are using a monorepo then even a shallow clone can be overkill. And if you rely on git history for conditional CI then a shallow clone will ruin the output of many git commands. So you could end up in an either/or optimization situation depending on the order your CI/CD organically grew and other architectural decisions you made.
  • gravypod 5y ago
    > Your CI will pay the time penalty during git clone instead of npm ci. Things like GitLab's CI runners will do a single clone then do `fetch`, `checkout`, and `clean` to checkout your repo. Git repo size isn't a huge bottleneck in CI performance.
• NaN1352 5y ago
  I don’t check in node_modules but do a backup once in a while. With frontend nowadays it’s sad but after six months it’s highly unlikely that my project will compile, not to mention the tooling like Vue, Vite, etc that has breaking changes. I mean it’s scary. You write a program and it WONT run if you just give it enough time. Locking versions is not really a solution since often times the tooling itself, and IDE extensions require newer versions of packages. Maybe you wanted to fix a typo a year down the line but oh no, now you need to figure out why Vite won’t start, why eslint dropped support for xyz, spend hpurs figuring out what you need to change in your configs, etc.
  • onion2k 5y ago
    I don't see how this would work in practise. You could use module-alias[1] to actually switch to the backup copy during dev or a local build, but that will only work when the backup is on the same machine you're building on, and then everyone on the team will need to have the same backup (or use a network drive for it I guess). If you don't check in your backup then nothing will get through CI or make it to production. Why not just check in node_modules and let git handle the 'backup' process? [1] https://www.npmjs.com/package/module-alias https://www.npmjs.com/package/module-alias
    • NaN1352 5y ago
      Sure I’m talking solo projects.
• flerp 5y ago
  Node_modules are OS specific.
• mynameismon 5y ago
  I feel Fossil's unversioned files [1] is the perfect answer to such issues. [1] https://fossil-scm.org/home/doc/trunk/www/unvers.wiki https://fossil-scm.org/home/doc/trunk/www/unvers.wiki
• ChrisMarshallNY 5y ago
  This is fairly standard "conservative configuration management" stuff. The company I worked for, used to do that with everything. In fact, one of the ways that they archived versions, was to create a bootable external hard disk clone, of the entire development machine, and store that. If you want to be absolutely sure that you have the complete building blocks, then you don't trust your package manager. Make local dupes of the packages, and integrate them into your own version control.
• logicandreason 5y ago
  To my former team 5 years ago; BAM! Told ya so. (They read HN like the Bible) I'll get hate mail now. lol
• stevage 5y ago
  Possibly dumb question. But does this prevent developers working on different platforms? My understanding was that the specific files installed could depend on the architecture of the system being installed on?
• littlecranky67 5y ago
  The article contains some false assumptions and false statements > Having your node_modules checked in guarantees that two developers running the code are running the exact same code with the exact same set of dependencies. No, it doesn't, your system environment is important. Any code executed can check for environment parameters and branch accordingly. For example, a simple "if (macOSVersion === "10.10") {} else { }" would run different code branches and possibly produce different results, even when executing the same code binaries. The reason you don't check in node_modules is that differences in system env during build time produces different build results - checking in node_modules fixes that, but does not handle system differences at runtime.
• penguin_booze 5y ago
  Can't speak for elsewhere, but at $WORK, the development and CI environments sit behind a package resolver (don't know if that's the right word) which transparently caches and proxies package downloads. So dependencies are implicitly persisted permanently by virtue of usage, even if only once.