5 ms·

Time to start committing dependencies to your VCS.

tlrobinson 11y ago
$ du -h -d0 node_modules 673M node_modules EDIT: Actually it looks like du is reporting double the size, it's really 337.2MB according to OS X Finder (apparently it's using 512 byte blocks on OS X https://stackoverflow.com/questions/23599131/why-is-du-command-showing-double-size-when-h-flag-is-sent https://stackoverflow.com/questions/23599131/why-is-du-comma...)
- nostrademons 11y ago
  git annex? (I get your point, though. My node_modules is 490M, and that's from a package.json that fits in one screenful on a terminal window.)
- jerf 11y ago
  So, fully serious and non-sarcastic question since I'm not a node developer, what is in that 673MB?
  - trump2016chalk 11y ago
    Awaiting a response to this as well
  - deleted 11y ago
    [deleted]
  - tlrobinson 11y ago
    It's mostly Babel and other build tools: https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32513c301 https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32... With npm every dependency gets it's own tree of dependencies, so there's tons of duplication. Also, this is using npm 2. With npm 3 that's approximately cut in half since it tries to flatten the hierarchy as much as possible, but I haven't switched to npm 3 yet (seems slower and still somewhat buggy)
    - jerf 11y ago
      "With npm every dependency gets it's own tree of dependencies, so there's tons of duplication." That sounds like the missing bit of info. If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.
      - tlrobinson 11y ago
        Good point. If I add my node_modules to a new git repo it ends up being about 65MB, which is better, but still significant, and I imagine the overhead of tracking an additional 100,000+ files is non-trivial.
      - etblg 11y ago
        That also shouldn't be true in the latest npm, which uses a global node_modules and a bit of trickery for different versions of the same library, instead of recursive node_modules.
    - nostrademons 11y ago
      Babel seems to be a terrible offender as far as dependencies go. In my node_modules, babel-preset-es2015 takes up 200M - as you've noticed, each individual transformer has its own copy of the core Babel runtime. And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel. The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.
      - nilliams 11y ago
        Yes, if you only write JS for Chrome and Node, not sure how many JS devs fall into that bracket!
  - ceejayoz 11y ago
    In many cases, a bunch of useless stuff. One of our devs added a material design webfont NPM package (IIRC, https://github.com/google/material-design-icons https://github.com/google/material-design-icons). Something like 40kb worth of font files, but the folder was 100+ MB because it included a PNGs of each icon at every imaginable resolution. I've seen others with 10+MB CSV files used by the unit tests.
    - jakub_g 11y ago
      In such case you should send them a pull request with a proper .npmignore file which contains whole of .gitignore plus rules to not publish those PNGs
- arrakeen 11y ago
  i've found 85M of the same library in my project: https://gist.github.com/bsdf/e0db8f5ac90fee5ccff8 https://gist.github.com/bsdf/e0db8f5ac90fee5ccff8
  - deleted 11y ago
    [deleted]
- mdavidn 11y ago
  A note on your edit: The difference you observed is not because of du. Your node_modules really does take 673 MB on disk. File systems divide the disk into segments, confusingly also called blocks. Only one file may occupy a block. On my Mac, the block size is 4 KB, so a file containing a single byte will still occupy 4 KB on disk. The du utility reports the latter. The Mac Finder reports the former in column view and both in the "Get Info" pane. Unpacked NPM modules have many, many small files, obviously.
  - tlrobinson 11y ago
    Ah, interesting, thanks.
bsimpson 11y ago
I believe this is common practice at firms that have a monorepo.
acemarke 11y ago
Re-re-posting a comment I've made in a few threads now, because I feel this needs to get more awareness: I'm still learning and prototyping my first JS/NPM/React project, but https://github.com/JamieMason/shrinkpack https://github.com/JamieMason/shrinkpack seems to me like it solves _most_ of the issues involved here. It pulls down all the tarballs, and updates the npm-shrinkwrap.json to point to those instead. That way you check in a much smaller "node_shrinkwrap" folder of a few hundred tarballs and 15-20MB, rather than a node_modules folder of 30K files and 150MB and a bunch of platform-specific build outputs. Still doesn't solve the issue of installing new/updated dependencies that might actually require pulling in something that vanished, but at least once you've done an install and run "shrinkpack", you've got everything you need to rebuild right there.
A1kmm 11y ago
I find most VCS software is not well suited to handling large immutable files (immutable in the sense there shouldn't be two different files with the same vendor, name, and exact version). With a distributed VCS, you probably don't want anyone doing a clone to have to download every version ever of every dependency. A binary repository like Nexus or Artifactory is very useful for this. To fix the npm instability problems, I use Nexus and configure it to proxy (and permanently cache) npmjs, and then use npm shrinkwrap.

$ du -h -d0 node_modules 673M node_modules EDIT: Actually it looks like du is reporting double the size, it's really 337.2MB according to OS X Finder (apparently it's using 512 byte blocks on OS X https://stackoverflow.com/questions/23599131/why-is-du-command-showing-double-size-when-h-flag-is-sent https://stackoverflow.com/questions/23599131/why-is-du-comma...)

nostrademons 11y ago
git annex? (I get your point, though. My node_modules is 490M, and that's from a package.json that fits in one screenful on a terminal window.)
jerf 11y ago
So, fully serious and non-sarcastic question since I'm not a node developer, what is in that 673MB?
- trump2016chalk 11y ago
  Awaiting a response to this as well
- deleted 11y ago
  [deleted]
- tlrobinson 11y ago
  It's mostly Babel and other build tools: https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32513c301 https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32... With npm every dependency gets it's own tree of dependencies, so there's tons of duplication. Also, this is using npm 2. With npm 3 that's approximately cut in half since it tries to flatten the hierarchy as much as possible, but I haven't switched to npm 3 yet (seems slower and still somewhat buggy)
  - jerf 11y ago
    "With npm every dependency gets it's own tree of dependencies, so there's tons of duplication." That sounds like the missing bit of info. If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.
    - tlrobinson 11y ago
      Good point. If I add my node_modules to a new git repo it ends up being about 65MB, which is better, but still significant, and I imagine the overhead of tracking an additional 100,000+ files is non-trivial.
    - etblg 11y ago
      That also shouldn't be true in the latest npm, which uses a global node_modules and a bit of trickery for different versions of the same library, instead of recursive node_modules.
  - nostrademons 11y ago
    Babel seems to be a terrible offender as far as dependencies go. In my node_modules, babel-preset-es2015 takes up 200M - as you've noticed, each individual transformer has its own copy of the core Babel runtime. And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel. The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.
    - nilliams 11y ago
      Yes, if you only write JS for Chrome and Node, not sure how many JS devs fall into that bracket!
- ceejayoz 11y ago
  In many cases, a bunch of useless stuff. One of our devs added a material design webfont NPM package (IIRC, https://github.com/google/material-design-icons https://github.com/google/material-design-icons). Something like 40kb worth of font files, but the folder was 100+ MB because it included a PNGs of each icon at every imaginable resolution. I've seen others with 10+MB CSV files used by the unit tests.
  - jakub_g 11y ago
    In such case you should send them a pull request with a proper .npmignore file which contains whole of .gitignore plus rules to not publish those PNGs
arrakeen 11y ago
i've found 85M of the same library in my project: https://gist.github.com/bsdf/e0db8f5ac90fee5ccff8 https://gist.github.com/bsdf/e0db8f5ac90fee5ccff8
- deleted 11y ago
  [deleted]
mdavidn 11y ago
A note on your edit: The difference you observed is not because of du. Your node_modules really does take 673 MB on disk. File systems divide the disk into segments, confusingly also called blocks. Only one file may occupy a block. On my Mac, the block size is 4 KB, so a file containing a single byte will still occupy 4 KB on disk. The du utility reports the latter. The Mac Finder reports the former in column view and both in the "Get Info" pane. Unpacked NPM modules have many, many small files, obviously.
- tlrobinson 11y ago
  Ah, interesting, thanks.

bsimpson 11y ago

I believe this is common practice at firms that have a monorepo.

acemarke 11y ago

Re-re-posting a comment I've made in a few threads now, because I feel this needs to get more awareness: I'm still learning and prototyping my first JS/NPM/React project, but https://github.com/JamieMason/shrinkpack https://github.com/JamieMason/shrinkpack seems to me like it solves _most_ of the issues involved here. It pulls down all the tarballs, and updates the npm-shrinkwrap.json to point to those instead. That way you check in a much smaller "node_shrinkwrap" folder of a few hundred tarballs and 15-20MB, rather than a node_modules folder of 30K files and 150MB and a bunch of platform-specific build outputs. Still doesn't solve the issue of installing new/updated dependencies that might actually require pulling in something that vanished, but at least once you've done an install and run "shrinkpack", you've got everything you need to rebuild right there.

A1kmm 11y ago

I find most VCS software is not well suited to handling large immutable files (immutable in the sense there shouldn't be two different files with the same vendor, name, and exact version). With a distributed VCS, you probably don't want anyone doing a clone to have to download every version ever of every dependency. A binary repository like Nexus or Artifactory is very useful for this. To fix the npm instability problems, I use Nexus and configure it to proxy (and permanently cache) npmjs, and then use npm shrinkwrap.