5 ms·

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses ex

by cedws 2y ago

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

MortyWaves 2y ago
Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.
Alifatisk 2y ago
At least, switch to pnpm minimize the bloat
- jefozabuss 2y ago
  I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/
  - jeffhuys 2y ago
    pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.
    - syncsynchalt 2y ago
      You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.
- MBCook 2y ago
  As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version? I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping. Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.
vinnymac 2y ago
You might be interested in e18e if you would like to see that change: https://e18e.dev/ https://e18e.dev/ They’ve done a lot of great work already.
- KTibow 2y ago
  Does this replace ljharb stuff?
hinkley 2y ago
I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used. Now if rxjs weren’t a dumpster fire…
TheRealPomax 2y ago
That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM. (And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)
- silverwind 2y ago
  It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish. On a typical project with a build step, only a `dist` folder would published.
  - TheRealPomax 2y ago
    Not a fan of that one myself (it's far easier to tell what doesn't belong in a package vs. what does belong in a package) but that option does exist, so as a maintainer you really have no excuse, and as a user you have multiple MR/PRs that you can file to help them fix their cruft. > On a typical project with a build step, only a `dist` folder would published. Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.
eitau_1 2y ago
It's not even funny: $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/* /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux: .r-xr-xr-x 129k root 1 Jan 1970 xsel /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows: .r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe .r-xr-xr-x 331k root 1 Jan 1970 clipboard_x86_64.exe (clipboardy ships executables and none of them can be run on NixOS btw)
- cedws 2y ago
  Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.
  - eitau_1 2y ago
    The executables are vendored in the repo [0]. [0] https://github.com/sindresorhus/clipboardy/tree/main/fallbacks https://github.com/sindresorhus/clipboardy/tree/main/fallbac...
- dicytea 2y ago
  I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages. I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited. I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.
hombre_fatal 2y ago
One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts. You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes. If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.
tanepiper 2y ago
Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.

by cedws 2y ago

MortyWaves 2y ago

Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.

Alifatisk 2y ago

At least, switch to pnpm minimize the bloat

jefozabuss 2y ago
I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/
- jeffhuys 2y ago
  pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.
  - syncsynchalt 2y ago
    You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.
MBCook 2y ago
As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version? I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping. Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.

vinnymac 2y ago

You might be interested in e18e if you would like to see that change: https://e18e.dev/ https://e18e.dev/ They’ve done a lot of great work already.

KTibow 2y ago
Does this replace ljharb stuff?

hinkley 2y ago

I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used. Now if rxjs weren’t a dumpster fire…

TheRealPomax 2y ago

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM. (And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)

silverwind 2y ago
It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish. On a typical project with a build step, only a `dist` folder would published.
- TheRealPomax 2y ago
  Not a fan of that one myself (it's far easier to tell what doesn't belong in a package vs. what does belong in a package) but that option does exist, so as a maintainer you really have no excuse, and as a user you have multiple MR/PRs that you can file to help them fix their cruft. > On a typical project with a build step, only a `dist` folder would published. Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.

eitau_1 2y ago

It's not even funny: $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/* /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux: .r-xr-xr-x 129k root 1 Jan 1970 xsel /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows: .r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe .r-xr-xr-x 331k root 1 Jan 1970 clipboard_x86_64.exe (clipboardy ships executables and none of them can be run on NixOS btw)

cedws 2y ago
Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.
- eitau_1 2y ago
  The executables are vendored in the repo [0]. [0] https://github.com/sindresorhus/clipboardy/tree/main/fallbacks https://github.com/sindresorhus/clipboardy/tree/main/fallbac...
dicytea 2y ago
I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages. I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited. I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.

hombre_fatal 2y ago

One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts. You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes. If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.

tanepiper 2y ago

Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.