7 ms·
NPMprune: Remove unnecessary files from node_modules to optimize storage
- lxe 3y agoWe did exactly this when packaging and deploying large node manifests at one of my former companies. Be super careful of removing large swaths of files. Out of 150,000 node modules in your manifest, I'm willing to bet at least one of them is doing something by reading one of these non-source files.
- sbarre 3y agoThis was my concern as well. Looking at the script source, it's just matching globs, so there isn't much smarts to this. I'm sure it works most of the time, but yeah.. Do JS packages need some kind of .prodignore file similar to other .ignore files? So with a flag passed, after doing an npm install, there's a extra cleanup step that removes explicitly marked files that aren't needed for running in prod? (Not a fully formed idea, I'm sure I'm not thinking of drawbacks with this) Edit: this sort of exists as the .npmignore file? https://docs.npmjs.com/cli/v10/using-npm/developers#keeping-files-out-of-your-package https://docs.npmjs.com/cli/v10/using-npm/developers#keeping-...
- rezonant 3y ago.npmignore is the opposite of `files`, it omits files when creating the package itself, that's different than trimming the files when the package is installed. That said, files/npmignore is the correct way to deal with this and you should never remove files from the packages you install without extremely good reasons, and when you do it, it should be very narrowly scoped and handled automatically as part of npm install. It should be totally valid to delete node_modules and reinstall everything without causing problems. This is also the biggest reason to never commit node_modules, aside from the pure insanity of commiting hundreds of thousands of vendor managed files and inviting merge conflicts when two branches change those files...
- QuadmasterXLII 3y agoTest and bifurcate I guess.
- lxe 3y agoIn production, preferably. This way you'll immediately find any issues and will have top priority allocated to fixing them.
- armchairhacker 3y agoI use https://pnpm.io https://pnpm.io whenever possible. It has many benefits, the main one is that the node modules are symlinked to one big repo in your home directory, so there isn’t nearly as much duplication.
- rpastuszak 3y agoYup, same here. I’ve saved 40gb of data by recursively removing all node_modules directories from my Mac and replacing npm with pnpm. I did notice small issues with some libraries (react testing library IIRC)
- paulddraper 3y ago*hard linked
- akoboldfrying 3y agoCurious whether this works on Windows? Symlinks are strange there (though hard links work fine on NTFS).
- armchairhacker 3y agoIt works. I think the only issue is that you may need to run `pnpm.exe` instead of `pnpm` (or I just installed it incorrectly. Installing on Windows is a bit tougher, but I haven't experienced issues with the symlinks).
- otteromkram 3y agoIf you don't want to risk adding something nefarious to your Google history, DuckDuckGo is a nice alternative. As my ol' grandpappy used to say, "Why wonder? Let's go search the Internets!"
- akoboldfrying 3y agoI'm not sure how to interpret this, other than as an elaborately snarky way of saying "Why don't you google the answer yourself, you lazy bum?"
- swatcoder 3y ago> In deployment scripts: > > wget -qO- https://raw.githubusercontent.com/xthezealot/npmprune/master/npmprune.sh https://raw.githubusercontent.com/xthezealot/npmprune/master... | sh -- -p Serious question: Is this the norm now? Are people actually executing unversioned wget'd shell scripts from random github users as part of their deployment workflow?
- petesergeant 3y ago> now For about the last 15 years
- nailer 3y agoThe threat model is exactly the same as executing untrusted, uninspected content you've downloaded locally. I could do some tricks where I sent different files based on user agent, but still... most people aren't inspecting the download anyway before running it.
- im3w1l 3y ago> I could do some tricks where I sent different files based on user agent Not from githubusercontent you couldn't. Which I'd say is where the majority of these scripts are hosted.
- bravetraveler 3y agoIt's been demonstrated that the server can tell if the client is piping or not https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b... I'm on mobile and this refuses to load due to SSL/HSTS problems. It's an interesting approach if you can get it to load. Point being... someone clever could make a bad day for the minority of users who do check first (but don't save/run what they explicitly checked) Running curl, saying "yup that looks good", then adding a pipe achieves little with actual malice. The payload can be completely hidden until it's too late The phrasing you use is important for any hope at safety - review/run the same downloaded thing.
- c0n5pir4cy 3y agoJust for package authors (or people looking for some easy pull requests) out there that might not know this exists. NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#files https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f.... This also extends to an .npmignore file that works similar to a .gitignore file.
- creatonez 3y agoJust beware that some files may seem unnecessary but are expected from an idiomatic npm package. Three things that come to mind -- a markdown file named README.md, any generated typescript definitions, and typescript/babel sourcemaps. And something I've seen far too often: please don't give a minified, rolled up bundle as the only option, otherwise you are chucking your library's users back into the dark ages of Bower.js.
- nfriedly 3y agoThe readme gets included automatically even if you don't specify it in the files field, ditto for the changelog license, and package.json. Compare https://github.com/express-rate-limit/express-rate-limit/blob/v7.1.5/package.json#L50-L53 https://github.com/express-rate-limit/express-rate-limit/blo... to https://www.npmjs.com/package/express-rate-limit?activeTab=code https://www.npmjs.com/package/express-rate-limit?activeTab=c... Agree with you about the other points.
- rezonant 3y agoStrongly agree on minification. You should not minify or bundle anything in your NPM package. That decision should only be made by the top level project if it wishes.
- josephg 3y agoYep. And if you’re writing typescript, please include type definitions, source maps, type definition source maps and the original typescript source. Having all of this stuff makes it possible to ctrl+click on functions in my libraries and read the corresponding source code. That’s a godsend during development - well worth a few extra kb of files in the npm module. tsconfig.json: "declaration": true, "declarationMap": true, "sourceMap": true, ... package.json (assuming typescript compiles src/ to dist/): "files": [ "dist/*", "src/*" ],
- an_ko 3y agoThis is wildly unsafe. - Some packages contain non-JS files for good reasons, and they may break in subtle unpredictable ways when you mess with the contents of their package. - Node.js will happily run JavaScript files even if they're not "*.js": A file like "hello.alsdfhlshdfl" works just fine as long as its content parses. There is no guarantee that your dependencies (and their recursive dependencies) don't statically or dynamically load files with completely arbitrary filenames. - If you distribute packages with license files stripped this way, you are violating licenses that require the license to be distributed along with the code. If this is actually a major issue for you, consider instead sending PRs to upstream to tidy up their package. This will also benefit other users.
- arthurwhite 3y ago- Of course, this entails the risk of occasional breakage. But for 99% of modules, this has no impact at runtime. - The patterns used to find files are specific enough to target only those files that are well known to be useless at runtime. - The license texts of these libraries can be copied and merged into a main LICENSE file. - Have you seen the number of modules installed by most major libraries? Making a pull request for each of them is humanly impossible and counter-productive. It's easier to use a simple script that releases dozens of MB in a few seconds.
- swatcoder 3y ago> But for 99% of modules, this has no impact at runtime. Traditionally, this wasn't an acceptable way to think about projects we engineers were being paid lots of money to build. As you note, a project may hoover in some absurd number of dependent libraries and you have no tooling that tells you which of those might fall in the 1% and what code paths in those 1% intersect with call stacks in your project. You have no idea what impact blindly deleting some "They're probably unnecessary" files in somebody else's code will have on your application and no insight into how to make sure your testing unearths problems. It's an invitation to phantom bugs of unknown scope and the most frustrating kind of debugging effort that comes from chasing those kinds of phantoms. It's already bad enough that people don't read and review their dependent code with the eye they bring to PR's from their on-team colleagues, but to then go futzing around and deleting things in the unread dependencies because you have a hunch that it's no big deal is about as far from software engineering as you can get.
- pavlov 3y agoWhy not use yarn? It has a much more reliable solution: https://yarnpkg.com/features/pnp https://yarnpkg.com/features/pnp
- arthurwhite 3y agoBecause its primary focus is on redefining how dependencies are stored and accessed, rather than modifying the contents of these dependencies. Useless files will still be there. Also, when you create a Docker image, you avoid packing in dev tools that aren't absolutely essential (such as Yarn).
- butshouldyou 3y agoFYI: The default node Docker images already include yarn.
- Alifatisk 3y agoJust use pnpm
- arthurwhite 3y agoWhile pnpm optimizes storage and reduces duplication, it does not inherently remove non-essential files (like documentation, Markdown, or test files) within the dependencies. Also, when you create a Docker image, you avoid packing in dev tools that aren't absolutely essential (such as pnpm).
- nusmella 3y agoEven the alpine nodejs images have pnpm and yarn nowadays
- dlrush 3y agoJust use PNPM
- Ayesh 3y agoYeah no. Npm already does it at the package registry with ignore/npmignore files, and that's the package authors choice. How much storage can you really save? 50MB? 200MB? is it really worth the risk of running rm on some glob pattern and cross your fingers the packages don't require any of the deleted files?
- arthurwhite 3y agoNot everyone uses the .npmignore file. Maybe it's the author's choice, but in the meantime, that's my personal storage space that's being used unnecessarily. I tested it recently on a clean install of Strapi: about 250 MB are freed up. Storage is cheap but that still represents a lot, especially inside a Docker image. The patterns used to find files are specific enough to target only those files that are well known to be useless at runtime.
- rezonant 3y agoOthers have pointed out that you have no idea which files are useless at runtime when inspecting their filename. Executable JS does not need the .js extension to be loaded by Node.js or any other runtime environment, and on server runtimes files can be read at runtime, so JSON files, markdown files, webassembly modules, or any other kind of non-JS content can have a runtime impact. You are taking a big risk of subtle breakage right now, and a big risk of breakage as you change your project code in the future, as you may start to invoke a code path that needs that resource in the future.
- dmitrygr 3y ago> Remove unnecessary [...] node_modules Try this bash one-liner :) find / -name node_modules -print0 | xargs -0 rm -rf
- leipert 3y agoYarn@1 has this autoclean feature: https://classic.yarnpkg.com/en/docs/cli/autoclean https://classic.yarnpkg.com/en/docs/cli/autoclean I used to use it, but at some point the hassle of the occasionally breaking package wasn’t worth it.
- mirekrusin 3y agoJust bundle your b/e production entrypoint as a single js file similar to f/e.
- joshmanders 3y agoCan someone explain to me why this is even necessary? I have at the time of this comment, 32 node projects on my machine, all with their own node_modules, and I'm using less than 200GB (total, including everything else on my machine) of my total 1TB hard drive space... Are people that concerned about the size of a directory on their machines?
- arthurwhite 3y agoMaybe not on their machine, but for Docker images that will be pulled a thousand times, yes.
- demomode 3y agoThere's so many things wrong with this stuff, from the motivation of the project to the (dangerous) implementation