6 ms·

Has npm finally figured out how to de-dupe dependencies? In one project, I have something like 47 copies of the same version of the same library, distributed a

by douche 10y ago

Has npm finally figured out how to de-dupe dependencies? In one project, I have something like 47 copies of the same version of the same library, distributed at all levels of the node_modules hierarchy. I try not to think about that JS tooling too hard, lest I start pulling my hair out and devolve into a screaming crazy person.

rplnt 10y ago
This sucks so hard. It's also quite easy to hit path length limits for certain operations (on windows) when you have this mess there.
- dgorges 10y ago
  Totally. We just ran into this issue. Running npm install inside a linux vm resulted in an endless loop, because the nfs mapping created path names that exceeded the length limit.
  - ishener 10y ago
    it's like an entire eco-system is fundamentally broken
    - fahrradflucht 10y ago
      Maybe, but it's not like there is no way of solving that issue. It's just that this problem can't be solved entirely by npm. Node has to make changes to it's module loader as well.
  - throwanem 10y ago
    NPM 3 tries, not entirely without success, to flatten the tree under node_modules/; I've had good results using it to resolve the kind of path length issues (in this case, with NTFS directories mounted in a VM) that you describe. Might be worth a look in your case as well.
- SquareWheel 10y ago
  This used to be the case. In npm3 and beyond it installs dependencies in a flat way, which seems to have solved the Windows file length limit problem.
Touche 10y ago
It's actually not possible due to the way Node looks up dependencies. You could have a situation like: node_modules/ a/ node_modules/ c/ (1.0.0) b/ node_modules/ c/ (1.0.0) c/ (2.0.0) Both a and b depend on c version 1.0.0, but since there's a version 2.0.0 in the root node_modules folder c can't be placed there, and has to be duplicated in a and b's own node_modules folder, otherwise Node couldn't find it for each of them.
- rplnt 10y ago
  You could use links, version in names, ... You'd just have problem to take out one submodule I guess, but I doubt that's a use case anywhere.
- ksherlock 10y ago
  It's entirely possible with symbolic links. https://github.com/rstacruz/pnpm https://github.com/rstacruz/pnpm
- RossM 10y ago
  I've built a fairly hacky solution to this before (for a different package manager) - it can be pretty simple: node_modules/ versions/ a@1.0.0 node_modules/ c -> ../../c@1.0.0 b@1.0.0 node_modules/ c -> ../../c@1.0.0 c@1.0.0/ c@2.0.0/ a -> versions/a@1.0.0 b -> versions/b@1.0.0 c -> versions/c@2.0.0
  - Touche 10y ago
    Some other people have linked to npm alternatives ied and pnpm which do essentially this.
DCoder 10y ago
npm v3 tries to dedupe packages during installation to reduce this problem. However, it still does not install packages in a deterministic way... [0]: https://docs.npmjs.com/how-npm-works/npm3-dupe https://docs.npmjs.com/how-npm-works/npm3-dupe [1]: https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet
k__ 10y ago
https://github.com/alexanderGugel/ied https://github.com/alexanderGugel/ied
- aidos 10y ago
  I was just about to write a comment to ask why it can't be done like this (using CAS / symlinks). So I guess it can. Are there any disadvantages to using ied over npm?
  - Klathmon 10y ago
    It's not fully stable yet. I tried a while ago, and after running into 3 showstoping bugs (and filing bugreports) i had to give up. Great idea though, and it probably has improved a bunch since then. This is one of the reasons why I love the JS ecosystem. You can even choose among a few tools you use to install packages from a package manager.
  - k__ 10y ago
    Private modules seem to be an issue at the moment, but a PR for this is pending. I used it successfully in a project and it even worked with native modules.
  - paradite 10y ago
    It does not support external packages from git remote url (github).
    - paradite 10y ago
      Update: Looks like it is supported now: https://github.com/alexanderGugel/ied/issues/7#issuecomment-237116655 https://github.com/alexanderGugel/ied/issues/7#issuecomment-...
ivan_gammel 10y ago
To me it looks like a silly architectural mistake made by NPM/Node developers, considering that there's already pretty good dependency management solution on the market that does the things right. Maven repositories have following structure, that allows to avoid duplication and take versions into account: /<vendor namespace>/<library>/<version>/<library artifact.ext> Vendor namespace itself is hierarchical and usually related to domain name, e.g. "com/mycompany/myapp". No idea, why this approach is not yet used in JS world (except the webjars), but it's high time to fix it this way.
guigar 10y ago
Yes, that's already solved with npm 3.

by douche 10y ago

rplnt 10y ago

This sucks so hard. It's also quite easy to hit path length limits for certain operations (on windows) when you have this mess there.

dgorges 10y ago
Totally. We just ran into this issue. Running npm install inside a linux vm resulted in an endless loop, because the nfs mapping created path names that exceeded the length limit.
- ishener 10y ago
  it's like an entire eco-system is fundamentally broken
  - fahrradflucht 10y ago
    Maybe, but it's not like there is no way of solving that issue. It's just that this problem can't be solved entirely by npm. Node has to make changes to it's module loader as well.
- throwanem 10y ago
  NPM 3 tries, not entirely without success, to flatten the tree under node_modules/; I've had good results using it to resolve the kind of path length issues (in this case, with NTFS directories mounted in a VM) that you describe. Might be worth a look in your case as well.
SquareWheel 10y ago
This used to be the case. In npm3 and beyond it installs dependencies in a flat way, which seems to have solved the Windows file length limit problem.

Touche 10y ago

It's actually not possible due to the way Node looks up dependencies. You could have a situation like: node_modules/ a/ node_modules/ c/ (1.0.0) b/ node_modules/ c/ (1.0.0) c/ (2.0.0) Both a and b depend on c version 1.0.0, but since there's a version 2.0.0 in the root node_modules folder c can't be placed there, and has to be duplicated in a and b's own node_modules folder, otherwise Node couldn't find it for each of them.

rplnt 10y ago
You could use links, version in names, ... You'd just have problem to take out one submodule I guess, but I doubt that's a use case anywhere.
ksherlock 10y ago
It's entirely possible with symbolic links. https://github.com/rstacruz/pnpm https://github.com/rstacruz/pnpm
RossM 10y ago
I've built a fairly hacky solution to this before (for a different package manager) - it can be pretty simple: node_modules/ versions/ a@1.0.0 node_modules/ c -> ../../c@1.0.0 b@1.0.0 node_modules/ c -> ../../c@1.0.0 c@1.0.0/ c@2.0.0/ a -> versions/a@1.0.0 b -> versions/b@1.0.0 c -> versions/c@2.0.0
- Touche 10y ago
  Some other people have linked to npm alternatives ied and pnpm which do essentially this.

DCoder 10y ago

npm v3 tries to dedupe packages during installation to reduce this problem. However, it still does not install packages in a deterministic way... [0]: https://docs.npmjs.com/how-npm-works/npm3-dupe https://docs.npmjs.com/how-npm-works/npm3-dupe [1]: https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet

k__ 10y ago

https://github.com/alexanderGugel/ied https://github.com/alexanderGugel/ied

aidos 10y ago
I was just about to write a comment to ask why it can't be done like this (using CAS / symlinks). So I guess it can. Are there any disadvantages to using ied over npm?
- Klathmon 10y ago
  It's not fully stable yet. I tried a while ago, and after running into 3 showstoping bugs (and filing bugreports) i had to give up. Great idea though, and it probably has improved a bunch since then. This is one of the reasons why I love the JS ecosystem. You can even choose among a few tools you use to install packages from a package manager.
- k__ 10y ago
  Private modules seem to be an issue at the moment, but a PR for this is pending. I used it successfully in a project and it even worked with native modules.
- paradite 10y ago
  It does not support external packages from git remote url (github).
  - paradite 10y ago
    Update: Looks like it is supported now: https://github.com/alexanderGugel/ied/issues/7#issuecomment-237116655 https://github.com/alexanderGugel/ied/issues/7#issuecomment-...

ivan_gammel 10y ago

To me it looks like a silly architectural mistake made by NPM/Node developers, considering that there's already pretty good dependency management solution on the market that does the things right. Maven repositories have following structure, that allows to avoid duplication and take versions into account: /<vendor namespace>/<library>/<version>/<library artifact.ext> Vendor namespace itself is hierarchical and usually related to domain name, e.g. "com/mycompany/myapp". No idea, why this approach is not yet used in JS world (except the webjars), but it's high time to fix it this way.

guigar 10y ago

Yes, that's already solved with npm 3.