6 ms·

Pasting a comment I made about this a while ago: The reason this keeps happening with NPM is because of absurd number of dependencies in the average node app.

by vlunkr 5y ago

Pasting a comment I made about this a while ago: The reason this keeps happening with NPM is because of absurd number of dependencies in the average node app. I have a tiny app I've been playing with using create-react-app. There are over 800 directories in node_modules. That absolutely dwarfs the number of any other language I've used. Even in a medium sized rails app, you likely have some awareness of what every dependency is. It's just impossible with npm. This makes it easier for someone to inject their package into the ecosystem whether it's actually very useful of not (like the colors package). One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, probably through manual review. Maybe it could be recursive, so that safe projects only rely on other safe projects.

zozbot234 5y ago
Package curation exists in other languages, too. C++ has Boost, and Haskell has its Haskell Platform. It helps avoid the pitfalls of languages with large standard libraries (where stability guarantees make "batteries included" turn into obsolete and bitrotting "dead batteries").
- regularfry 5y ago
  This is an idea that every ecosystem eventually realises it needs. Once you've got enough versions of enough libraries that A and B both need C but at different versions, you start to need curation, although that need might not become pressing enough to do anything about for a while. But once you've got curation, cryptographically trusting that curation becomes viable in a way that cryptographic trust of the original packages often isn't. Putting a layer of "distributions" over language ecosystems, in the same way that "distributions" solve the problem of putting enough mutually-compatible library versions together to get Linux to work, is, I think, inevitable.
  - ljm 5y ago
    Another aspect I think is, how much open source dependency do you actually need for your project. You can probably get by with fewer packages than you expect, which also makes it more feasible to run a curated package system. For example, you can't submit any old Haskell library you put together to Hackage, or you couldn't at the last time I checked. It had to meet certain minimum standards to be considered. I honestly don't know where you would start in terms of curating NPM, precisely because of the dependency on dependencies. You'd end up curating half of the ecosystem.
    - zozbot234 5y ago
      > For example, you can't submit any old Haskell library you put together to Hackage You can, but you need to figure out what your library is doing and translate it to category theory.
  - derefr 5y ago
    > Once you've got enough versions of enough libraries that A and B both need C but at different versions, you start to need curation Node specifically doesn’t have this constraint, though, as A’s C and B’s C can be loaded independently into the same VM, each one hermetic and only visible to its parent. (This is probably half of why Node’s ecosystem became the way it did, now that I think about it; every other ecosystem hits increasing numbers of constraint-resolution conflict problems as dependency hierarchy depth increases, and so limits itself in hierarchy depth to avoid this.)
    - zozbot234 5y ago
      > Node specifically doesn’t have this constraint, though, as A’s C and B’s C can be loaded independently into the same VM, each one hermetic and only visible to its parent. Rust can do this as well, though most version variation is handled via semver rules wrt. compatibility. I think this exact requirement led to some controversy in the Go community at some point? Though they should now have a module system that allows for this?
    - regularfry 5y ago
      It's less of a problem, but it bites as soon as you need to pass a data structure from one C to the other. That's less frequent, but does happen.
- vlunkr 5y ago
  Cool, I'm glad there's precedent for it. I'm not opposed to adding guardrails as the post suggests, but it feels like a band-aid on much deeper problem.
- dimitrios1 5y ago
  Then you have languages like Go, where it feels like the norm is relying on the standard lib, and any dependency really must be providing value to justify it. My node projects typically have thousands of dependencies, my java and C++ projects have hundreds. My go projects typically have 8-12, and have never once exceeded 40. Not to stan the Go language itself, I just wish this philosophy was the prevalent one in languages.
Zababa 5y ago
> That absolutely dwarfs the number of any other language I've used. Rust has this flaw too. Last time I wanted to compile a simple project using actix-web and probably a database library, I had 200 crates to compile. In both cases I think it's due partially to preferring small packages/crates.
- zozbot234 5y ago
  It's ultimately Rust's reliance on generic code that forces us to deal with so many packages at build time. In idiomatic C/C++ much of the reusable code you compile ultimately turns into reusable shared objects, that sit in binary form on your fs with their corresponding include files. This is only possible in rare cases with Rust, because the "library" equivalents (crates) don't know how their generic types and code will be instantiated downstream. Everything is the equivalent of a "header only" lib.
- geodel 5y ago
  Yeah, in language design they might be on cutting edge. But in software engineering they want to be close to 90's, 00's cultish software design patterns, DRY principle and so on.
- deaddabe 5y ago
  Note that this separation is necessary if you want to achieve somewhat parallel compilation in Rust. Every proc_macro needs its own separate crate, so I end up providing two crates for macro-based stuff. At least, the stdlib seems to have enough batteries included for the dependency count to not be as high as NodeJS projects. From my experience I would rate the count of dependencies in this order: NodeJS > Rust > Python > C/C++ Though I cannot explain why this is ordered this way between {Node, Rust, Python}. Again, batteries included? Language popularity? Beginners-friendly programming language?
  - zozbot234 5y ago
    AIUI, rustc now has ad-hoc parallelization of compile within a single crate. The defined compilation unit is still the crate though, as opposed to the single file in C/C++.
ragnese 5y ago
> The reason this keeps happening with NPM is because of absurd number of dependencies in the average node app. But why does that happen? There are now lots of languages that make it trivial to add dependencies. While I find projects in those other languages to also have too many dependencies, it's no where near what happens in JS apps. I'm thinking of projects I've recently worked on in Rust, PHP, and Java. Java projects seem to be a distant second-place to JavaScript projects when it comes to willy-nilly dependencies. It's not a rhetorical question: Why is the culture with JS so much worse about this? I absolutely hate that I'm going to suggest this, but is it just because of the average skill level and experience of people working on JS projects? Or is it because JS is such a bug-prone programming language that we're all afraid of actually authoring any more code than we absolutely have to, because we know we'll waste hours debugging things that should be relatively simple? I honestly don't know.
- CamouflagedKiwi 5y ago
  The process of uploading packages onto Maven Central is... not modern. npm (arguably?) makes this a lot easier than any other language, therefore npm developers do more of it.
- yoz-y 5y ago
  I think it's the sheer amount of programmers using JS. It's also a very approachable language, so it makes it easy to learn the language before learning good practices.
- dschooh 5y ago
  This may not be the whole story, but one of the reasons is that JavaScript does not have a standard library. Corollaries are: several different module systems, application frameworks and bundlers exist.
  - lucideer 5y ago
    Node does have a standard library. I've seen this same weird line parroted by so many people in the comments here: it's very odd. If you're talking about server & browser having different APIs that's equivalent to any other language having different platform apis.
davman 5y ago
I think curated packages is part of what https://tidelift.com/ https://tidelift.com/ is trying to achieve.
deleted 5y ago
[deleted]
dgb23 5y ago
Can we determine whether the package/LoC ratio actually has the quality impact that people think it has? Or dependencies/project? Because I don't think those are useful metrics at all. In fact this is just code re-use and should probably be happening to a significant degree. Some actually interesting properties are: - basic performance - signal to noise ratio - API stability - robustness - clarity and simplicity... And more of that. But just saying "a lot of dependencies bad" is not useful. Maybe in an ideal world we have even _more_ dependencies? I don't know. What I do know is that I care about the above points much more.
- vlunkr 5y ago
  > And more of that. But just saying "a lot of dependencies bad" is not useful. I disagree. The primary problem is that proper dependency auditing is incredibly time-consuming, especially if you want to stay up-to-date. The reality is that most people simply don't do it all.
adolph 5y ago
In the below graph, create-react-app has 66 package dependencies and 88 links between packages. The purpose of the graph isn't to disagree with the statement of "800 directories" but to illustrate that part of the problem is depth. The maximum depth (in my hand count) is seven. https://npm.anvaka.com/#/view/2d/create-react-app https://npm.anvaka.com/#/view/2d/create-react-app

by vlunkr 5y ago

zozbot234 5y ago

Package curation exists in other languages, too. C++ has Boost, and Haskell has its Haskell Platform. It helps avoid the pitfalls of languages with large standard libraries (where stability guarantees make "batteries included" turn into obsolete and bitrotting "dead batteries").

regularfry 5y ago
This is an idea that every ecosystem eventually realises it needs. Once you've got enough versions of enough libraries that A and B both need C but at different versions, you start to need curation, although that need might not become pressing enough to do anything about for a while. But once you've got curation, cryptographically trusting that curation becomes viable in a way that cryptographic trust of the original packages often isn't. Putting a layer of "distributions" over language ecosystems, in the same way that "distributions" solve the problem of putting enough mutually-compatible library versions together to get Linux to work, is, I think, inevitable.
- ljm 5y ago
  Another aspect I think is, how much open source dependency do you actually need for your project. You can probably get by with fewer packages than you expect, which also makes it more feasible to run a curated package system. For example, you can't submit any old Haskell library you put together to Hackage, or you couldn't at the last time I checked. It had to meet certain minimum standards to be considered. I honestly don't know where you would start in terms of curating NPM, precisely because of the dependency on dependencies. You'd end up curating half of the ecosystem.
  - zozbot234 5y ago
    > For example, you can't submit any old Haskell library you put together to Hackage You can, but you need to figure out what your library is doing and translate it to category theory.
- derefr 5y ago
  > Once you've got enough versions of enough libraries that A and B both need C but at different versions, you start to need curation Node specifically doesn’t have this constraint, though, as A’s C and B’s C can be loaded independently into the same VM, each one hermetic and only visible to its parent. (This is probably half of why Node’s ecosystem became the way it did, now that I think about it; every other ecosystem hits increasing numbers of constraint-resolution conflict problems as dependency hierarchy depth increases, and so limits itself in hierarchy depth to avoid this.)
  - zozbot234 5y ago
    > Node specifically doesn’t have this constraint, though, as A’s C and B’s C can be loaded independently into the same VM, each one hermetic and only visible to its parent. Rust can do this as well, though most version variation is handled via semver rules wrt. compatibility. I think this exact requirement led to some controversy in the Go community at some point? Though they should now have a module system that allows for this?
  - regularfry 5y ago
    It's less of a problem, but it bites as soon as you need to pass a data structure from one C to the other. That's less frequent, but does happen.
vlunkr 5y ago
Cool, I'm glad there's precedent for it. I'm not opposed to adding guardrails as the post suggests, but it feels like a band-aid on much deeper problem.
dimitrios1 5y ago
Then you have languages like Go, where it feels like the norm is relying on the standard lib, and any dependency really must be providing value to justify it. My node projects typically have thousands of dependencies, my java and C++ projects have hundreds. My go projects typically have 8-12, and have never once exceeded 40. Not to stan the Go language itself, I just wish this philosophy was the prevalent one in languages.

Zababa 5y ago

> That absolutely dwarfs the number of any other language I've used. Rust has this flaw too. Last time I wanted to compile a simple project using actix-web and probably a database library, I had 200 crates to compile. In both cases I think it's due partially to preferring small packages/crates.

zozbot234 5y ago
It's ultimately Rust's reliance on generic code that forces us to deal with so many packages at build time. In idiomatic C/C++ much of the reusable code you compile ultimately turns into reusable shared objects, that sit in binary form on your fs with their corresponding include files. This is only possible in rare cases with Rust, because the "library" equivalents (crates) don't know how their generic types and code will be instantiated downstream. Everything is the equivalent of a "header only" lib.
geodel 5y ago
Yeah, in language design they might be on cutting edge. But in software engineering they want to be close to 90's, 00's cultish software design patterns, DRY principle and so on.
deaddabe 5y ago
Note that this separation is necessary if you want to achieve somewhat parallel compilation in Rust. Every proc_macro needs its own separate crate, so I end up providing two crates for macro-based stuff. At least, the stdlib seems to have enough batteries included for the dependency count to not be as high as NodeJS projects. From my experience I would rate the count of dependencies in this order: NodeJS > Rust > Python > C/C++ Though I cannot explain why this is ordered this way between {Node, Rust, Python}. Again, batteries included? Language popularity? Beginners-friendly programming language?
- zozbot234 5y ago
  AIUI, rustc now has ad-hoc parallelization of compile within a single crate. The defined compilation unit is still the crate though, as opposed to the single file in C/C++.

ragnese 5y ago

> The reason this keeps happening with NPM is because of absurd number of dependencies in the average node app. But why does that happen? There are now lots of languages that make it trivial to add dependencies. While I find projects in those other languages to also have too many dependencies, it's no where near what happens in JS apps. I'm thinking of projects I've recently worked on in Rust, PHP, and Java. Java projects seem to be a distant second-place to JavaScript projects when it comes to willy-nilly dependencies. It's not a rhetorical question: Why is the culture with JS so much worse about this? I absolutely hate that I'm going to suggest this, but is it just because of the average skill level and experience of people working on JS projects? Or is it because JS is such a bug-prone programming language that we're all afraid of actually authoring any more code than we absolutely have to, because we know we'll waste hours debugging things that should be relatively simple? I honestly don't know.

CamouflagedKiwi 5y ago
The process of uploading packages onto Maven Central is... not modern. npm (arguably?) makes this a lot easier than any other language, therefore npm developers do more of it.
yoz-y 5y ago
I think it's the sheer amount of programmers using JS. It's also a very approachable language, so it makes it easy to learn the language before learning good practices.
dschooh 5y ago
This may not be the whole story, but one of the reasons is that JavaScript does not have a standard library. Corollaries are: several different module systems, application frameworks and bundlers exist.
- lucideer 5y ago
  Node does have a standard library. I've seen this same weird line parroted by so many people in the comments here: it's very odd. If you're talking about server & browser having different APIs that's equivalent to any other language having different platform apis.

davman 5y ago

I think curated packages is part of what https://tidelift.com/ https://tidelift.com/ is trying to achieve.

deleted 5y ago

[deleted]

dgb23 5y ago

Can we determine whether the package/LoC ratio actually has the quality impact that people think it has? Or dependencies/project? Because I don't think those are useful metrics at all. In fact this is just code re-use and should probably be happening to a significant degree. Some actually interesting properties are: - basic performance - signal to noise ratio - API stability - robustness - clarity and simplicity... And more of that. But just saying "a lot of dependencies bad" is not useful. Maybe in an ideal world we have even _more_ dependencies? I don't know. What I do know is that I care about the above points much more.

vlunkr 5y ago
> And more of that. But just saying "a lot of dependencies bad" is not useful. I disagree. The primary problem is that proper dependency auditing is incredibly time-consuming, especially if you want to stay up-to-date. The reality is that most people simply don't do it all.

adolph 5y ago

In the below graph, create-react-app has 66 package dependencies and 88 links between packages. The purpose of the graph isn't to disagree with the statement of "800 directories" but to illustrate that part of the problem is depth. The maximum depth (in my hand count) is seven. https://npm.anvaka.com/#/view/2d/create-react-app https://npm.anvaka.com/#/view/2d/create-react-app