7 ms·

As a C++ developer I can’t imagine 1700 dependencies in a relatively new code base. What are all those libraries doing?

canadev 8y ago
I think the answer is everything in your standard library plus some of the stuff you would use 3rd party dependencies for in your own project. Maybe standard libraries need to get bigger for JS.
- slededit 8y ago
  Even in C with its tiny standard library you don’t see this kind of explosion. There has to be more to it than that.
  - bfred_it 8y ago
    npm handles sub-dependencies with ease so people aren't afraid to pull in smaller dependencies. Also since the code has to be downloaded (or uploaded to small lambdas, for example), JS developers are generally weary of depending on large libraries.
    - krapp 8y ago
      >Also since the code has to be downloaded (or uploaded to small lambdas, for example), JS developers are generally weary of depending on large libraries. Isn't a dependency tree of thousands of small, interdependent libraries essentially just a large, distributed library? Is there really a benefit to that versus compiling those dependencies to a single file, especially given how many NPM packages are just a single small function? The end result in many cases would probably be smaller than a lot of old JQuery plugins.
      - eeZah7Ux 8y ago
        > Isn't a dependency tree of thousands of small, interdependent libraries essentially just a large, distributed library? Yes and it's a thousand times worse than a single, large library.
  - cyphar 8y ago
    The problem is that it is far too easy to publish a new library that contains a single function. Due to history and other considerations, this is much harder in C (you have to wrangle CMake or autotools, and then package your library for distributions). While I do think making things easier for developers is a good goal, making the wrong things too easy (that is, making libraries that become very heavily depended on) results in the problems of Node.js (and I would argue the same is true for Rust). Maybe I'm just a curmudgeon, but I really have a sour taste in my mouth when I look at how Node.js and Rust do library management -- it's making it too easy to make a small library which doesn't really work and then people depend on it. I once tried to write a simple IMAP cloning utility in Rust and found 4 IMAP crates -- none of them worked properly and all of them incorrectly handled several core parts of the IMAPv4 spec. I figured out later that they all appeared to be forks of one another (or they copied each others' APIs), but that doesn't help matters -- why are forked crates taking up more space in the global crate namespace? The "imap" crate doesn't implement IMAP properly! Python had this problem (in a lesser degree) too with pip, but I think having a larger standard library and lots of time to mature allowed them to overcome it.
    - mcguire 8y ago
      "I once tried to write a simple IMAP cloning utility in Rust and found 4 IMAP crates -- none of them worked properly and all of them incorrectly handled several core parts of the IMAPv4 spec. I figured out later that they all appeared to be forks of one another (or they copied each others' APIs), but that doesn't help matters -- why are forked crates taking up more space in the global crate namespace? The "imap" crate doesn't implement IMAP properly!" I noticed this problem with Perl and CPAN: there were dozens (well, several) email-sending packages, none of which were near complete. (Speaking SMTP is easy, an actual MTA is hard.) I suspect there are similar issues in Java-land, although I haven't used Maven-derivatives enough to find out. It's the wave of the future: make it really easy to share and use libraries, and get a crap-ton of low-quality libraries.
  - tomatotomato37 8y ago
    I think with C in particular it's counterbalanced by it's extreme compile times forcing devs to be picky with their libraries. Another factor that probably helps is that most operating systems are built off C derivatives and thus usually are already carrying common libararies as dlls
    - blattimwind 8y ago
      > C in particular [...] it's extreme compile times How so?
      - tomatotomato37 8y ago
        Simple really, the more static libraries you add, the longer it takes. This is most pronounced in the libraries themselves that usually don't risk the performance hit of using dlls, so you get less dependency-of-dependency.
    - pjc50 8y ago
      Compile times are not extreme for C, only for C++.
    - AboutTheWhisles 8y ago
      SQLite, which is a single 6MB .c file compiles in under a second in any compiler and under 0.01 seconds with tcc.
    - jungler 8y ago
      It's not the compile times that keep C codebases slim, it's the primitive build system. There's no notion of modules, so every project has to cobble together an equivalent thing, generally relying on make, autotools and its successors, and the result is both incompatible across projects and incompatible across slightly different build environments. It's a failing of the Unix model with a small silver lining in that C projects tend to go out of their way not to have to make the build any harder than it already is.
  - nordsieck 8y ago
    >>> As a C++ developer I can’t imagine 1700 dependencies in a relatively new code base. >> I think the answer is everything in your standard library plus some of the stuff you would use 3rd party dependencies for in your own project. > Even in C with its tiny standard library you don’t see this kind of explosion. I think one of the largest factors is that client-side JS doesn't have a good solution to the problem of dead code elimination. There are solutions like Google Closure (the JS-to-JS complier), but it's difficult to set up and not many people use it. Instead, it seems like people have moved to lots of small dependencies so they can essentially do dead code elimination by hand.
    - rozenmd 8y ago
      Webpack?
Waterluvian 8y ago
A discordant attempt at a standard library.
blattimwind 8y ago
Standard library by linear combination of independent components.
detaro 8y ago
It's a lot easier to add a dependency or a sub-dependency, so people split libraries further than they'd do in C++ and are less likely to write code/include it in the library directly instead of just importing another dependency for it. A "dependency" can be just one or two functions, something you'd never make a library for in C++. And of course if it is common for dependencies to have dependencies, than those dependencies also are gonna have dependencies and ... E.g. imagine C++'s Boost libraries: That's I think ~150 libraries with dependencies among each other? Now imagine there was no need to publish them together, so each of them is an individual library that could pick any random dependency, not just primarily from inside the set of 150. And split some of them up in 4-10 pieces doing parts of their task. And external projects of course also then depend only one some subset of those parts. A relative lack of a standard library maybe seeded the principle that people go looking for libraries for small things.
GrumpyNl 8y ago
This is why, this package is downloaded over a million times a week, all it does is trim a string. https://www.npmjs.com/package/trim https://www.npmjs.com/package/trim Can you imagine you need to depend on a external library for a trim function. Dont get me starting about the security issues.
- blattimwind 8y ago
  What feels bizarre to me is that you can't get the package from npmjs.com, there are no links to code etc. on there. Instead, use the API as a human (https://registry.npmjs.org/trim https://registry.npmjs.org/trim) to find the download link (https://registry.npmjs.org/trim/-/trim-0.0.1.tgz https://registry.npmjs.org/trim/-/trim-0.0.1.tgz), extract it locally, then find the source.
- slededit 8y ago
  It would make sense if it handled gnarly Unicode issues. It appears this does not however.
  - maxxxxx 8y ago
    I just started doing some JavaScript work and from what I have seen so far, a lot of packages are just amateur work thrown together quickly. In the C/C++ world I couldn't see stuff like this getting any adoption but in JavaScript this seems to happen.
- krapp 8y ago
  The problem here isn't depending on an external library for a trim function -- all it does is wrap string.replace, so minimal if any non-native functionality is actually being provided. The problem is that something this basic is considered a complete library. If you know regex you could write this on your coffee break. If you don't, you could look it up and still write in on your coffee break. If you want trim and do anything else with a string, you need another dependency tree. This is not even really a problem with javascript lacking a standard library, so much as a problem with the accepted practices of the javascript community.
maxxxxx 8y ago
I couldn't imagine this either but now that I am working with JavaScript a little I can see how lacking the JavaScript standard libraries are. It's very easy to find something on npm quickly and even trivial packages often have other dependencies on other trivial packages so the whole thing just cascades. I think I will slowly build up my own library for things like string and array manipulation. But in the end JavaScript needs a much better standard library.
- danShumway 8y ago
  People always say this, but I would actually prefer to see an even smaller standard library in Javascript. That's one of the reasons why I'm interested in WASM. There are obviously upsides to having a large standard library, but there are also significant downsides: because of Javascript's position on the web, it is very hard to remove things from the language when we get them wrong, and because Javascript is so widely used, it is very difficult to know in advance which implementations and styles of coding should be preferred. So for example, I'm happy that we have native Promises now, but I'm also happy that we waited until basically the entirety of external library authors had settled on an interface, and I'm even happier that we decided that deferreds were unnecessary since they could be easily recreated with normal promises. Of course there are downsides to preferring external libraries instead of native ones, but in Javascript this is a calculated choice. And I agree that JS culture encourages developers to take this too far. Even with a tiny standard library you probably don't need a leftpad package. It's just that those downsides look a lot worse, because in the JS community we lack good security practices about freezing packages. We also assume that NPM is secure by default, instead of a fancy wrapper around Github. We don't have a way to make sites immutable. And we have really stinking awful sandboxing in NodeJS, and (arguably) insufficient sandboxing in the browser. I typically get a little bit of pushback on this, but I advise people who are building end-user applications or a website as opposed to a library to commit their dependencies to Git. I also advise enterprise developers to avoid using packages that haven't been audited unless they're willing to read through the source code themselves -- and if you find an audited package, download that version and check the hash. That one in particular is a hard sell, I've heard developers tell me that it's literally impossible for companies to audit their Javascript libraries. I find that mindset really, well, disappointing -- especially since those same companies seem to have no problem blaming NPM for not auditing everyone's libraries. On the browser side, I advise people to self-host packages instead of using commercial CDNs, or at least to use subresource integrity policies[0]. That doesn't protect you from a compromised server, but it does protect you from a good number of XSS attacks. This is annoying of course. In an environment like Linux, on the server you'd use CentOS and you'd have strong guarantees about stability because you wouldn't be installing random packages from Github. On Linux you also get better checksums around your packages. Linux sandboxing isn't very good, but it's not like Node's is any better. I think on the web, we're really behind on this stuff. But I don't think expanding the standard library helps with any of that. I think it's a band-aid fix. [0]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
marcosdumay 8y ago
Some months ago I spent a week doing some proof of concept in Javascript and then mostly abandoned it. At it latest stage it depends directly on 8 libraries for running, and 16 libraries for building (because JS development is fucked-up). It also has 1103 total dependencies.
pornel 8y ago
Each does one thing well. Imagine Boost, but with every feature available for cherry-picking separately. Because there's almost no friction to add a new dependency, it's often easier to add one simple thing than roll your own. Note that nobody uses 1700 dependencies directly. Your project might use 5 deps for the 5 things it needs, but each of these libraries will use a few libraries for smaller pieces of functionality their library is composed of, and so on.

I think the answer is everything in your standard library plus some of the stuff you would use 3rd party dependencies for in your own project. Maybe standard libraries need to get bigger for JS.

slededit 8y ago
Even in C with its tiny standard library you don’t see this kind of explosion. There has to be more to it than that.
- bfred_it 8y ago
  npm handles sub-dependencies with ease so people aren't afraid to pull in smaller dependencies. Also since the code has to be downloaded (or uploaded to small lambdas, for example), JS developers are generally weary of depending on large libraries.
  - krapp 8y ago
    >Also since the code has to be downloaded (or uploaded to small lambdas, for example), JS developers are generally weary of depending on large libraries. Isn't a dependency tree of thousands of small, interdependent libraries essentially just a large, distributed library? Is there really a benefit to that versus compiling those dependencies to a single file, especially given how many NPM packages are just a single small function? The end result in many cases would probably be smaller than a lot of old JQuery plugins.
    - eeZah7Ux 8y ago
      > Isn't a dependency tree of thousands of small, interdependent libraries essentially just a large, distributed library? Yes and it's a thousand times worse than a single, large library.
- cyphar 8y ago
  The problem is that it is far too easy to publish a new library that contains a single function. Due to history and other considerations, this is much harder in C (you have to wrangle CMake or autotools, and then package your library for distributions). While I do think making things easier for developers is a good goal, making the wrong things too easy (that is, making libraries that become very heavily depended on) results in the problems of Node.js (and I would argue the same is true for Rust). Maybe I'm just a curmudgeon, but I really have a sour taste in my mouth when I look at how Node.js and Rust do library management -- it's making it too easy to make a small library which doesn't really work and then people depend on it. I once tried to write a simple IMAP cloning utility in Rust and found 4 IMAP crates -- none of them worked properly and all of them incorrectly handled several core parts of the IMAPv4 spec. I figured out later that they all appeared to be forks of one another (or they copied each others' APIs), but that doesn't help matters -- why are forked crates taking up more space in the global crate namespace? The "imap" crate doesn't implement IMAP properly! Python had this problem (in a lesser degree) too with pip, but I think having a larger standard library and lots of time to mature allowed them to overcome it.
  - mcguire 8y ago
    "I once tried to write a simple IMAP cloning utility in Rust and found 4 IMAP crates -- none of them worked properly and all of them incorrectly handled several core parts of the IMAPv4 spec. I figured out later that they all appeared to be forks of one another (or they copied each others' APIs), but that doesn't help matters -- why are forked crates taking up more space in the global crate namespace? The "imap" crate doesn't implement IMAP properly!" I noticed this problem with Perl and CPAN: there were dozens (well, several) email-sending packages, none of which were near complete. (Speaking SMTP is easy, an actual MTA is hard.) I suspect there are similar issues in Java-land, although I haven't used Maven-derivatives enough to find out. It's the wave of the future: make it really easy to share and use libraries, and get a crap-ton of low-quality libraries.
- tomatotomato37 8y ago
  I think with C in particular it's counterbalanced by it's extreme compile times forcing devs to be picky with their libraries. Another factor that probably helps is that most operating systems are built off C derivatives and thus usually are already carrying common libararies as dlls
  - blattimwind 8y ago
    > C in particular [...] it's extreme compile times How so?
    - tomatotomato37 8y ago
      Simple really, the more static libraries you add, the longer it takes. This is most pronounced in the libraries themselves that usually don't risk the performance hit of using dlls, so you get less dependency-of-dependency.
  - pjc50 8y ago
    Compile times are not extreme for C, only for C++.
  - AboutTheWhisles 8y ago
    SQLite, which is a single 6MB .c file compiles in under a second in any compiler and under 0.01 seconds with tcc.
  - jungler 8y ago
    It's not the compile times that keep C codebases slim, it's the primitive build system. There's no notion of modules, so every project has to cobble together an equivalent thing, generally relying on make, autotools and its successors, and the result is both incompatible across projects and incompatible across slightly different build environments. It's a failing of the Unix model with a small silver lining in that C projects tend to go out of their way not to have to make the build any harder than it already is.
- nordsieck 8y ago
  >>> As a C++ developer I can’t imagine 1700 dependencies in a relatively new code base. >> I think the answer is everything in your standard library plus some of the stuff you would use 3rd party dependencies for in your own project. > Even in C with its tiny standard library you don’t see this kind of explosion. I think one of the largest factors is that client-side JS doesn't have a good solution to the problem of dead code elimination. There are solutions like Google Closure (the JS-to-JS complier), but it's difficult to set up and not many people use it. Instead, it seems like people have moved to lots of small dependencies so they can essentially do dead code elimination by hand.
  - rozenmd 8y ago
    Webpack?

Waterluvian 8y ago

A discordant attempt at a standard library.

blattimwind 8y ago

Standard library by linear combination of independent components.

detaro 8y ago

It's a lot easier to add a dependency or a sub-dependency, so people split libraries further than they'd do in C++ and are less likely to write code/include it in the library directly instead of just importing another dependency for it. A "dependency" can be just one or two functions, something you'd never make a library for in C++. And of course if it is common for dependencies to have dependencies, than those dependencies also are gonna have dependencies and ... E.g. imagine C++'s Boost libraries: That's I think ~150 libraries with dependencies among each other? Now imagine there was no need to publish them together, so each of them is an individual library that could pick any random dependency, not just primarily from inside the set of 150. And split some of them up in 4-10 pieces doing parts of their task. And external projects of course also then depend only one some subset of those parts. A relative lack of a standard library maybe seeded the principle that people go looking for libraries for small things.

GrumpyNl 8y ago

This is why, this package is downloaded over a million times a week, all it does is trim a string. https://www.npmjs.com/package/trim https://www.npmjs.com/package/trim Can you imagine you need to depend on a external library for a trim function. Dont get me starting about the security issues.

blattimwind 8y ago
What feels bizarre to me is that you can't get the package from npmjs.com, there are no links to code etc. on there. Instead, use the API as a human (https://registry.npmjs.org/trim https://registry.npmjs.org/trim) to find the download link (https://registry.npmjs.org/trim/-/trim-0.0.1.tgz https://registry.npmjs.org/trim/-/trim-0.0.1.tgz), extract it locally, then find the source.
slededit 8y ago
It would make sense if it handled gnarly Unicode issues. It appears this does not however.
- maxxxxx 8y ago
  I just started doing some JavaScript work and from what I have seen so far, a lot of packages are just amateur work thrown together quickly. In the C/C++ world I couldn't see stuff like this getting any adoption but in JavaScript this seems to happen.
krapp 8y ago
The problem here isn't depending on an external library for a trim function -- all it does is wrap string.replace, so minimal if any non-native functionality is actually being provided. The problem is that something this basic is considered a complete library. If you know regex you could write this on your coffee break. If you don't, you could look it up and still write in on your coffee break. If you want trim and do anything else with a string, you need another dependency tree. This is not even really a problem with javascript lacking a standard library, so much as a problem with the accepted practices of the javascript community.

maxxxxx 8y ago

I couldn't imagine this either but now that I am working with JavaScript a little I can see how lacking the JavaScript standard libraries are. It's very easy to find something on npm quickly and even trivial packages often have other dependencies on other trivial packages so the whole thing just cascades. I think I will slowly build up my own library for things like string and array manipulation. But in the end JavaScript needs a much better standard library.

danShumway 8y ago
People always say this, but I would actually prefer to see an even smaller standard library in Javascript. That's one of the reasons why I'm interested in WASM. There are obviously upsides to having a large standard library, but there are also significant downsides: because of Javascript's position on the web, it is very hard to remove things from the language when we get them wrong, and because Javascript is so widely used, it is very difficult to know in advance which implementations and styles of coding should be preferred. So for example, I'm happy that we have native Promises now, but I'm also happy that we waited until basically the entirety of external library authors had settled on an interface, and I'm even happier that we decided that deferreds were unnecessary since they could be easily recreated with normal promises. Of course there are downsides to preferring external libraries instead of native ones, but in Javascript this is a calculated choice. And I agree that JS culture encourages developers to take this too far. Even with a tiny standard library you probably don't need a leftpad package. It's just that those downsides look a lot worse, because in the JS community we lack good security practices about freezing packages. We also assume that NPM is secure by default, instead of a fancy wrapper around Github. We don't have a way to make sites immutable. And we have really stinking awful sandboxing in NodeJS, and (arguably) insufficient sandboxing in the browser. I typically get a little bit of pushback on this, but I advise people who are building end-user applications or a website as opposed to a library to commit their dependencies to Git. I also advise enterprise developers to avoid using packages that haven't been audited unless they're willing to read through the source code themselves -- and if you find an audited package, download that version and check the hash. That one in particular is a hard sell, I've heard developers tell me that it's literally impossible for companies to audit their Javascript libraries. I find that mindset really, well, disappointing -- especially since those same companies seem to have no problem blaming NPM for not auditing everyone's libraries. On the browser side, I advise people to self-host packages instead of using commercial CDNs, or at least to use subresource integrity policies[0]. That doesn't protect you from a compromised server, but it does protect you from a good number of XSS attacks. This is annoying of course. In an environment like Linux, on the server you'd use CentOS and you'd have strong guarantees about stability because you wouldn't be installing random packages from Github. On Linux you also get better checksums around your packages. Linux sandboxing isn't very good, but it's not like Node's is any better. I think on the web, we're really behind on this stuff. But I don't think expanding the standard library helps with any of that. I think it's a band-aid fix. [0]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

marcosdumay 8y ago

Some months ago I spent a week doing some proof of concept in Javascript and then mostly abandoned it. At it latest stage it depends directly on 8 libraries for running, and 16 libraries for building (because JS development is fucked-up). It also has 1103 total dependencies.

pornel 8y ago

Each does one thing well. Imagine Boost, but with every feature available for cherry-picking separately. Because there's almost no friction to add a new dependency, it's often easier to add one simple thing than roll your own. Note that nobody uses 1700 dependencies directly. Your project might use 5 deps for the 5 things it needs, but each of these libraries will use a few libraries for smaller pieces of functionality their library is composed of, and so on.