5 ms·

Depends on Every Package in NPM

87 points by fctorial 5y ago

deleted 5y ago
[deleted]
vladharbuz 5y ago
To be clear, this only depends on 1000 packages, seemingly because of a limitation of npm. What's actually pretty hilarious is that this probably isn't too far off the number of dependencies an average Javascript project has. `npm install vue-cli` installs 306 packages weighing in at 245MB. :)
- madjam002 5y ago
  The project I’m working on now has 3,879 direct/indirect dependencies, beat that
  - WanderPanda 5y ago
    Is that still considered "working", or already "battling"
  - marcosdumay 5y ago
    Hum? I've once reached 10k dependencies by just installing vue and a graphics plugin in an experiment.
- patates 5y ago
  In my previous job, some PHP developers loved to pick on JS with that same accusation, then we checked the composer packages folder... Also, working with gradle now... I'd take the npm + webpack any day.
  - jeltz 5y ago
    Maybe php has changed the last couple of years but the last time I worked with it the numbers of dependencies where nowhere near a typical node project.
    - egeozcan 5y ago
      PHP has a standard library that's just incomparably larger than node's. That explains the number of packages. The size, OTOH, is usually in the same ballpark of a node project. If I import chrome from npm (for testing) in some way (directly or transitional), then node leaps ahead, when not, usually composer takes the lead.
- jamesrom 5y ago
  It appears to be the 1000 packages with the highest PageRank, as pulled from https://anvaka.github.io/npmrank/ https://anvaka.github.io/npmrank/ So it's safe to say that this package has the most transitive dependencies in npm.
  - OskarS 5y ago
    If you wanna do this bad boy for real, you depend on this one and the next 999 packages. Actually, this can become a fun optimization problem. Which 1000 packages do you depend on to cover as much of NPM as possible?
    - madeofpalk 5y ago
      I thought that's what this was.
      - koolba 5y ago
        Many of these packages have no dependencies. A new package that depends on this and anything not already downstream would have more dependencies.
    - travisjungroth 5y ago
      Might get you close: Until you have 1000 Find the package not included that adds the most deps If it adds less than everything you have Add it Else Kick out anything that adds less than that And then make that a package and repeat as desired.
      - OskarS 5y ago
        Yeah, that's the greedy algorithm and it will probably take you pretty far, but there's no guarantee that it's optimal. What if the package has lots of dependencies, but they are already covered by other packages. Do you kick other packages out? Which ones? Or is it better to keep them in and add some other packages? This problem is almost certainly NP-complete (it's basically the the "set cover" problem from Karp's 21 NP-complete problems), so for something the size of NPM it's almost certainly infeasible. Your greedy version would probably get you a pretty nice result, though.
Decabytes 5y ago
This is why I don’t get the whole “solving X problem in Y lines of code” And then importing some graphics/math library and using that all throughout the code. It misrepresents all the code that is actually used to solve the problem. If I wanted to solve a problem in 1 like of code I would just package it up into a library and call a single function. I’m not saying using other libraries is bad, but even when I write a script that uses over a thousand lines of hand written python and calls a dozen pandas functions I know that the actual loc to solve the problem is much larger, and wouldn’t even be possible without certain libraries.
- thitcanh 5y ago
  To be fair, that always refers to the LOC I wrote. If I can write monaLisa() and get a nice painting on canvas then that really was 1 LOC. What line count doesn’t include is the weight of the dependency, which may or may not matter.
  - brixon 5y ago
    Dependency weight is more a factor for maintenance than initial development. The apps I write I have to support for many years, so this dependency weight does concern me. By dependency weight I am more referring to the breadth of authors/maintainers and not the actual amount of code. When working in the Java/.NET worlds you still deal with a ginormous amount of dependency code, but they are one maintainer that is funded. The downside of monolithic dependencies is lack of flexibility and vendor lock in.
brian_herman 5y ago
Here is the npm log when you install it. https://gist.github.com/GalaxyBrainHuman/9d272f7f6fbbe5144302d80ab013976e https://gist.github.com/GalaxyBrainHuman/9d272f7f6fbbe514430...
- dncornholio 5y ago
  Pretty boring.. I was expecting developers asking for coffee, jobs and tons of depricated warnings like every decent node project
menotyou 5y ago
What is actually the licence of a npm development depending on so many packages? Is the most restrictive licence in the dependecy tree the one which determines the license requirement of my program? Do I have to publish my source code if any of the dependencies is GPL?
- jerf 5y ago
  There's no guarantee that you can release such a program at all. Things can have conflicting licenses in which case it is simply not valid to have both of them at the same time. For example, a strong copyleft requiring all source code to be released, and a library with a binary blob for which source release is impossible or prohibited.
- cddotdotslash 5y ago
  IANAL, but I've been involved in a few acquisitions and when you get to the part about code ownership, the acquiring company will often do a license scan of the entire dependencies folder (which includes sub dependencies). So if any of those files show up as GPL, you technically could be in violation. Now, if a package licensed as MIT includes a dependency on a package licensed as GPL, I don't know if that's a violation by the parent project owner because the parent package doesn't actually distribute the GPL code, but rather includes a reference to it, so that the installer running "npm install" fetches it. But I would imagine that the end user who is pulling those packages and actually installing all the dependencies would be in violation because the final packaged code used to deliver the service actually contains the GPL code.
joshmanders 5y ago
Hoarders did it first by almost a decade. https://github.com/jfhbrook/hoarders/issues/2 https://github.com/jfhbrook/hoarders/issues/2
cwmma 5y ago
reminds me of hoarders
pornel 5y ago
npm has only 1.5 million packages. With max 1000 deps per package limit you only need barely over 2 levels deep to really depend on them all.
- floydnoel 5y ago
  Perhaps it is only 1,000 top level dependencies? Because it seems like larger projects would hit over 1,000 total
pjfin123 5y ago
Left pad your strings, no import necessary!
yoz-y 5y ago
NPM often gets flak, but just right now I installed youtube-dl on a raspbian install and it has brought over 100 dependencies including adwaita-icon-theme, libqt5printsupport5, libgtk-3-0, x11-xserver-utils, libwayland-server0. However installing it from pip demands nothing and the program works. (yes, I know that it can use ffmpeg to remux videos, but from pip it also works without that).
- wildrhythms 5y ago
  Is this a problem with npm, or a problem with the youtube-dl project's dependencies?

87 points by fctorial 5y ago

deleted 5y ago

[deleted]

To be clear, this only depends on 1000 packages, seemingly because of a limitation of npm. What's actually pretty hilarious is that this probably isn't too far off the number of dependencies an average Javascript project has. `npm install vue-cli` installs 306 packages weighing in at 245MB. :)

madjam002 5y ago
The project I’m working on now has 3,879 direct/indirect dependencies, beat that
- WanderPanda 5y ago
  Is that still considered "working", or already "battling"
- marcosdumay 5y ago
  Hum? I've once reached 10k dependencies by just installing vue and a graphics plugin in an experiment.
patates 5y ago
In my previous job, some PHP developers loved to pick on JS with that same accusation, then we checked the composer packages folder... Also, working with gradle now... I'd take the npm + webpack any day.
- jeltz 5y ago
  Maybe php has changed the last couple of years but the last time I worked with it the numbers of dependencies where nowhere near a typical node project.
  - egeozcan 5y ago
    PHP has a standard library that's just incomparably larger than node's. That explains the number of packages. The size, OTOH, is usually in the same ballpark of a node project. If I import chrome from npm (for testing) in some way (directly or transitional), then node leaps ahead, when not, usually composer takes the lead.
jamesrom 5y ago
It appears to be the 1000 packages with the highest PageRank, as pulled from https://anvaka.github.io/npmrank/ https://anvaka.github.io/npmrank/ So it's safe to say that this package has the most transitive dependencies in npm.
- OskarS 5y ago
  If you wanna do this bad boy for real, you depend on this one and the next 999 packages. Actually, this can become a fun optimization problem. Which 1000 packages do you depend on to cover as much of NPM as possible?
  - madeofpalk 5y ago
    I thought that's what this was.
    - koolba 5y ago
      Many of these packages have no dependencies. A new package that depends on this and anything not already downstream would have more dependencies.
  - travisjungroth 5y ago
    Might get you close: Until you have 1000 Find the package not included that adds the most deps If it adds less than everything you have Add it Else Kick out anything that adds less than that And then make that a package and repeat as desired.
    - OskarS 5y ago
      Yeah, that's the greedy algorithm and it will probably take you pretty far, but there's no guarantee that it's optimal. What if the package has lots of dependencies, but they are already covered by other packages. Do you kick other packages out? Which ones? Or is it better to keep them in and add some other packages? This problem is almost certainly NP-complete (it's basically the the "set cover" problem from Karp's 21 NP-complete problems), so for something the size of NPM it's almost certainly infeasible. Your greedy version would probably get you a pretty nice result, though.

Decabytes 5y ago

This is why I don’t get the whole “solving X problem in Y lines of code” And then importing some graphics/math library and using that all throughout the code. It misrepresents all the code that is actually used to solve the problem. If I wanted to solve a problem in 1 like of code I would just package it up into a library and call a single function. I’m not saying using other libraries is bad, but even when I write a script that uses over a thousand lines of hand written python and calls a dozen pandas functions I know that the actual loc to solve the problem is much larger, and wouldn’t even be possible without certain libraries.

thitcanh 5y ago
To be fair, that always refers to the LOC I wrote. If I can write monaLisa() and get a nice painting on canvas then that really was 1 LOC. What line count doesn’t include is the weight of the dependency, which may or may not matter.
- brixon 5y ago
  Dependency weight is more a factor for maintenance than initial development. The apps I write I have to support for many years, so this dependency weight does concern me. By dependency weight I am more referring to the breadth of authors/maintainers and not the actual amount of code. When working in the Java/.NET worlds you still deal with a ginormous amount of dependency code, but they are one maintainer that is funded. The downside of monolithic dependencies is lack of flexibility and vendor lock in.

brian_herman 5y ago

Here is the npm log when you install it. https://gist.github.com/GalaxyBrainHuman/9d272f7f6fbbe5144302d80ab013976e https://gist.github.com/GalaxyBrainHuman/9d272f7f6fbbe514430...

dncornholio 5y ago
Pretty boring.. I was expecting developers asking for coffee, jobs and tons of depricated warnings like every decent node project

menotyou 5y ago

What is actually the licence of a npm development depending on so many packages? Is the most restrictive licence in the dependecy tree the one which determines the license requirement of my program? Do I have to publish my source code if any of the dependencies is GPL?

jerf 5y ago
There's no guarantee that you can release such a program at all. Things can have conflicting licenses in which case it is simply not valid to have both of them at the same time. For example, a strong copyleft requiring all source code to be released, and a library with a binary blob for which source release is impossible or prohibited.
cddotdotslash 5y ago
IANAL, but I've been involved in a few acquisitions and when you get to the part about code ownership, the acquiring company will often do a license scan of the entire dependencies folder (which includes sub dependencies). So if any of those files show up as GPL, you technically could be in violation. Now, if a package licensed as MIT includes a dependency on a package licensed as GPL, I don't know if that's a violation by the parent project owner because the parent package doesn't actually distribute the GPL code, but rather includes a reference to it, so that the installer running "npm install" fetches it. But I would imagine that the end user who is pulling those packages and actually installing all the dependencies would be in violation because the final packaged code used to deliver the service actually contains the GPL code.

joshmanders 5y ago

Hoarders did it first by almost a decade. https://github.com/jfhbrook/hoarders/issues/2 https://github.com/jfhbrook/hoarders/issues/2

cwmma 5y ago

reminds me of hoarders

pornel 5y ago

npm has only 1.5 million packages. With max 1000 deps per package limit you only need barely over 2 levels deep to really depend on them all.

floydnoel 5y ago
Perhaps it is only 1,000 top level dependencies? Because it seems like larger projects would hit over 1,000 total

pjfin123 5y ago

Left pad your strings, no import necessary!

yoz-y 5y ago

NPM often gets flak, but just right now I installed youtube-dl on a raspbian install and it has brought over 100 dependencies including adwaita-icon-theme, libqt5printsupport5, libgtk-3-0, x11-xserver-utils, libwayland-server0. However installing it from pip demands nothing and the program works. (yes, I know that it can use ffmpeg to remux videos, but from pip it also works without that).

wildrhythms 5y ago
Is this a problem with npm, or a problem with the youtube-dl project's dependencies?