25 ms·

Ride down into JavaScript dependency hell

133 points by nikolalsvk 7y ago

dmitriid 7y ago
I'm not buying the narrative of "js dependency hell what are they thinking" anymore. Try installing a package for Rust. Or Go. Or ... any language, really.
- bsagdiyev 7y ago
  I’m not sure whataboutism is the way to fix the well-known issues the JS ecosystem has in package management. When I’ve installed a Go package or Python package I haven’t ended up with sometimes hundreds or thousands of sub-dependencies. Some packages can be ridiculous, but nothing like I have seen running “npm i” for something that seems like it should be simple. I apologize for not having an example off-hand but this keeps coming up precisely because it is an issue unique to JS in this case, even if other languages have it to a degree.
  - sippeangelo 7y ago
    Try Feathers, a "lightweight web-framework for creating real-time applications and REST APIs" clocking in at about 600 transitive dependencies. How do you audit this?
    - bsagdiyev 7y ago
      Link for the curious: https://github.com/feathersjs/feathers https://github.com/feathersjs/feathers
    - jfkebwjsbx 7y ago
      I think they just don't audit nor review the code they depend on.
  - dmitriid 7y ago
    It's now whataboutism. It's finger pointing from a lot of people who develop in languages whose story is only marginally better. As I wrote in a sibling comment, I have a project I haven't even started yet in Rust [1]. And it's measly 6 dependencies pull a total of 197. [1] https://news.ycombinator.com/item?id=22841742 https://news.ycombinator.com/item?id=22841742
    - bsagdiyev 7y ago
      I took a look, this sounds more unique to Rust than Go or Python, but they all can do it to some degree. I have no hands on experience with Rust to really have any reasonable input on this so I don’t want to make an uninformed statement.
    - jfkebwjsbx 7y ago
      If what you say is true, then Rust is suffering the same fate. 197 dependencies is too many to trust.
- shakna 7y ago
  Gatsby has 19k dependencies. I don't think I can find something like that for Rust. The largest, and not exactly admirable, I can find is reqwest that drags in 97. Whilst 100 is a huge number... It's an enormous gap from the many thousands.
  - dmitriid 7y ago
    Javascript doesn't have a standard library (something a lot of languages provide), and nodejs's built-in functions library are... lacking. Also, the culture of providing small packages, being around for longer, and so on and so forth... I have a project I haven't even started yet in Rust. Three dependencies in Cargo.toml. They download and install 34 dependencies. I've now added three more (from kube-rs readme). It's now 197 dependencies. And so on.
  - mrighele 7y ago
    And that it is not even the worst I have seen. I think the worst offenders are development tools like webpack. I have a small project that I work on from time to time which uses 5 libraries (react, a map library and a chart library), typescript, and react-scripts (which I guess pulls in webpack and all the rest). This is what happens when I run a npm audit (I havent' touched it for a couple of months) > found 38934 vulnerabilities (38916 low, 18 moderate) in 906346 scanned packages > run `npm audit fix` to fix 38620 of them. > 314 vulnerabilities require manual review. See the full report for details. While the real numbers are probably lower because there is a lot of duplication inside the node_modules folder, I find this ... astounding.
- swiley 7y ago
  There are bad packages in rust, go, python (1), even C. The worst of these are closer to the norm for JavaScript though. Personally I don’t feel comfortable running JavaScript packages outside of some kind of sandbox (usually a VM) on my personal computers. (1) in fact, seeing some of the worst kept me away from python for far too long until I realized plenty of python software doesn’t even need pypi and most only need it for a couple things with no transitive dependencies
- jt2190 7y ago
  Yes. Using third-party code is not a binary decision between “write everything yourself” and “use all the things”, it is part of the ongoing conversation about optimization: Development time, deployable size, build time, tool support, community support, training, documentation, etc.
- CyberDildonics 7y ago
  If you mean that javascript gets a disproportional amount of hate, that might be because of how much more it is being used. It is true that rust seems to be ignoring the sins of the past and diving right in to having lots of dependencies, with the solution being to manage them instead of weeding them out in some way.
- tristanstcyr 7y ago
  I agree that this isn't totally specific to JavaScript. It's just that the ecosystem is very extensive and so there are a lot of packages, because you don't need to write your own code for a lot of existing problems. There is perhaps a something about the JavaScript audience tending to attract less advanced implementers. So you end up with a lot of otherwise considered trivial and low-quality utilities being published and pulled down from the registry. Rust and Go tend to be for more advanced scenarios where things like performance, security and support matters tremendously.
awinter-py 7y ago
express (common web framework) seems to have only 51 deps in lockfile jekyll has 13 I don't disagree that lots of deps = supply chain risk But there have always been variations between projects & kinds of projects re how many deps they pull in. Try following someone's ipython data science tutorial, it's requirements.txt for days I think lack of a standard lib early on plus hipster functional culture made the '10s JS leaders like small monofunctional libs. Focus on transpilation plus lack of tree-shaking in early versions of webpack may have also created an incentive for small things.
- dgellow 7y ago
  A python equivalent to "express" would be Flask https://palletsprojects.com/p/flask/ https://palletsprojects.com/p/flask/ I see 4 dependencies in the setup.py, and they are all from the same team of maintainers (i.e the Pallet team): - Werkzeug - Jinja2 - itsdangerous - click https://github.com/pallets/flask/blob/master/setup.py https://github.com/pallets/flask/blob/master/setup.py https://github.com/pallets/flask/network/dependencies https://github.com/pallets/flask/network/dependencies
  - awinter-py 7y ago
    'pip freeze' results from a few other py web frameworks: sanic = 18, fastapi = 3 but one of them is a 7MB download (pydantic) yeah it's less, but also python has an amazing stdlib
  - oefrha 7y ago
    To be fair, express is a production server itself whereas werkzeug is not, so you need to add something like gunicorn or uwsgi (yeah I know nginx supports uwsgi_pass natively). click is not necessary in production though (it's only used in the dev CLI). That said, 4 dependencies is the norm in Python land, whereas 13 dependencies is an outlier in JS land. Edit: oops, misread, gp said express has 51 deps, not 13.
  - wffurr 7y ago
    The Python standard library is a lot more capable than the JS equivalents (Node or browser API). That eliminates the need for a lot of dependencies. There's been a TC39 proposal for a JS stdlib for about 2 years now: https://github.com/tc39/proposal-javascript-standard-library https://github.com/tc39/proposal-javascript-standard-library
pizzapim 7y ago
So what does NPM do differently than plugin managers from other languages? I can't name one where the dependency hell is this big.
- leetrout 7y ago
  My personal take having “grown up” with all of this - JS is one of the worst ecosystems for relying on external packages for everything. It’s more the culture than the tooling. One of the things I love about Go is generally people are a little more forgiving to copying something around a couple times instead of making a lib for everything.
  - smitty1e 7y ago
    Inviting the question of why the culture hasn't sprouted a "package rationalizer" to produce a curated, tested, noise-filtered distribution of useful functionality. Come to think of it, something that does for packages what jQuery did to bring (some) sanity to browser incompatibilities.
    - ng12 7y ago
      Because we can't even agree on whether React or Vue or Angular is best. In fact this is the root of the issue. It's important to realize that `node_modules` for any project in Popular Framework X will be almost identical. Most of the packages are related to compiling, bundling, and the development environment and a small minority are actual client-side dependencies. Even the frameworks themselves are pretty light -- React has a grand total of 2 dependencies. It's packaging the entire toolchain that causes most of the bloat.
  - pixelrevision 7y ago
    Go in particular has an amazing standard library and very clear guidelines on how to write code. JavaScript does not have either of those things so it promotes people releasing competing libraries with similar functionality written in various different styles.
  - 7y ago
deleted 7y ago
[deleted]
sproketboy 7y ago
JavaScript is what happens when you lock toddlers in a room on their own for a month.
dwheeler 7y ago
Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems. "small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code." Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open Source Software", The Linux Foundation & The Laboratory for Innovation Science at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L. Hoffman. https://www.coreinfrastructure.org/programs/census-program-ii/ https://www.coreinfrastructure.org/programs/census-program-i... (This is a contrast of citations from other works; you can see the specific citations in the report, but I'm trying to do this from a phone. In any case, the issue is the massive difference between JavaScript and everything else, and I think this quotation shows it nicely.)
- dep_b 7y ago
  iOS dependencies in Cocoapods are including less than 1 other packages on average (1). The most I see are Alamofire (a networking package people keep using for reasons that I do not fully understand) and Starscream (a Websocket implementation, that's pretty hard to write from scratch to be honest). Having unnecessary dependencies is pretty much frowned upon. One of the reasons is that the iOS SDK is really complete by itself while Javascript doesn't even ship with a usable Date implementation. If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically. (1) totally based on gut feeling, not based on hard data
  - hn_throwaway_99 7y ago
    Java didn't have a decent date implementation until Java 8. > If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically. As this article points out, I think that lodash is that for many people.
    - 7y ago
Waterluvian 7y ago
I have a project with a total of eight dependencies. I walked away for six months as it was stable. I come back to add a few features and npm tells me I have over 38 000 vulnerabilities of different severity levels. So many that 'npm audit' just freezes up. So there's that too.
- samsquire 7y ago
  My stuff breaks whenever I try upgrade it. People refactoring stuff.
  - brobdingnagians 7y ago
    These types of issues are the biggest reason why I avoid javascript/npm projects. I came back to a project after a few months and it was broken, had to rewrite some parts and upgrade other parts just to get it to run again.
    - williamdclt 7y ago
      I'm sorry but for all the flaws of the NPM ecosystem, version locking is not one (both `npm` and `yarn` do much much better than anything in Python). There's no reason a correctly set-up project would magically break with time. I don't love NPM anymore than the next guy, but my blame will go to the dev in this case.
      - Waterluvian 7y ago
        Python dependency management is a disaster. But I find myself using far fewer dependencies with Python projects so I suffer the pain less.
        jessaustin 7y ago
        This is more of that "wet streets cause rain" logic we hear about...
      - 7y ago
dgellow 7y ago
I think that we are in a situation where the number of dependency doesn't have any meaning anymore for people working on web frontend projects. Recently while revamping parts of the CI/CD at $WORK, I found out that running npm install for one of our frontend project downloads around 1 million dependencies. And npm audit reports a completely ridiculous number of security issues with them. It's just so absurd and nonsensical, nobody can understand what that even mean. Seems to me to be a crazy huge liability, but quite a lot of people seem to be fine with that. I don't know what to do from that information, but trying to create some awareness around the issue feels like talking about the number of operations per second a CPU does with someone who has zero idea what a realistic range is. It's seen as a high number, but doesn't have any meaning attached to it. Edit: I don't have access to the project anymore to check more details (such as installation time), but I checked some personal notes I took when I found this out, the redacted output from npm audit is the following $ npm audit <REDACTED> vulnerabilities found - Packages audited: 927016 Severity: <REDACTED> Low | <REDACTED> Moderate Done in 41.62s. So slightly below the 1 million mark. Edit 2: Now I'm wondering ... maybe the "packages audited" from npm audit doesn't mean "you have that number of packages, and I audited them". But if that's the case, that's a terrible UX, and I have zero idea what that number mean, which kind of support my point.
- Ayesh 7y ago
  A million dependencies sound horribly excessive, even by JS standards.
  - dgellow 7y ago
    That was my reaction too.
- awinter-py 7y ago
  how long does that take to install? is there a local npm mirror / cache?
- crazygringo 7y ago
  I mean npm only has slightly more than 1 million packages, last I checked. This doesn't make even the slightest bit of sense. Did somebody somehow just import... everything by accident? Did npm hit a bug? Is there some infinite recursion going on that eventually hits a wall? This obviously isn't representative.
Dunedan 7y ago
That page itself is an example what's wrong with using JavaScript on the web: It's used in places where it's not necessary. In this case the site doesn't apply a stylesheet with JavaScript disabled. JavaScript is great when it's used for enabling interactive websites, but nowadays more and more websites use it to break features, which don't need JavaScript at all.
parhamn 7y ago
What’s worse is that the whole installing multiple versions of the same package for different sub dependencies doesn’t make any sense when when the language supports singletons and module level variables with has no way of specifying which version to import. I was shocked this behavior exists when a site broke by upgrading a sub dependency. Turned out they both installed react so they used a different “creatContext” and now there were two context instances instead of one.
- tristanstcyr 7y ago
  React would normally be a peerDependency and so you shouldn't run into this problem if dependencies are specified properly. If not, NPM module flattening should take care of it. Then worst case, you can also tell webpack to bundle a specific installation. There are some ways to solve these problems...
  - parhamn 7y ago
    Sure (upvote) and we dealt with it accordingly after some confused digging. On a more general level I’ve never seen something like that in a package manager. It seems as though some of the language features (like the singleton I mentioned) would make it obvious that arbitrarily duplicating packages and scoping them to sub packages is a major no-no, as in most languages I’ve dealt with.
    - jessaustin 7y ago
      It seems like a weird thing about React, or maybe how you're using it, that this is a problem. Most packages, and the language itself, handle multiple simultaneous versions of dependencies without issue. If packages are doing the old-fashioned "stick a property on window" thing instead of just getting required or imported browserify-style, they can step on each others' toes (and it's totally possible that React is dumb enough to step on its own toes). It has been years since that was necessary. browserify or more recent tools like parcel or webpack should be used instead.
      - 7y ago
johannes1234321 7y ago
My concern with npm packages isn't the number and small size of the packages itself, but that most of them are maintained and owned by individuals who do work with little review by others and who could disappear any time. If an individual goes away or takes harmful action it takes time till this is noticed and till a surviving fork emerges. If more libraries were maintained by groups in a shared/collaborative way, more of those risks would go away.
tluyben2 7y ago
I cannot understand how people work with this. I work with many different technologies and try to avoid JS, but some times I have to. The past weeks I have worked on a React Native project that was written by someone else; what a horror show. I mean it wasn't the worst code by the previous guy but even in a few months a lot is simply broken and not 'best practice' anymore. Compared to most other environments I work with, iteration is somewhat faster when you finally have everything working, but the dependencies and ecosystem is horrible imho. A bigger issue with all of it is that in my line of work, for the backend of firmware I cannot just depend on 3rd party libs as they will get audited, so I need to audit them before. Ofcourse most people don't have that issue.
- williamdclt 7y ago
  I don't agree with you on this. The JS (or NPM) way makes it a pain to audit a project due to thousands of dependencies, but it's far easier to "have everything working". `npm i && npm start` is as good as it gets in terms of "getting started" friction (with version locking): python is worse, ruby is worse, PHP is worse, Java is worse. React-Native has more moving pieces indeed, but it's not really JS's (or NPM's) fault: you get all problems of iOS and Android at once. I will definitely agree with you that working on React-Native is a pain, to be able to work with modern tools need to know JS, optionally TS, Obj-C, Swift, Java, Kotlin, Ruby (for Fastlane) and Gradle's custom language. Insane.
  - dgellow 7y ago
    I've done 3 years of Golang as my main language, my experience is: - go build ./... builds everything from the current project - go test ./... tests everything from the current project - go install ./... installs everything from the current project That's one of the best thing ever. I can go to any project, I know how to build, test, install, and can directly be productive. On the other hand, I started learning C++ one year ago, and it's the exact opposite, each project has its own homemade build system that isn't supported by one platform or the other.
    - diggan 7y ago
appleflaxen 7y ago
If you want the 3D view of Gatsby dependencies without waiting for the initial 2D view to populate, you can go here: https://npm.anvaka.com/#/view/3d/gatsby https://npm.anvaka.com/#/view/3d/gatsby
overgard 7y ago
Sadly the alternative isn't much better: the old school model of not having packages often leads to libraries never getting updated at all. In C++ often adding a library is such a chore once you get through all the linking issues, as you battle the 4000 conflicting compiler switches, and compiler versions, that once it works you never want to change it again. I think maybe the best solution we have so far are robust "batteries included" standard libraries like python, where even if they're huge theyre at least curated and maintained by a central group.
- fxtentacle 7y ago
  I feel like Bazel and Cmake have solved this sufficiently well. I regularly compile larger projects from scratch and usually the process is to call vcvars (I'm on Windows) and then Bazel and then an hour later I have a folder full of cached lib and dll files so that linking my actual executable only takes a second. I would be so incredibly grateful to have something comparable in JavaScript that quickly checks static typing for my entire project. But the closest I have found is Rust/Elm, so not using JavaScript anymore.
tmcw 7y ago
How to write a silly post about JavaScript dependencies: - Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems. - Talk about devDependencies and dependencies without considering how they might be entirely different, say… one kind gets bundled, the other doesn't. - Always use npm, so you can highlight duplicates. Don't use yarn. - Adopt a narrow definition of 'dependency hell' so you can focus on the thing that JavaScript does imperfectly (minimizing number of dependencies) and avoid talking about the things that it does well (handling conflicting deps, solving diamond deps, resolving dependency chains) - Don't worry about explaining how any of the issues you're discussing have real-world implications. Have companies been failing because they run out of disk space because of node_modules? - Take automated security audits extremely seriously. A backtracking bug in the argument parser for the web framework's local CLI tool? Of course it's not exploitable, when… it's a web app, and the CLI tool is something you run locally. But add it to the count as if it is!
- seemslegit 7y ago
  How to downplay the severity of the problem: Ignore the ridiculously small functionality scope that acceptably qualifies for a js package in a way that does not exist in any other ecosystem and ads a big multiplier in front of all the other risks involved.
  - jessaustin 7y ago
    If these claims were true, we would have seen some examples by now. Instead, we get some tired invocations of "leftpad!" along with some more tired FUD. Who has been hacked because they used javascript instead of some other "more mature" language?
    - seemslegit 7y ago
      There is more to dependency risk than 'getting hacked' - correctness, maintainability, IP due diligence all play a role.
      - 7y ago
tomaszs 7y ago
I treat Npm packages like a JavaScript and PHP. There was a need and it was done. In the short run it is good enough. But in the long run it may become complicated. For example what about essencial packages that wont be maintained anymore? What about depending on millions of lines of code from unknown sources? What about single dependency that is npm? It is a risk too.
jessaustin 7y ago
200 megabytes is not that bad. I’ve seen it rise above 700 MB easily. A reasonable complaint! Instead of npm, try pnpm [0]. It uses file links to avoid duplication. [0] https://pnpm.js.org/ https://pnpm.js.org/
montroser 7y ago
It's not really hell though, is it? Hell is infinite pain and agony. This is more like an occasional bother. Yes, we get comically large numbers when we look through node_modules. But, real "dependency hell" is when you have situations that take unbounded manual effort to resolve. How often do we end up with impossible circular dependencies? Or conflicting versions clobbering each other? Or non-deterministic builds introducing random behavior? That all is commonplace even today with other platforms like python, and rarely an issue with node. I'd much rather occasionally sneer at the size of node_modules than any of that actual hell.
- ng12 7y ago
  In my experience is the people who complain most audibly about npm are the same folks who think nothing of spending a few days debugging classpath issues.
- uhoh-itsmaciek 7y ago
  Yes, exactly. I used to think isolating dependencies like npm/yarn do, where each package has its own copy of its own dependency tree (ignoring deduping), was crazy, and potential conflicts should be a forcing function to encourage you to minimize dependencies in the first place. But then I started using these systems and it's by far the least time I've ever spent worrying about deps. There are downsides to this and how it encourages dependency proliferation, and I still believe in minimizing your dependencies, but it's the least bad system I've seen.
zacksinclair 7y ago
I can't help but find it ironic that in middle of this oft-repeated JS bitch fest the author's own company is promoting their NPM package.
- jessaustin 7y ago
  You have to market differently to different people. TFA is intended for devs who are forced to use javascript even though they prefer python 2. They would write a different article for skilled js developers.
bob1029 7y ago
Sometimes I feel like I am becoming increasingly out-of-touch with reality with how far this NPM/JS ecosystem nonsense has been taken. Is anyone still using this crap for de novo business applications that have real-world consequences? I have no problem if you want to build fantastical, sprawling node implementations for fun and potentially personal profit, but when your decisions start to involve other professionals who are simply trying to get the features implemented so they can go home... I feel like we need to really think about the consequences of making these kinds of technical choices, especially in 2020 given all the horror stories circulating. Someone is eventually going to have to come in after the fact and scrape all the gore off the walls and redo your webapp in .NET/Rust/Go/etc. Why not start there and be done with it? You could save so much frustration. Is it about subjective/artistic preferences? At what point should the business owner start getting involved in these technical decisions? If the business owner found out their webapp was trapped in hell because you had a personal aesthetic attraction to one language over another much more pragmatic language, how do you think they would respond?
- mythrwy 7y ago
  Full page POST reloads on each data submission aren't going to fly with most users, even business users. Neither is reloading a page to get new data. My personal feeling about CRUD is this, you don't have to get fully on the crazy train and spend all your time with packages and build tools to achieve a nice product. Yarn is better for me. Very sparing package use. And a little light Vue with server side templating. No I don't need you managing routes there JS framework, that never was a good idea in my estimation. (Or maybe it is and I don't get it but I like incremental changes and simplicity). Once the dust settles perhaps I'll move on to a more complex setup but I don't have time for it now. I'm trying to get stuff done. However this doesn't have to mean eliminating modern behavior or appearance. Which means some JavaScript. It's going to be required in general by end users.
  - stepbeek 7y ago
    Fwiw I've found turbolinks gives me enough of an SPA feel that POST reloads aren't a huge problem. If we need a very dynamic experience then Vue would-be my go to tool,but for the other 90% this is fine.
takinola 7y ago
So what exactly is the problem? I see a lot of talk about dependency hell and node_modules bloat but I don't really hear much about what the actual downside is. Is it performance (I haven't experienced it)? Is it taking up disk space (I have terabytes to spare)? Does it add complexity (I would argue it reduces complexity)? What exactly is the problem?
calibas 7y ago
If you think this is just npm, try running "cargo update" on a older Rust project.
seumars 7y ago
This wouldn't be much of a problem if developers weren't as reliant on frameworks as JS developers are — who for some reason love to blame others for "reinventing the wheel". What do people even use lodash for? Do you really need a whole library because you can't write a function to sort an array? This all reminds me of the npm left-pad scandal. To me the problem was not that a single package broke thousands of projects, but that thousands of developers depended on a package for such a simple feature.
8bitsrule 7y ago
People who sin shouldn't complain about winding up in hell.

133 points by nikolalsvk 7y ago

I'm not buying the narrative of "js dependency hell what are they thinking" anymore. Try installing a package for Rust. Or Go. Or ... any language, really.

bsagdiyev 7y ago
I’m not sure whataboutism is the way to fix the well-known issues the JS ecosystem has in package management. When I’ve installed a Go package or Python package I haven’t ended up with sometimes hundreds or thousands of sub-dependencies. Some packages can be ridiculous, but nothing like I have seen running “npm i” for something that seems like it should be simple. I apologize for not having an example off-hand but this keeps coming up precisely because it is an issue unique to JS in this case, even if other languages have it to a degree.
- sippeangelo 7y ago
  Try Feathers, a "lightweight web-framework for creating real-time applications and REST APIs" clocking in at about 600 transitive dependencies. How do you audit this?
  - bsagdiyev 7y ago
    Link for the curious: https://github.com/feathersjs/feathers https://github.com/feathersjs/feathers
  - jfkebwjsbx 7y ago
    I think they just don't audit nor review the code they depend on.
- dmitriid 7y ago
  It's now whataboutism. It's finger pointing from a lot of people who develop in languages whose story is only marginally better. As I wrote in a sibling comment, I have a project I haven't even started yet in Rust [1]. And it's measly 6 dependencies pull a total of 197. [1] https://news.ycombinator.com/item?id=22841742 https://news.ycombinator.com/item?id=22841742
  - bsagdiyev 7y ago
    I took a look, this sounds more unique to Rust than Go or Python, but they all can do it to some degree. I have no hands on experience with Rust to really have any reasonable input on this so I don’t want to make an uninformed statement.
  - jfkebwjsbx 7y ago
    If what you say is true, then Rust is suffering the same fate. 197 dependencies is too many to trust.
shakna 7y ago
Gatsby has 19k dependencies. I don't think I can find something like that for Rust. The largest, and not exactly admirable, I can find is reqwest that drags in 97. Whilst 100 is a huge number... It's an enormous gap from the many thousands.
- dmitriid 7y ago
  Javascript doesn't have a standard library (something a lot of languages provide), and nodejs's built-in functions library are... lacking. Also, the culture of providing small packages, being around for longer, and so on and so forth... I have a project I haven't even started yet in Rust. Three dependencies in Cargo.toml. They download and install 34 dependencies. I've now added three more (from kube-rs readme). It's now 197 dependencies. And so on.
- mrighele 7y ago
  And that it is not even the worst I have seen. I think the worst offenders are development tools like webpack. I have a small project that I work on from time to time which uses 5 libraries (react, a map library and a chart library), typescript, and react-scripts (which I guess pulls in webpack and all the rest). This is what happens when I run a npm audit (I havent' touched it for a couple of months) > found 38934 vulnerabilities (38916 low, 18 moderate) in 906346 scanned packages > run `npm audit fix` to fix 38620 of them. > 314 vulnerabilities require manual review. See the full report for details. While the real numbers are probably lower because there is a lot of duplication inside the node_modules folder, I find this ... astounding.
swiley 7y ago
There are bad packages in rust, go, python (1), even C. The worst of these are closer to the norm for JavaScript though. Personally I don’t feel comfortable running JavaScript packages outside of some kind of sandbox (usually a VM) on my personal computers. (1) in fact, seeing some of the worst kept me away from python for far too long until I realized plenty of python software doesn’t even need pypi and most only need it for a couple things with no transitive dependencies
jt2190 7y ago
Yes. Using third-party code is not a binary decision between “write everything yourself” and “use all the things”, it is part of the ongoing conversation about optimization: Development time, deployable size, build time, tool support, community support, training, documentation, etc.
CyberDildonics 7y ago
If you mean that javascript gets a disproportional amount of hate, that might be because of how much more it is being used. It is true that rust seems to be ignoring the sins of the past and diving right in to having lots of dependencies, with the solution being to manage them instead of weeding them out in some way.
tristanstcyr 7y ago
I agree that this isn't totally specific to JavaScript. It's just that the ecosystem is very extensive and so there are a lot of packages, because you don't need to write your own code for a lot of existing problems. There is perhaps a something about the JavaScript audience tending to attract less advanced implementers. So you end up with a lot of otherwise considered trivial and low-quality utilities being published and pulled down from the registry. Rust and Go tend to be for more advanced scenarios where things like performance, security and support matters tremendously.

awinter-py 7y ago

express (common web framework) seems to have only 51 deps in lockfile jekyll has 13 I don't disagree that lots of deps = supply chain risk But there have always been variations between projects & kinds of projects re how many deps they pull in. Try following someone's ipython data science tutorial, it's requirements.txt for days I think lack of a standard lib early on plus hipster functional culture made the '10s JS leaders like small monofunctional libs. Focus on transpilation plus lack of tree-shaking in early versions of webpack may have also created an incentive for small things.

dgellow 7y ago
A python equivalent to "express" would be Flask https://palletsprojects.com/p/flask/ https://palletsprojects.com/p/flask/ I see 4 dependencies in the setup.py, and they are all from the same team of maintainers (i.e the Pallet team): - Werkzeug - Jinja2 - itsdangerous - click https://github.com/pallets/flask/blob/master/setup.py https://github.com/pallets/flask/blob/master/setup.py https://github.com/pallets/flask/network/dependencies https://github.com/pallets/flask/network/dependencies
- awinter-py 7y ago
  'pip freeze' results from a few other py web frameworks: sanic = 18, fastapi = 3 but one of them is a 7MB download (pydantic) yeah it's less, but also python has an amazing stdlib
- oefrha 7y ago
  To be fair, express is a production server itself whereas werkzeug is not, so you need to add something like gunicorn or uwsgi (yeah I know nginx supports uwsgi_pass natively). click is not necessary in production though (it's only used in the dev CLI). That said, 4 dependencies is the norm in Python land, whereas 13 dependencies is an outlier in JS land. Edit: oops, misread, gp said express has 51 deps, not 13.
- wffurr 7y ago
  The Python standard library is a lot more capable than the JS equivalents (Node or browser API). That eliminates the need for a lot of dependencies. There's been a TC39 proposal for a JS stdlib for about 2 years now: https://github.com/tc39/proposal-javascript-standard-library https://github.com/tc39/proposal-javascript-standard-library

pizzapim 7y ago

So what does NPM do differently than plugin managers from other languages? I can't name one where the dependency hell is this big.

leetrout 7y ago
My personal take having “grown up” with all of this - JS is one of the worst ecosystems for relying on external packages for everything. It’s more the culture than the tooling. One of the things I love about Go is generally people are a little more forgiving to copying something around a couple times instead of making a lib for everything.
- smitty1e 7y ago
  Inviting the question of why the culture hasn't sprouted a "package rationalizer" to produce a curated, tested, noise-filtered distribution of useful functionality. Come to think of it, something that does for packages what jQuery did to bring (some) sanity to browser incompatibilities.
  - ng12 7y ago
    Because we can't even agree on whether React or Vue or Angular is best. In fact this is the root of the issue. It's important to realize that `node_modules` for any project in Popular Framework X will be almost identical. Most of the packages are related to compiling, bundling, and the development environment and a small minority are actual client-side dependencies. Even the frameworks themselves are pretty light -- React has a grand total of 2 dependencies. It's packaging the entire toolchain that causes most of the bloat.
- pixelrevision 7y ago
  Go in particular has an amazing standard library and very clear guidelines on how to write code. JavaScript does not have either of those things so it promotes people releasing competing libraries with similar functionality written in various different styles.
- 7y ago

deleted 7y ago

[deleted]

sproketboy 7y ago

JavaScript is what happens when you lock toddlers in a room on their own for a month.

dwheeler 7y ago

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems. "small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code." Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open Source Software", The Linux Foundation & The Laboratory for Innovation Science at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L. Hoffman. https://www.coreinfrastructure.org/programs/census-program-ii/ https://www.coreinfrastructure.org/programs/census-program-i... (This is a contrast of citations from other works; you can see the specific citations in the report, but I'm trying to do this from a phone. In any case, the issue is the massive difference between JavaScript and everything else, and I think this quotation shows it nicely.)

dep_b 7y ago
iOS dependencies in Cocoapods are including less than 1 other packages on average (1). The most I see are Alamofire (a networking package people keep using for reasons that I do not fully understand) and Starscream (a Websocket implementation, that's pretty hard to write from scratch to be honest). Having unnecessary dependencies is pretty much frowned upon. One of the reasons is that the iOS SDK is really complete by itself while Javascript doesn't even ship with a usable Date implementation. If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically. (1) totally based on gut feeling, not based on hard data
- hn_throwaway_99 7y ago
  Java didn't have a decent date implementation until Java 8. > If somebody could just come up with a standard library for JavaScript that is really widely adopted I think the amount of packages can be cut back dramatically. As this article points out, I think that lodash is that for many people.
  - 7y ago

Waterluvian 7y ago

I have a project with a total of eight dependencies. I walked away for six months as it was stable. I come back to add a few features and npm tells me I have over 38 000 vulnerabilities of different severity levels. So many that 'npm audit' just freezes up. So there's that too.

samsquire 7y ago
My stuff breaks whenever I try upgrade it. People refactoring stuff.
- brobdingnagians 7y ago
  These types of issues are the biggest reason why I avoid javascript/npm projects. I came back to a project after a few months and it was broken, had to rewrite some parts and upgrade other parts just to get it to run again.
  - williamdclt 7y ago
    I'm sorry but for all the flaws of the NPM ecosystem, version locking is not one (both `npm` and `yarn` do much much better than anything in Python). There's no reason a correctly set-up project would magically break with time. I don't love NPM anymore than the next guy, but my blame will go to the dev in this case.
    - Waterluvian 7y ago
      Python dependency management is a disaster. But I find myself using far fewer dependencies with Python projects so I suffer the pain less.
      - jessaustin 7y ago
        This is more of that "wet streets cause rain" logic we hear about...
    - 7y ago

dgellow 7y ago

I think that we are in a situation where the number of dependency doesn't have any meaning anymore for people working on web frontend projects. Recently while revamping parts of the CI/CD at $WORK, I found out that running npm install for one of our frontend project downloads around 1 million dependencies. And npm audit reports a completely ridiculous number of security issues with them. It's just so absurd and nonsensical, nobody can understand what that even mean. Seems to me to be a crazy huge liability, but quite a lot of people seem to be fine with that. I don't know what to do from that information, but trying to create some awareness around the issue feels like talking about the number of operations per second a CPU does with someone who has zero idea what a realistic range is. It's seen as a high number, but doesn't have any meaning attached to it. Edit: I don't have access to the project anymore to check more details (such as installation time), but I checked some personal notes I took when I found this out, the redacted output from npm audit is the following $ npm audit <REDACTED> vulnerabilities found - Packages audited: 927016 Severity: <REDACTED> Low | <REDACTED> Moderate Done in 41.62s. So slightly below the 1 million mark. Edit 2: Now I'm wondering ... maybe the "packages audited" from npm audit doesn't mean "you have that number of packages, and I audited them". But if that's the case, that's a terrible UX, and I have zero idea what that number mean, which kind of support my point.

Ayesh 7y ago
A million dependencies sound horribly excessive, even by JS standards.
- dgellow 7y ago
  That was my reaction too.
awinter-py 7y ago
how long does that take to install? is there a local npm mirror / cache?
crazygringo 7y ago
I mean npm only has slightly more than 1 million packages, last I checked. This doesn't make even the slightest bit of sense. Did somebody somehow just import... everything by accident? Did npm hit a bug? Is there some infinite recursion going on that eventually hits a wall? This obviously isn't representative.

Dunedan 7y ago

That page itself is an example what's wrong with using JavaScript on the web: It's used in places where it's not necessary. In this case the site doesn't apply a stylesheet with JavaScript disabled. JavaScript is great when it's used for enabling interactive websites, but nowadays more and more websites use it to break features, which don't need JavaScript at all.

parhamn 7y ago

What’s worse is that the whole installing multiple versions of the same package for different sub dependencies doesn’t make any sense when when the language supports singletons and module level variables with has no way of specifying which version to import. I was shocked this behavior exists when a site broke by upgrading a sub dependency. Turned out they both installed react so they used a different “creatContext” and now there were two context instances instead of one.

tristanstcyr 7y ago
React would normally be a peerDependency and so you shouldn't run into this problem if dependencies are specified properly. If not, NPM module flattening should take care of it. Then worst case, you can also tell webpack to bundle a specific installation. There are some ways to solve these problems...
- parhamn 7y ago
  Sure (upvote) and we dealt with it accordingly after some confused digging. On a more general level I’ve never seen something like that in a package manager. It seems as though some of the language features (like the singleton I mentioned) would make it obvious that arbitrarily duplicating packages and scoping them to sub packages is a major no-no, as in most languages I’ve dealt with.
  - jessaustin 7y ago
    It seems like a weird thing about React, or maybe how you're using it, that this is a problem. Most packages, and the language itself, handle multiple simultaneous versions of dependencies without issue. If packages are doing the old-fashioned "stick a property on window" thing instead of just getting required or imported browserify-style, they can step on each others' toes (and it's totally possible that React is dumb enough to step on its own toes). It has been years since that was necessary. browserify or more recent tools like parcel or webpack should be used instead.
    - 7y ago

johannes1234321 7y ago

My concern with npm packages isn't the number and small size of the packages itself, but that most of them are maintained and owned by individuals who do work with little review by others and who could disappear any time. If an individual goes away or takes harmful action it takes time till this is noticed and till a surviving fork emerges. If more libraries were maintained by groups in a shared/collaborative way, more of those risks would go away.

tluyben2 7y ago

I cannot understand how people work with this. I work with many different technologies and try to avoid JS, but some times I have to. The past weeks I have worked on a React Native project that was written by someone else; what a horror show. I mean it wasn't the worst code by the previous guy but even in a few months a lot is simply broken and not 'best practice' anymore. Compared to most other environments I work with, iteration is somewhat faster when you finally have everything working, but the dependencies and ecosystem is horrible imho. A bigger issue with all of it is that in my line of work, for the backend of firmware I cannot just depend on 3rd party libs as they will get audited, so I need to audit them before. Ofcourse most people don't have that issue.

williamdclt 7y ago
I don't agree with you on this. The JS (or NPM) way makes it a pain to audit a project due to thousands of dependencies, but it's far easier to "have everything working". `npm i && npm start` is as good as it gets in terms of "getting started" friction (with version locking): python is worse, ruby is worse, PHP is worse, Java is worse. React-Native has more moving pieces indeed, but it's not really JS's (or NPM's) fault: you get all problems of iOS and Android at once. I will definitely agree with you that working on React-Native is a pain, to be able to work with modern tools need to know JS, optionally TS, Obj-C, Swift, Java, Kotlin, Ruby (for Fastlane) and Gradle's custom language. Insane.
- dgellow 7y ago
  I've done 3 years of Golang as my main language, my experience is: - go build ./... builds everything from the current project - go test ./... tests everything from the current project - go install ./... installs everything from the current project That's one of the best thing ever. I can go to any project, I know how to build, test, install, and can directly be productive. On the other hand, I started learning C++ one year ago, and it's the exact opposite, each project has its own homemade build system that isn't supported by one platform or the other.
  - diggan 7y ago

appleflaxen 7y ago

If you want the 3D view of Gatsby dependencies without waiting for the initial 2D view to populate, you can go here: https://npm.anvaka.com/#/view/3d/gatsby https://npm.anvaka.com/#/view/3d/gatsby

overgard 7y ago

Sadly the alternative isn't much better: the old school model of not having packages often leads to libraries never getting updated at all. In C++ often adding a library is such a chore once you get through all the linking issues, as you battle the 4000 conflicting compiler switches, and compiler versions, that once it works you never want to change it again. I think maybe the best solution we have so far are robust "batteries included" standard libraries like python, where even if they're huge theyre at least curated and maintained by a central group.

fxtentacle 7y ago
I feel like Bazel and Cmake have solved this sufficiently well. I regularly compile larger projects from scratch and usually the process is to call vcvars (I'm on Windows) and then Bazel and then an hour later I have a folder full of cached lib and dll files so that linking my actual executable only takes a second. I would be so incredibly grateful to have something comparable in JavaScript that quickly checks static typing for my entire project. But the closest I have found is Rust/Elm, so not using JavaScript anymore.

tmcw 7y ago

How to write a silly post about JavaScript dependencies: - Don't mention other languages, lest you reveal that most of them have similar dependency bloat problems. - Talk about devDependencies and dependencies without considering how they might be entirely different, say… one kind gets bundled, the other doesn't. - Always use npm, so you can highlight duplicates. Don't use yarn. - Adopt a narrow definition of 'dependency hell' so you can focus on the thing that JavaScript does imperfectly (minimizing number of dependencies) and avoid talking about the things that it does well (handling conflicting deps, solving diamond deps, resolving dependency chains) - Don't worry about explaining how any of the issues you're discussing have real-world implications. Have companies been failing because they run out of disk space because of node_modules? - Take automated security audits extremely seriously. A backtracking bug in the argument parser for the web framework's local CLI tool? Of course it's not exploitable, when… it's a web app, and the CLI tool is something you run locally. But add it to the count as if it is!

seemslegit 7y ago
How to downplay the severity of the problem: Ignore the ridiculously small functionality scope that acceptably qualifies for a js package in a way that does not exist in any other ecosystem and ads a big multiplier in front of all the other risks involved.
- jessaustin 7y ago
  If these claims were true, we would have seen some examples by now. Instead, we get some tired invocations of "leftpad!" along with some more tired FUD. Who has been hacked because they used javascript instead of some other "more mature" language?
  - seemslegit 7y ago
    There is more to dependency risk than 'getting hacked' - correctness, maintainability, IP due diligence all play a role.
    - 7y ago

tomaszs 7y ago

I treat Npm packages like a JavaScript and PHP. There was a need and it was done. In the short run it is good enough. But in the long run it may become complicated. For example what about essencial packages that wont be maintained anymore? What about depending on millions of lines of code from unknown sources? What about single dependency that is npm? It is a risk too.

jessaustin 7y ago

200 megabytes is not that bad. I’ve seen it rise above 700 MB easily. A reasonable complaint! Instead of npm, try pnpm [0]. It uses file links to avoid duplication. [0] https://pnpm.js.org/ https://pnpm.js.org/

montroser 7y ago

It's not really hell though, is it? Hell is infinite pain and agony. This is more like an occasional bother. Yes, we get comically large numbers when we look through node_modules. But, real "dependency hell" is when you have situations that take unbounded manual effort to resolve. How often do we end up with impossible circular dependencies? Or conflicting versions clobbering each other? Or non-deterministic builds introducing random behavior? That all is commonplace even today with other platforms like python, and rarely an issue with node. I'd much rather occasionally sneer at the size of node_modules than any of that actual hell.

ng12 7y ago
In my experience is the people who complain most audibly about npm are the same folks who think nothing of spending a few days debugging classpath issues.
uhoh-itsmaciek 7y ago
Yes, exactly. I used to think isolating dependencies like npm/yarn do, where each package has its own copy of its own dependency tree (ignoring deduping), was crazy, and potential conflicts should be a forcing function to encourage you to minimize dependencies in the first place. But then I started using these systems and it's by far the least time I've ever spent worrying about deps. There are downsides to this and how it encourages dependency proliferation, and I still believe in minimizing your dependencies, but it's the least bad system I've seen.

zacksinclair 7y ago

I can't help but find it ironic that in middle of this oft-repeated JS bitch fest the author's own company is promoting their NPM package.

jessaustin 7y ago
You have to market differently to different people. TFA is intended for devs who are forced to use javascript even though they prefer python 2. They would write a different article for skilled js developers.

bob1029 7y ago

Sometimes I feel like I am becoming increasingly out-of-touch with reality with how far this NPM/JS ecosystem nonsense has been taken. Is anyone still using this crap for de novo business applications that have real-world consequences? I have no problem if you want to build fantastical, sprawling node implementations for fun and potentially personal profit, but when your decisions start to involve other professionals who are simply trying to get the features implemented so they can go home... I feel like we need to really think about the consequences of making these kinds of technical choices, especially in 2020 given all the horror stories circulating. Someone is eventually going to have to come in after the fact and scrape all the gore off the walls and redo your webapp in .NET/Rust/Go/etc. Why not start there and be done with it? You could save so much frustration. Is it about subjective/artistic preferences? At what point should the business owner start getting involved in these technical decisions? If the business owner found out their webapp was trapped in hell because you had a personal aesthetic attraction to one language over another much more pragmatic language, how do you think they would respond?

mythrwy 7y ago
Full page POST reloads on each data submission aren't going to fly with most users, even business users. Neither is reloading a page to get new data. My personal feeling about CRUD is this, you don't have to get fully on the crazy train and spend all your time with packages and build tools to achieve a nice product. Yarn is better for me. Very sparing package use. And a little light Vue with server side templating. No I don't need you managing routes there JS framework, that never was a good idea in my estimation. (Or maybe it is and I don't get it but I like incremental changes and simplicity). Once the dust settles perhaps I'll move on to a more complex setup but I don't have time for it now. I'm trying to get stuff done. However this doesn't have to mean eliminating modern behavior or appearance. Which means some JavaScript. It's going to be required in general by end users.
- stepbeek 7y ago
  Fwiw I've found turbolinks gives me enough of an SPA feel that POST reloads aren't a huge problem. If we need a very dynamic experience then Vue would-be my go to tool,but for the other 90% this is fine.

takinola 7y ago

So what exactly is the problem? I see a lot of talk about dependency hell and node_modules bloat but I don't really hear much about what the actual downside is. Is it performance (I haven't experienced it)? Is it taking up disk space (I have terabytes to spare)? Does it add complexity (I would argue it reduces complexity)? What exactly is the problem?

calibas 7y ago

If you think this is just npm, try running "cargo update" on a older Rust project.

seumars 7y ago

This wouldn't be much of a problem if developers weren't as reliant on frameworks as JS developers are — who for some reason love to blame others for "reinventing the wheel". What do people even use lodash for? Do you really need a whole library because you can't write a function to sort an array? This all reminds me of the npm left-pad scandal. To me the problem was not that a single package broke thousands of projects, but that thousands of developers depended on a package for such a simple feature.

8bitsrule 7y ago

People who sin shouldn't complain about winding up in hell.