4 ms·

Going to try and give a reasonably systematic answer to this: Firstly: the title of the post is different to the content. The title asks why dependencies are a

Going to try and give a reasonably systematic answer to this: Firstly: the title of the post is different to the content. The title asks why dependencies are a mess, the content cites problems with bundlers. These are entirely distinct topics. Bundlers don't handle dependencies and package managers (mostly) don't concern themselves with bundling. They're different problems and have surprisingly little in common. So the above breaks it down to two questions: 1. Content: Is javascript bundling a mess? 2. Title: Are javascript dependencies a mess? To answer question (1) let's first look at it comparatively. Are other language bundlers a mess? Obviously they don't have any as bundling is a client-side problem and javascript is the only client-side target. So it's hard to gauge this. Maybe bundling is a fundamentally hard problem that will always necessarily be a mess. Subjective opinion: I think there are obvious ways bundling could be done better and it surprises me they haven't been done but speaking from a place of inaction I'm reluctant to be too critical here. To answer question (2) let's look at it comparatively again: are javascript's package management and library ecosystem more or less of a mess than Java's, Python's, PHP's, Golang's, Rust's, etc.? Some may disagree here but I think the answer is objectively "no": - The Ant/Maven/Gradle/Bazek/sbt ecosystem is absurd complexity, and I've seen plenty Java .m2 dirctories worse than node_modules. I think Java must only escape the criticism JS gets here because every Java dev I've met is so reliant on their IDE they don't even know .m2 exists (same goes for ObjectiveC/Swift) - Python is both better and worse than Java. On one hand, the manager ecosystem is even more unapproachable: the setuptools easy_install pip pipx conda poetry venv pyenv nightmare isn't even helped by consistently demarcated config files in your repo root: good luck figuring out what to run depending on whether there's setup.cfg setup.py requirements.txt pyproject.toml etc. (if they even exist at all - pip not requiring project root dep config is why so many Python readmes just have long handwritten accounts of which deps you need to install manually). On the other hand, python's actual dependency tree handling errs on the oversimplified side. Have two transitive dependencies on separate incompatible versions of a single library? Tough shit, rewrite your code. Go seems to be leaning too heavily on Git and falling into the trap of modelling their package management system on the implementation rather than modelling the implementation on user needs. It's not terrible though, but that's perhaps due to its age. PHP is probably the least messy, and Rust - while making some mistakes - is new enough to be following some good modern examples. But overall NPM is by far one of the LEAST messy package management ecosystems.

mike_hearn 4y ago
Other languages do have bundlers and JS is not the only client side target. In the Java world this is called building an uber-jar and if you want to "tree shake" it you'd use ProGuard or R8. Conveniently, R8 uses the same config file syntax as ProGuard so there's a standard for defining what to keep/rename/etc. Or if compiling to native code, then GraalVM native image which deletes dead code and creates a single binary at the same time. These tools seem to work better than the JS equivalents in my experience, or at least they are a lot more stable. It's probably because they're products of companies who want to maintain stability for their users, they aren't just projects that were written, uploaded to GitHub and then eventually abandoned. ProGuard has been maintained for over two decades, R8 was written for Android and is maintained by Google, GraalVM is a 10+ year project maintained by Oracle etc. The Java dependencies situation is not so messy, I think. You've named a variety of different build tools but they all agree on where to find dependencies, how they are named, how to download them, and dependency graphs aren't quite as large or deep as they get in the JS world. Most projects use Maven or Gradle. Ant is obsolete, Bazel is really only used by Google or Google-adjacent firms and sbt is for a different language. Your ~/.m2/repository directory is a cache of every library version you ever downloaded, it's not the set of JARs used in a specific project. The biggest problems JVM world modules have are: 1. The half-completed, not really successful attempt at modular JARs which encapsulate their internals. Adoption is slow, and it's not unheard of to find popular libraries with broken module metadata. 2. If you get version conflicts the build tool has to pick one version, usually the highest. The JVM can do isolation of dependency graphs, but the integration between build/deployment tools and those APIs isn't there. Fine if that library is backwards compatible. A mess if it's not. 3. The general prevalence of 'shading' (bundling) as a way to resolve such issues. 4. The near total reliance on Maven Central as the one big server. Of course most language ecosystems have this issue, but it used to be better with many dependencies sitting on other repositories. Over time those other repos went away and now it's all just Sonatype and Central.
- lucideer 4y ago
  That's a good insight on bundling and not one I had considered. I guess bundling for dynamic network delivery is so different in my mind to bundling for portability (& possibly one-time network delivery). But while they're certainly different I'm sure there are many similar concerns. Just reinforces my instinct then I guess, that JS bundlers are ripe for improvement. On your remarks around the Java package management ecosystem, I was aware of a fair amount what you point out, & the rest is helpfully elucidating, but it didn't seem to contradict my point that it's pretty messy?

mike_hearn 4y ago

Other languages do have bundlers and JS is not the only client side target. In the Java world this is called building an uber-jar and if you want to "tree shake" it you'd use ProGuard or R8. Conveniently, R8 uses the same config file syntax as ProGuard so there's a standard for defining what to keep/rename/etc. Or if compiling to native code, then GraalVM native image which deletes dead code and creates a single binary at the same time. These tools seem to work better than the JS equivalents in my experience, or at least they are a lot more stable. It's probably because they're products of companies who want to maintain stability for their users, they aren't just projects that were written, uploaded to GitHub and then eventually abandoned. ProGuard has been maintained for over two decades, R8 was written for Android and is maintained by Google, GraalVM is a 10+ year project maintained by Oracle etc. The Java dependencies situation is not so messy, I think. You've named a variety of different build tools but they all agree on where to find dependencies, how they are named, how to download them, and dependency graphs aren't quite as large or deep as they get in the JS world. Most projects use Maven or Gradle. Ant is obsolete, Bazel is really only used by Google or Google-adjacent firms and sbt is for a different language. Your ~/.m2/repository directory is a cache of every library version you ever downloaded, it's not the set of JARs used in a specific project. The biggest problems JVM world modules have are: 1. The half-completed, not really successful attempt at modular JARs which encapsulate their internals. Adoption is slow, and it's not unheard of to find popular libraries with broken module metadata. 2. If you get version conflicts the build tool has to pick one version, usually the highest. The JVM can do isolation of dependency graphs, but the integration between build/deployment tools and those APIs isn't there. Fine if that library is backwards compatible. A mess if it's not. 3. The general prevalence of 'shading' (bundling) as a way to resolve such issues. 4. The near total reliance on Maven Central as the one big server. Of course most language ecosystems have this issue, but it used to be better with many dependencies sitting on other repositories. Over time those other repos went away and now it's all just Sonatype and Central.

lucideer 4y ago
That's a good insight on bundling and not one I had considered. I guess bundling for dynamic network delivery is so different in my mind to bundling for portability (& possibly one-time network delivery). But while they're certainly different I'm sure there are many similar concerns. Just reinforces my instinct then I guess, that JS bundlers are ripe for improvement. On your remarks around the Java package management ecosystem, I was aware of a fair amount what you point out, & the rest is helpfully elucidating, but it didn't seem to contradict my point that it's pretty messy?