4 ms·

At some point we will do a more in depth write-up about this, but I think we have a very unique take on this at RunKit as we've effectively installed every vers

At some point we will do a more in depth write-up about this, but I think we have a very unique take on this at RunKit as we've effectively installed every version of every package. It's actually more interesting than that, we've installed every reasonable combinatorial dependency result of every version of every package on npm. Long story short, I think there was tremendous insight in npm 2's original model of having every dependency duplicated. Beyond allowing neat things like diffing two versions of the same package (https://runkit.com/tolmasky/api-diff-example-4 https://runkit.com/tolmasky/api-diff-example-4 ), I think it most accurately portrays the meaning of a package/version. The fundamental problem with trying to "unify" all the dependencies a program will need, is that it almost guarantees that code will be running in an untested state. Package A was written when Dependency 1 was at version 1.0.0. By assigning it the semver range 1.x.x (instead of the strict 1.0.0 dependency), that means that at some later point it will run against code its never seen or was tested against before. Sure, semver ranges are supposed to protect against this having a "meaningful" effect, but this simply kicks the can down the responsibility chain, changing the error to "human error" as if it makes a difference. For example, let's say at the time of publishing Dependency 1 had a bug that Package A accounted for. According to semver, a fix for this bug would qualify as a 1.0.1, but now Package A's workaround may actually break. The fact of the matter is: 1. rarely is code fully tested, 2. humans can't be trusted to properly assign semver, 3. semver doesn't even account for every possible "meaningful" change (I've seen performance changes that had devastating effects despite the "result" being the same). Different code is different code. Now, you may be saying, what does this have to do with npm, it sounds like what you want is strict versioning -- the problem is that compounding this already tenuous situation with versions that change depending on the combination of dependencies you have in order to maximize deduping makes this problem even worse. Let me give you a security example: 1. Package A relies on Dependency 1, ranged at 1.x.x. 2. Dependency 1 is found to have a security problem, so a 1.0.1 fix is issued. 3. Package A correctly no longer suffers from this problem. Later, Package B is added, which someone has maliciously, 10 sup-denendencies deep, added a strict dependency to Dependency 1 1.0.0. The deduping algorithm now says: 1. 1.x.x and 1.0.0 are the requirements, I will use 1.0.0. Now Package A is using an insecure version of Dependency 1. Previously, if every package was given its own dependencies, the issue would be scoped to Package B, instead it now infects everyone that relies on this Dependency. Sure, you can fix this by saying "dedupe after giving every package its highest possible dependency", but now you still suffer from monkey-patching affecting other code. You end up with very difficult to understand results. Sometimes, all the code uses the same package, other times not, seemingly randomly. The problem is that a performance issue is being turned into a functional issue. Package should be black boxes, not things you change the guts of regularly. There exists a tool for sharing dependencies: "peerDependencies". This says "I expect this to be around" vs. "I am including this". The basic idea here is that you would never say "let's arbitrarily replace functions I've written in my program with approximately the same function in order to be able to be able to share them and reduce program size". This is nuts! Package versions should be considered hashes that only coincidentally have human names. NPM 2 is the closest I've seen anything come to this, and it creates the least surprises. Most the remedies I've seen revolve around solving performance or efficiency problems this has. But I think it should first be important to make it right, THEN make it fast. You can definitely imagine a system that uses content-hashes to get the space efficiency without the cost of the conceptual separation. In fact, IED does this quite well: https://github.com/alexanderGugel/ied https://github.com/alexanderGugel/ied

Dylan16807 10y ago
I'm a little confused on what you're proposing. Should package A get the update to 1.0.1 or not? If it does get the update then you still have the problem of "Sometimes, all the code uses the same package, other times not, seemingly randomly." If it doesn't get the update then you keep having a security flaw.
- tolmasky 10y ago
  There are two things going on here: 1. the desire to allow every package to install its dependencies separately to guarantee deterministic behavior, and 2. not too take up a lot of disk space. The desired result in this SPECIFIC scenario is simple: 1. Package B contains the faulty Dependency 1 1.0.0. 2. Package A contains the correct Dependency 1 1.0.1. This is the best case scenario in this situation: Package A is not subject to the security problems of Dependency 1 1.0.0. Package B IS of course, but critical parts of your code are not exposed. If these were the only two packages involved, it would be simple. However the problem arrises from having LOTS of packages, and thus wanting to dedupe, which then leads to having to resolve these semver ranges, or being left with the unfortunate indeterministic behavior of sometimes these being distinct installations and sometimes not. Thus, to achieve the desired result above AND ensure packages are only installed once on disk, you want to dedupe at the FILESYSTEM level. This is how ied behaves (and I encourage everyone to take a look at its implementation, it is very well done). Essentially, every package is installed and contains symlinks to their dependencies, that way it only takes up space on the hard disk once. However, thanks to node's treat-symlinks-as-real-filepaths option, as far as node is concerned these are distinct installations (as if you had done no deduping at all). Now you have the result from above AND the minimum space on disk.
  - bassislife 10y ago
    Your proposal only works if you don't need ABI-compatibility, especially relevant if the build process needs to be parallelizable. It all depends on what sense you provide to the terms: "minor version" and how you expect your software to be built.

Dylan16807 10y ago

I'm a little confused on what you're proposing. Should package A get the update to 1.0.1 or not? If it does get the update then you still have the problem of "Sometimes, all the code uses the same package, other times not, seemingly randomly." If it doesn't get the update then you keep having a security flaw.

tolmasky 10y ago
There are two things going on here: 1. the desire to allow every package to install its dependencies separately to guarantee deterministic behavior, and 2. not too take up a lot of disk space. The desired result in this SPECIFIC scenario is simple: 1. Package B contains the faulty Dependency 1 1.0.0. 2. Package A contains the correct Dependency 1 1.0.1. This is the best case scenario in this situation: Package A is not subject to the security problems of Dependency 1 1.0.0. Package B IS of course, but critical parts of your code are not exposed. If these were the only two packages involved, it would be simple. However the problem arrises from having LOTS of packages, and thus wanting to dedupe, which then leads to having to resolve these semver ranges, or being left with the unfortunate indeterministic behavior of sometimes these being distinct installations and sometimes not. Thus, to achieve the desired result above AND ensure packages are only installed once on disk, you want to dedupe at the FILESYSTEM level. This is how ied behaves (and I encourage everyone to take a look at its implementation, it is very well done). Essentially, every package is installed and contains symlinks to their dependencies, that way it only takes up space on the hard disk once. However, thanks to node's treat-symlinks-as-real-filepaths option, as far as node is concerned these are distinct installations (as if you had done no deduping at all). Now you have the result from above AND the minimum space on disk.
- bassislife 10y ago
  Your proposal only works if you don't need ABI-compatibility, especially relevant if the build process needs to be parallelizable. It all depends on what sense you provide to the terms: "minor version" and how you expect your software to be built.