9 ms·

It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the loc

by jchw 5y ago

It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json. Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version. This avoids the node-ipc issue entirely, at least until you update the go.mod.

lamontcg 5y ago
How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?
- jchw 5y ago
  As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
  - lamontcg 5y ago
    Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5? Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints? Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
    - coder543 5y ago
      I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file. If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
      - lamontcg 5y ago
        Well after 12 years of using ruby and bundler and maintaining a bundler-ish depsolver at work, and playing around a bit with cargo, I can say that it is becoming clearer as to why I don't grok go modules at all. The lack of a depsolver is a curious choice... I don't think my example was remotely "broken" at all, that's just another day doing software development.
        deleted 5y ago
        [deleted]
        coder543 5y ago
        > I don't think my example was remotely "broken" at all, that's just another day doing software development. It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby. I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.
    - 5y ago
  - typical182 5y ago
    I think you might have a small typo here: > by choosing the lowest version specified in any package that depends on a given package It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
    - jchw 5y ago
      Yep, I regrettably described the system wrong. It’s the highest version specified, and thus minimum version possible.
      - typical182 5y ago
        That's a nice & concise way to describe it.
        lamontcg 5y ago
        Minimalistic Version Selection might have been a better term for it. That did confuse me a lot. Selecting the maximum makes a lot more sense, since that gets SemVer assumptions correct.
steveklabnik 5y ago
This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.
- jchw 5y ago
  Yes, this is true; I think the article is definitely making the distinction vs NPM and not Rust.
  - rjmorris 5y ago
    The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.
    - jchw 5y ago
      It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.
      - munificent 5y ago
        They definitely should have said that then. Simply referring to "lockfiles" paints with a very broad brush that includes a number of package managers that don't have the problems that NPM does.
        jchw 5y ago
        I believe Cargo does still have less strictness over dep versions than Go modules, since it will never use a module newer than the one specified in any go.mod file. Lockfiles are generally not honored recursively, and I don’t think Cargo is different here? Hope I’m not spreading misinformation, though I couldn’t find any docs with a cursory glance. I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.
woojoo666 5y ago
If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?
- typical182 5y ago
  That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself. Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730 https://news.ycombinator.com/item?id=30871730 The best complete explanation is probably here: https://research.swtch.com/vgo-principles https://research.swtch.com/vgo-principles
  - woojoo666 5y ago
    But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update
    - hashhar 5y ago
      The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.
      - woojoo666 5y ago
        But if its a manual explicit choice, that adds more friction to these patches, and many developers may not update them at all. It's a trade off
        5y ago

by jchw 5y ago

lamontcg 5y ago

How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?

jchw 5y ago
As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
- lamontcg 5y ago
  Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5? Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints? Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
  - coder543 5y ago
    I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file. If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
    - lamontcg 5y ago
      Well after 12 years of using ruby and bundler and maintaining a bundler-ish depsolver at work, and playing around a bit with cargo, I can say that it is becoming clearer as to why I don't grok go modules at all. The lack of a depsolver is a curious choice... I don't think my example was remotely "broken" at all, that's just another day doing software development.
      - deleted 5y ago
        [deleted]
      - coder543 5y ago
        > I don't think my example was remotely "broken" at all, that's just another day doing software development. It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby. I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.
  - 5y ago
- typical182 5y ago
  I think you might have a small typo here: > by choosing the lowest version specified in any package that depends on a given package It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
  - jchw 5y ago
    Yep, I regrettably described the system wrong. It’s the highest version specified, and thus minimum version possible.
    - typical182 5y ago
      That's a nice & concise way to describe it.
      - lamontcg 5y ago
        Minimalistic Version Selection might have been a better term for it. That did confuse me a lot. Selecting the maximum makes a lot more sense, since that gets SemVer assumptions correct.

steveklabnik 5y ago

This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.

jchw 5y ago
Yes, this is true; I think the article is definitely making the distinction vs NPM and not Rust.
- rjmorris 5y ago
  The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.
  - jchw 5y ago
    It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.
    - munificent 5y ago
      They definitely should have said that then. Simply referring to "lockfiles" paints with a very broad brush that includes a number of package managers that don't have the problems that NPM does.
      - jchw 5y ago
        I believe Cargo does still have less strictness over dep versions than Go modules, since it will never use a module newer than the one specified in any go.mod file. Lockfiles are generally not honored recursively, and I don’t think Cargo is different here? Hope I’m not spreading misinformation, though I couldn’t find any docs with a cursory glance. I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.

woojoo666 5y ago

If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?

typical182 5y ago
That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself. Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730 https://news.ycombinator.com/item?id=30871730 The best complete explanation is probably here: https://research.swtch.com/vgo-principles https://research.swtch.com/vgo-principles
- woojoo666 5y ago
  But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update
  - hashhar 5y ago
    The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.
    - woojoo666 5y ago
      But if its a manual explicit choice, that adds more friction to these patches, and many developers may not update them at all. It's a trade off
      - 5y ago