7 ms·

> There is no way for changes in the outside world—such as a new version of a dependency being published—to automatically affect a Go build. > Unlike most othe

> There is no way for changes in the outside world—such as a new version of a dependency being published—to automatically affect a Go build. > Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module. I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build. The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build. It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes. Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.

klodolph 5y ago
Discussing "what is a lockfile" is a bit of a headache because different languages have different files which do different things. Generally speaking, there's some file which specifies the dependency versions and some file with cryptographic checksums of the all transitive dependencies. In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock. Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.) Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC. If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.
- munificent 5y ago
  > any new, poisoned version of libB Conversely, you will get any old security-buggy version of libB instead. Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer. I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
  - philosopher1234 5y ago
    Go just expects you to manually trigger the updates. Thats all. It still is in favor of updating to take security fixes, so i think your argument is wrong.
    - munificent 5y ago
      Let's say my_app uses package foo which uses package bar. It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it. In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
      - geodel 5y ago
        You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running. It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.
typical182 5y ago
There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted. Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario. In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems. Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example): https://research.swtch.com/npm-colors https://research.swtch.com/npm-colors That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here. All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)
- goodoldneon 5y ago
  Wouldn't build size increase a lot if transitive dependencies were pinned to direct dependency lockfiles? Like if library A says "use version 1.0.0 of library X" and library B says "use version 1.0.1 of library X", then you'd likely end up bundling duplicate code in your build. Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
  - typical182 5y ago
    FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary. That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is: https://encore.dev/guide/go.mod https://encore.dev/guide/go.mod That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.
deleted 5y ago
[deleted]
infogulch 5y ago
The author is saying that Go provides the same guarantees with just a package list in the go.mod file that other package managers need both a package list and lock file to solve. go.sum is essentially a distributed / community maintained transparency log of published versions of packages.
- munificent 5y ago
  Maybe I'm just not familiar with it enough but I don't see how merging a package manifest and lockfile into a single file is a net win. This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it. Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
  - typical182 5y ago
    FWIW, there are two machine-formatted sections of go.mod -- the first for direct dependencies, the second section for indirect dependencies. (That's as of Go 1.17. Previously, that information was communicated via machine-generated comments in a single section).
    - munificent 5y ago
      That seems reasonable. I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
      - arccy 5y ago
        but if you ever bump up the version of an indirect dependency (maybe to pick up a bugfix earlier), is this now a direct or indirect dependency?
jchw 5y ago
It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json. Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version. This avoids the node-ipc issue entirely, at least until you update the go.mod.
- lamontcg 5y ago
  How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?
  - jchw 5y ago
    As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
    - lamontcg 5y ago
      Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5? Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints? Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
      - coder543 5y ago
amtamt 5y ago
Perhaps "A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies." And "If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules." Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified. I am not sure if all other package managers include checksum with dependency version.
- gmfawcett 5y ago
  > the go command may verify the hash If we're talking about reproducible builds, the word "may" seems concerning here?
  - typical182 5y ago
    I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model: https://en.wikipedia.org/wiki/Trust_on_first_use https://en.wikipedia.org/wiki/Trust_on_first_use More on sum.golang.org: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md https://go.googlesource.com/proposal/+/master/design/25530-s...
    - gmfawcett 5y ago
      Thank you for the clarification!
mirekrusin 5y ago
Npm lockfile will refer to same package because you can't republish npm with the same tag (there is also content hash in lockfile). Referring to version tag in git from go's mod can't guarantee that because you can overwrite tag in git. Am I wrong?
- coder543 5y ago
  Yes. The go.sum file that sits alongside go.mod keeps track of the hashes so that no modification like that can be made, and dependency fetches actually transparently go through a module proxy/mirror that keeps those same hashes as well, and it will prevent you from getting an altered version of a known module even if you’re starting a new project and don’t have a sum file yet. Versions can’t be republished.
  - mirekrusin 5y ago
    Thanks for clarification, indeed I can see go.sum being checked in on few go package repos I've checked, nice.

klodolph 5y ago

Discussing "what is a lockfile" is a bit of a headache because different languages have different files which do different things. Generally speaking, there's some file which specifies the dependency versions and some file with cryptographic checksums of the all transitive dependencies. In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock. Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.) Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC. If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.

munificent 5y ago
> any new, poisoned version of libB Conversely, you will get any old security-buggy version of libB instead. Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer. I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
- philosopher1234 5y ago
  Go just expects you to manually trigger the updates. Thats all. It still is in favor of updating to take security fixes, so i think your argument is wrong.
  - munificent 5y ago
    Let's say my_app uses package foo which uses package bar. It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it. In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
    - geodel 5y ago
      You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running. It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.

typical182 5y ago

There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted. Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario. In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems. Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example): https://research.swtch.com/npm-colors https://research.swtch.com/npm-colors That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here. All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)

goodoldneon 5y ago
Wouldn't build size increase a lot if transitive dependencies were pinned to direct dependency lockfiles? Like if library A says "use version 1.0.0 of library X" and library B says "use version 1.0.1 of library X", then you'd likely end up bundling duplicate code in your build. Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
- typical182 5y ago
  FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary. That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is: https://encore.dev/guide/go.mod https://encore.dev/guide/go.mod That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.

deleted 5y ago

[deleted]

infogulch 5y ago

The author is saying that Go provides the same guarantees with just a package list in the go.mod file that other package managers need both a package list and lock file to solve. go.sum is essentially a distributed / community maintained transparency log of published versions of packages.

munificent 5y ago
Maybe I'm just not familiar with it enough but I don't see how merging a package manifest and lockfile into a single file is a net win. This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it. Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
- typical182 5y ago
  FWIW, there are two machine-formatted sections of go.mod -- the first for direct dependencies, the second section for indirect dependencies. (That's as of Go 1.17. Previously, that information was communicated via machine-generated comments in a single section).
  - munificent 5y ago
    That seems reasonable. I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
    - arccy 5y ago
      but if you ever bump up the version of an indirect dependency (maybe to pick up a bugfix earlier), is this now a direct or indirect dependency?

jchw 5y ago

It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json. Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version. This avoids the node-ipc issue entirely, at least until you update the go.mod.

lamontcg 5y ago
How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?
- jchw 5y ago
  As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
  - lamontcg 5y ago
    Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5? Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints? Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
    - coder543 5y ago

amtamt 5y ago

Perhaps "A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies." And "If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules." Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified. I am not sure if all other package managers include checksum with dependency version.

gmfawcett 5y ago
> the go command may verify the hash If we're talking about reproducible builds, the word "may" seems concerning here?
- typical182 5y ago
  I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model: https://en.wikipedia.org/wiki/Trust_on_first_use https://en.wikipedia.org/wiki/Trust_on_first_use More on sum.golang.org: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md https://go.googlesource.com/proposal/+/master/design/25530-s...
  - gmfawcett 5y ago
    Thank you for the clarification!

mirekrusin 5y ago

Npm lockfile will refer to same package because you can't republish npm with the same tag (there is also content hash in lockfile). Referring to version tag in git from go's mod can't guarantee that because you can overwrite tag in git. Am I wrong?

coder543 5y ago
Yes. The go.sum file that sits alongside go.mod keeps track of the hashes so that no modification like that can be made, and dependency fetches actually transparently go through a module proxy/mirror that keeps those same hashes as well, and it will prevent you from getting an altered version of a known module even if you’re starting a new project and don’t have a sum file yet. Versions can’t be republished.
- mirekrusin 5y ago
  Thanks for clarification, indeed I can see go.sum being checked in on few go package repos I've checked, nice.