7 ms·

Go vgo: Semantic Versioning and Human Error

131 points by aleksi 8y ago

49bc 8y ago
> The issue Sam called out about handling the case where someone isn't following the spec. This can be on purpose or by accident. It's not uncommon to find it. This is a common occurance. So common - in fact - that I think it’s almost always better to either pin everything or nothing and hope the unit test will find any breaking behavior.
- mfer 8y ago
  This is a slightly tangential issue to pinning. With MVS and vgo you cannot set a maximum version. When the resolver walks the tree it doesn't know when a version is too new and could break things. Even if it pins it could pin an incompatible version. Maximum version information is not communicated. Note, I'm the posts author.
  - kardianos 8y ago
    Matt: > This is a slightly tangential issue to pinning. With MVS and vgo you cannot set a maximum version. When the resolver walks the tree it doesn't know when a version is too new and could break things. Even if it pins it could pin an incompatible version. This just isn't true. The number you put in the go.mod file can't encode the max version: true. But it won't change underneath you so when you run "vgo list -m" and determine the solved version, it won't change from that.
    - mfer 8y ago
      go.mod isn't a lock file. It doesn't contain the entire dependency tree the way other lock files do unless you choose to track, at the top level, the entire dependency tree yourself. There aren't tools to prune or update that for changes in the underlying transitive dependencies, either. It's not a lock and from what I gather from Russ not intended to be. The solved version doesn't mean it's the right or even a compatible version. This issue here is no maximum version meaning it could solve for an incompatible version due to not having all the information about the complete tree.
      - kardianos 8y ago
        go.mod paired with MVS effectively locks versions into place. It must be computed, but it effectively locks all versions into place for reproducible builds. > It's not a lock and from what I gather from Russ not intended to be. go.mod files lock versions into place. If you disagree read the code / design documents. I find it dishonest to say something technically true "It's not a lock [file]" and yet not be true, as it together locks versions in place.
        mappu 8y ago
        > it effectively locks all versions into place for reproducible builds Assuming git tags don't move.
        andrewchambers 8y ago
        Modverify is optional and detects that.
    - jitl 8y ago
      You're right, MVS is deterministic without a lockfile, which is what you're asserting it solves. The issue it does have is one where a transitive dependency is known bad by one of your immediate dependencies, but there's no way to declare that to the resolver. Here's a concrete case: 1. I depend on cool-framework, min 1.5.0 2. I depend on boring-library, min 1.0.0 3. cool-framework depends on boring-library, min 1.0.0 4. cool-framework receives bug reports that boring-library 1.5.0 breaks cool-framework. 5. I upgrade boring-library to min 1.5.0. Everything builds and tests okay, but I get breakage in my staging env. There wasn't non-deterministic package install behavior, but I still ended up with breakage that could have been prevented after (4) in a system that allows library dependencies to articulate more complex version constraints than "min version". In Rubygems or Python, with or without a lockfile, cool-framework could update its dependency on boring-library to specify "min 1.0.0, less-than 1.5.0" until the breakage is fixed.
      - infogulch 8y ago
        Ok so what happens when boring-library v1.5.1 is released that fixes the breakage, right after the maintainer of cool-framework goes on vacation? Now you have the opposite problem: you can't update to a perfectly compatible version because one of your dependencies can enforce arbitrarily complex constraints on which libraries you can include. Now the problem isn't a simple "oops gotta revert the upgrade until later" which either library can fix, now the issue is that one of your dependencies is blocking all upgrades for (now) no reason. There are problems either way. I think the other benefits that MVS enables is worth choosing one of these problems over the other. Edit: As mentioned upthread, vgo supports exclusions for the top-level module. This doesn't directly help here currently, but what if exclusions in dependencies produced a warning of a possible incompatibility instead? I think that would solve the issue of you not knowing that an upgrade could break things.
        zenhack 8y ago
        I think a pretty good spot in the design space would be to respect specified incompatibilities with specific versions, but not ranges. Maybe even require a comment attached to the incompatibility constraint, which would explain the problem. Bugs happen, and users should have some tools for that situation, but if a library is repeatedly releasing one broken version after another, you probably just shouldn't use it.
- bennofs 8y ago
  Pinning does not solve this. You still need a solver to generate new pin versions or update the pinned versions. Or do you manually pin/update each of the dependencies (including transitive ones)?
  - reidrac 8y ago
    Yes: the transitive dependencies is the big problem here, and a hard one to solve. I can handle direct dependencies manually, more or less, because that's the code I'm using directly and I know what are my requirements. But then I'm mostly clueless regarding dependencies of my dependencies, and if those aren't pinned by the project that is using them directly, all sort of bad things can happen. And this is why I love Linux distributions so much: they've been solving this problem for a long time. Whenever I can, between using upstream or a slightly older version maintained by my distro, I choose the latter.
infogulch 8y ago
One possible 'solution' to this is to disallow publishing minor versions that break the api, basically compare all the public signatures to see if they change. E.g. adding a new field is ok, changing a field type is not, adding varargs parameter is ok, changing the order of parameters is not etc etc. The problem with this solution is that behavior is part of the api, but it is not encoded anywhere that can be compared automatically. That's exactly what happened in the example in the linked article. Two new fields were added (no problem, that's allowed), and the implementation of another field was changed. This change is either fine because it doesn't change the behavior (refactoring, optimization, fix a bug, etc) or not fine and it changes behavior that breaks existing code, which is what happened in the example. This behavior was not encoded anywhere, so an automatic publishing gate wouldn't have caught this.
- weberc2 8y ago
  It’s not just public signatures, its also things like types—even private types. For example, if public struct type Foo has a private member of private struct type bar, and private type bar is changed to add a field, this changes the public interface because the memory layout of the type has changed.
  - ninkendo 8y ago
    eh... does this matter though? Go uses static linkage, you have to recompile your code when you update third party libraries anyway.
    - kasey_junk 8y ago
      It can cause behavior changes. Especially around memory usage & gc.
    - pjmlp 8y ago
      That is not longer the truth for quite some time now. Yes, Go uses static linking by default, but it also supports dynamic linking and also has a plugin concept built on top of it.
- 8y ago
dsr_ 8y ago
Minimum-version-selection is exactly equivalent to giving an opaque cookie to the name of every version. "I need libfoo version ScreamingOrangutan" There's nothing to be done with a numeric string other than match it, so you might as well choose something memorable or with meaning to humans.
- infogulch 8y ago
  No? An MVS constraint like "1.2.0" will match any "1.X.Y" where X >= 2, and the one that's chosen is the first version that matches all the constraints. Lets say you've walked the dependency graph for a project and somewhere dependency A is requested 3 times: "v1.0.0", "v1.2.0", "v1.2.1" which each mean (respectively): anything at or after v1.0.0 anything at or after v1.2.0 anything at or after v1.2.1 The minimum version that satisfies all of those constraints is simply the last listed version, v1.2.1, so that version is selected. Note, you can manually choose a later version by asking for it directly, e.g. add "A v1.2.10" directly to the dependencies of your root project and it will be chosen. Or add a new dependency that requests "A v1.3.0" or update a different dependency that requests "A v1.3.2" and the last one will be chosen. I'm surprised there is still so much confusion about what MVS actually does. The actual algorithm is just a sort function. I blame the name, it doesn't describe what it does very well.
  - dsr_ 8y ago
    This is perfectly reasonable -- and not what was described to me when I asked about it, which also fit the name better than this does. That is a terrible name.
- kardianos 8y ago
  Nope. MVS is designed so that each version can be explicitly ordered within a major version. Non versioned commits are assigned to the "v0" major version and compared by date of commit I believe.
mfer 8y ago
I'm amused... While this was on the homepage or hacker news, vgo was marked as accepted. It's the timing I find amusing. https://github.com/golang/go/issues/24301#issuecomment-390766926 https://github.com/golang/go/issues/24301#issuecomment-39076...
- kardianos 8y ago
  Yes, it is amusing that this objection to the vgo proposal are only now coming forward when it was proposed 2018-03-20, 2 months ago, and the design document a month before then.
  - sagichmal 8y ago
    Russ and the core team ignored the dependency management issue for years, and only responded to the community-led `dep` initiative after a year and a half. The `vgo` proposal was something like a hundred pages of dense technical details, and Sam has a full-time job. I think the sense of urgency you're implying is both artificial and unfair. It's been years already; we can wait another month or two for the details to shake out.
    - nemo1618 8y ago
      >The `vgo` proposal was something like a hundred pages of dense technical details Which document are you referring to? The one linked in https://github.com/golang/go/issues/24301 https://github.com/golang/go/issues/24301 is only about 600 lines: https://github.com/golang/proposal/blob/master/design/24301-versioned-go.md https://github.com/golang/proposal/blob/master/design/24301-...
    - geodel 8y ago
      > The `vgo` proposal was something like a hundred pages of dense technical details, and Sam has a full-time job Does that mean 2-3 months are not enough to read 100 pages document and respond specially from people who are most concerned about package management? Or that Sam should be the only one to respond to this?
rwcarlsen 8y ago
For the uninitiated, vgo has a way to blacklist/exclude certain versions of dependency modules, but it only works for the top-level module being built. Exclusions from deeper dependencies are not honored in order to avoid an np-complete satisfiability problem (https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs).
- mfer 8y ago
  Two quick thoughts... 1. By not automatically hoisting that information there is a requirement for the developer to know those details for all transitive dependencies and handle those issues themselves. A task that was automated under dep (and in other languages) is now a manual task in vgo. I do not look forward to doing that when I import Kubernetes into a project as a dependency. 2. Writing a solver that can work over a rather large codebase (e.g., kubernetes) in a timely manner (as fast as those for other languages) can and has been done. Why are we trying to avoid satisfaction problems, whom do they benefit, and why?
- infogulch 8y ago
  Dependencies' exclusions are not honored, but what if an exclusion produces a warning or notice if MVS happens to select that exclusion? That would help solve the problem of you not knowing that a later version inadvertently breaks a dependency without taking away your (as the top-level module maintainer) choice.
orivej 8y ago
vgo was explicitly designed to balance out two needs: (1) the need to use known good versions of the dependencies, and (2) the need not to burden dependency consumers with meaningless constraints (especially with an upper limit on the version). This article shows* why the first need is important, but it does not give vgo the credit for satisfying it with its minimal version selection (and, indeed, for providing a more stable and hence reliable solution than maximum version selection), and it misses the value of the second need. In my experience, the upper limit on the minor version is most often arbitrary, and in some cases when it is not, a future minor version reverts the mistakenly introduced incompatibility. Therefore vgo approach has unique advantages over other version selection methods, and it should not be discounted for the lack of a feature necessary to provide them. * The article says that "Prior to 1.4.0 there was one function of MaxMsgSize" which "had previously set the size on both send and receive" but it does not substantiate this claim, and it may be false since go-grpc 1.3.0 documents that "MaxMsgSize returns a ServerOption to set the max message size in bytes for inbound mesages" https://github.com/grpc/grpc-go/blob/v1.3.0/server.go#L166 https://github.com/grpc/grpc-go/blob/v1.3.0/server.go#L166, and it has not changed in go-grpc 1.12.0 https://github.com/grpc/grpc-go/blob/v1.12.0/server.go#L228 https://github.com/grpc/grpc-go/blob/v1.12.0/server.go#L228 which strongly suggests that this is not a bug.
- mfer 8y ago
  > it does not give vgo the credit for satisfying it with its minimal version selection (and, indeed, for providing a more stable and hence reliable solution than maximum version selection) I did not argue that because it doesn't do that. In an effort to not spoil things, because Sam Boyer has done a lot of work on this and wants to write about it, I won't say to much. MVS does use a maximum. It's the major version number in SemVer. It's implicit. You can't override it when dealing with transitive dependencies. For it to always be a safe value people have to always follow SemVer. Unfortunately, they don't. In a perfect world this would work. Unfortunately, people are fallible and we need a system that works in light of that.
  - orivej 8y ago
    vgo trades some safety (by not supporting upper bounds) for some utility (by not artificially limiting the lifetime of a released library). Yet package managers that do support upper bounds do not guarantee safety, because libraries may not specify upper bounds or they may specify too broad bounds. This is a trade-off, and I have not seen anything convincing about why vgo position on this trade-off is unreasonable. Can you give an example where vgo prevents use of a library where another approach does not? The main difference in the expressive power between vgo and traditional approaches is that the latter can restrict your use of libraries together more. vgo does not need a perfect world: it is practical in the imperfect one. > MVS does use a maximum. It's the major version number in SemVer. So, we can not force a library that wants dependency v3 to use dependency v2 (and vice versa), even if the author of the library knows that it works with either v2 or v3. This is a loss of vgo. On the other hand, if another library can only work with v2, and yet another can only work with v3, vgo allows the use of both in the same application. This looks like an acceptable win for the price of that loss.
tyrankh 8y ago
Preface: vgo _specifically_ calls out the fact that maintainers of libraries have to be backwards-compatible within a major version, and that the onus is on users to put their trust into libraries not to break them appropriately: > Modules are assumed to follow the import compatibility rule—packages in any newer version should work as well as older ones—so a dependency requirement gives only a minimum version, never a maximum version or a list of incompatible later versions. (https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs) Back to the article - it seems predicated on this scenario: > “Our project depends on X@v1.5.0 right now, but it doesn’t work with X@v1.7.0 or newer. We want to be good citizens and adapt, but we just don’t have the bandwidth right now.” If your deps + your transitive deps for some package are: - 1.5.0 (you) - 1.5.1 (some transitive dep) - 1.4.7 (some transitive dep) vgo will choose 1.5.1. However, if your deps for some package are are: - 1.5.0 (you) - 1.5.1 (some transitive dep) - 1.7.3 (some transitive dep) vgo will choose 1.7.3 and presumedly your app will break. In other dep managers, you might specify <1.7.0. How would this work? Grab two versions of the package (1.5.1 and 1.7.3), rewrite the import paths of the stuff that requires 1.7.3, and kind of opaquely have two version of the same thing? Or perhaps modify the way the import "xyz" works to be more opaque to solve this problem somehow? There's no nice solution to this. This seems a fairly reasonable tradeoff; on the upside is a _very_ fast, very simple, and very predictable dependency manager. On the downside is that I have to really think about which libraries I trust not to break me instead of relying on my tool to specify ranges and the like. Generally though, the ask from vgo is that folks care about backwards compatibility and think about trust, rather than covering up the issue. It's not going to be great for everyone, but I like the straight forwardness of it.
- bennofs 8y ago
  Other dep managers could at least tell you about the conflict. Maybe that's what vgo needs, a way to specify a conflict bound so that any downstream user is notified if the depgraph has conflicts (ofc conflict bounds would only apply after del resolution as simple checks otherwise it wouldn't be MVS)?
  - tyrankh 8y ago
    > Maybe that's what vgo needs, a way to specify a conflict bound This is a valid idea; you should make a proposal or issue for it. Alternatively, it seems like something that would be easy to add separately - I personally like the idea of vgo as more of a simple, unix-like building block that you'd build on top of. Adding a small utility CLI checks incompatibilities with `vgo list` + some incompatibilities.txt file would be pretty easy.
tschellenbach 8y ago
Super confusing that they picked a name that's so similar to VG (the tool to manage your go workspaces, similar to python's virtualenv): https://github.com/getstream/vg https://github.com/getstream/vg
- mseepgood 8y ago
  The name is not important, it's temporary. It's going to become the regular "go" tool in one of the next releases.
tschellenbach 8y ago
1. You should always fix your versions so things don't automatically upgrade 2. A good test suite will catch when things break so you don't find out in production. 3. Combining the above 2 it becomes easy to occasionally see which packages changed and upgrade to more recent versions. 4. After trying out a ton of dependency management tools (python, ruby, node, go, java) I think Yarn is the nicest solution i've worked with so far.
- TheDong 8y ago
  > don't automatically upgrade You've missed the point; code neither automatically upgrades with MVS or with a lockfile. In both cases, it's a conscious decision to do so. Helm likely had to update its dependencies (e.g. k8s dependency for k8s version compatibility) to solve a business problem, which makes it difficult to trivially downgrade that dependency (and thus break the business) > A good test suite will catch when things break so you don't find out in production helm has a good test suite; it wasn't good enough. A test suite is rarely, if ever, "good enough" to catch all bugs. That being the case, we must design in such a way that when a bug does occur, it can be easily resolved. vgo reduces how many knobs we can twiddle to resolve a bug. > After trying out a ton of dependency management tools (python, ruby, node, go, java) None of those are dependency management tools. python has easy_install and two versions of pip. ruby has gem, bundler, and the mess that predated that. node has npm and yarn. go has dep, glide, gb, go get, and dozens of others. Java has maven, gradle, and others. > I think Yarn is the nicest solution i've worked with so far Totally irrelevant information without any additional data on that conclusion.
jrs95 8y ago
MVS sucks, except maybe to protect some users of libraries from their own stupidity. I'd rather not have to deal with that. The way Cargo and other tools do it works perfectly fine and is well integrated with SemVer. Default behavior is to upgrade to the latest non-major release, but it's easy to override for libraries that are more volatile or less trusted.
dilap 8y ago
So what's happened here is there's a bug in in gRPC 1.8. What the OP wants is a way to say, "hey, our software is broken with this buggy version of gRPC," as part of the dependency requirements. I can see some value in that, but it also feels like a case where just documenting it in the README would not be super-unreasonable. As a practical matter, any time a dependency A says it wants some other dependency B at version V, if you are using B at a version U>V, you are running in a untested configuration, and may encounter problems. (But! this will only happen if you are specifically requesting U>V, or if you have some other requirement A2 requesting B@U. In the latter case, again, you are running a new, untested combination of software, and there might be problems.) An alternative to this would just be allowing multiple copies of B at the same major revision, B.U and B.V. But due to Go's use of package-level global variables, that is just as likely, if not more so, to cause bugs, since not all packages will see the same set of shared variables. It does seem like extending the vgo to allow != version requirements might be reasonable. Not full-on <=, which breaks builds or leaves them unsat forever, but just a != to say "hey this version is buggy and we know we don't work with it."
- RVuRnvbM2e 8y ago
  > So what's happened here is there's a bug in in gRPC 1.8. Yeah I don't understand how this is vgo's problem. In the follup post it shows that they are importing the buggy gRPC version directly: https://codeengineered.com/blog/2018/golang-vgo-broken-dep-tree/ https://codeengineered.com/blog/2018/golang-vgo-broken-dep-t... So doesn't that mean that they can just file a bug report upstream to get the regression fixed in 1.8.1, and exclude 1.8.0 in the meantime?
  - dolmen 8y ago
    I agree. I don't see a vgo issue. The behavior in the example was broken by a release, but could have as well been restored by a patch release as restoring behaviour would not have broken the API. See https://news.ycombinator.com/item?id=17123583 https://news.ycombinator.com/item?id=17123583
- mzl 8y ago
  For vgo there is an explicit goal is to avoid NP-completeness for some reason. Systems of not-equals are NP-hard (trivially models 3-colouring), so allowing not-equals constraints on package versions is a non-starter if the stated goals of vgo are to be maintained. Personally I think that some kind !=-requirements are needed for package managers just for this case, and thus the choice of dependency resolution approach vgo has is wrong IMHO.

131 points by aleksi 8y ago

49bc 8y ago

> The issue Sam called out about handling the case where someone isn't following the spec. This can be on purpose or by accident. It's not uncommon to find it. This is a common occurance. So common - in fact - that I think it’s almost always better to either pin everything or nothing and hope the unit test will find any breaking behavior.

mfer 8y ago
This is a slightly tangential issue to pinning. With MVS and vgo you cannot set a maximum version. When the resolver walks the tree it doesn't know when a version is too new and could break things. Even if it pins it could pin an incompatible version. Maximum version information is not communicated. Note, I'm the posts author.
- kardianos 8y ago
  Matt: > This is a slightly tangential issue to pinning. With MVS and vgo you cannot set a maximum version. When the resolver walks the tree it doesn't know when a version is too new and could break things. Even if it pins it could pin an incompatible version. This just isn't true. The number you put in the go.mod file can't encode the max version: true. But it won't change underneath you so when you run "vgo list -m" and determine the solved version, it won't change from that.
  - mfer 8y ago
    go.mod isn't a lock file. It doesn't contain the entire dependency tree the way other lock files do unless you choose to track, at the top level, the entire dependency tree yourself. There aren't tools to prune or update that for changes in the underlying transitive dependencies, either. It's not a lock and from what I gather from Russ not intended to be. The solved version doesn't mean it's the right or even a compatible version. This issue here is no maximum version meaning it could solve for an incompatible version due to not having all the information about the complete tree.
    - kardianos 8y ago
      go.mod paired with MVS effectively locks versions into place. It must be computed, but it effectively locks all versions into place for reproducible builds. > It's not a lock and from what I gather from Russ not intended to be. go.mod files lock versions into place. If you disagree read the code / design documents. I find it dishonest to say something technically true "It's not a lock [file]" and yet not be true, as it together locks versions in place.
      - mappu 8y ago
        > it effectively locks all versions into place for reproducible builds Assuming git tags don't move.
        andrewchambers 8y ago
        Modverify is optional and detects that.
  - jitl 8y ago
    You're right, MVS is deterministic without a lockfile, which is what you're asserting it solves. The issue it does have is one where a transitive dependency is known bad by one of your immediate dependencies, but there's no way to declare that to the resolver. Here's a concrete case: 1. I depend on cool-framework, min 1.5.0 2. I depend on boring-library, min 1.0.0 3. cool-framework depends on boring-library, min 1.0.0 4. cool-framework receives bug reports that boring-library 1.5.0 breaks cool-framework. 5. I upgrade boring-library to min 1.5.0. Everything builds and tests okay, but I get breakage in my staging env. There wasn't non-deterministic package install behavior, but I still ended up with breakage that could have been prevented after (4) in a system that allows library dependencies to articulate more complex version constraints than "min version". In Rubygems or Python, with or without a lockfile, cool-framework could update its dependency on boring-library to specify "min 1.0.0, less-than 1.5.0" until the breakage is fixed.
    - infogulch 8y ago
      Ok so what happens when boring-library v1.5.1 is released that fixes the breakage, right after the maintainer of cool-framework goes on vacation? Now you have the opposite problem: you can't update to a perfectly compatible version because one of your dependencies can enforce arbitrarily complex constraints on which libraries you can include. Now the problem isn't a simple "oops gotta revert the upgrade until later" which either library can fix, now the issue is that one of your dependencies is blocking all upgrades for (now) no reason. There are problems either way. I think the other benefits that MVS enables is worth choosing one of these problems over the other. Edit: As mentioned upthread, vgo supports exclusions for the top-level module. This doesn't directly help here currently, but what if exclusions in dependencies produced a warning of a possible incompatibility instead? I think that would solve the issue of you not knowing that an upgrade could break things.
      - zenhack 8y ago
        I think a pretty good spot in the design space would be to respect specified incompatibilities with specific versions, but not ranges. Maybe even require a comment attached to the incompatibility constraint, which would explain the problem. Bugs happen, and users should have some tools for that situation, but if a library is repeatedly releasing one broken version after another, you probably just shouldn't use it.
bennofs 8y ago
Pinning does not solve this. You still need a solver to generate new pin versions or update the pinned versions. Or do you manually pin/update each of the dependencies (including transitive ones)?
- reidrac 8y ago
  Yes: the transitive dependencies is the big problem here, and a hard one to solve. I can handle direct dependencies manually, more or less, because that's the code I'm using directly and I know what are my requirements. But then I'm mostly clueless regarding dependencies of my dependencies, and if those aren't pinned by the project that is using them directly, all sort of bad things can happen. And this is why I love Linux distributions so much: they've been solving this problem for a long time. Whenever I can, between using upstream or a slightly older version maintained by my distro, I choose the latter.

infogulch 8y ago

One possible 'solution' to this is to disallow publishing minor versions that break the api, basically compare all the public signatures to see if they change. E.g. adding a new field is ok, changing a field type is not, adding varargs parameter is ok, changing the order of parameters is not etc etc. The problem with this solution is that behavior is part of the api, but it is not encoded anywhere that can be compared automatically. That's exactly what happened in the example in the linked article. Two new fields were added (no problem, that's allowed), and the implementation of another field was changed. This change is either fine because it doesn't change the behavior (refactoring, optimization, fix a bug, etc) or not fine and it changes behavior that breaks existing code, which is what happened in the example. This behavior was not encoded anywhere, so an automatic publishing gate wouldn't have caught this.

weberc2 8y ago
It’s not just public signatures, its also things like types—even private types. For example, if public struct type Foo has a private member of private struct type bar, and private type bar is changed to add a field, this changes the public interface because the memory layout of the type has changed.
- ninkendo 8y ago
  eh... does this matter though? Go uses static linkage, you have to recompile your code when you update third party libraries anyway.
  - kasey_junk 8y ago
    It can cause behavior changes. Especially around memory usage & gc.
  - pjmlp 8y ago
    That is not longer the truth for quite some time now. Yes, Go uses static linking by default, but it also supports dynamic linking and also has a plugin concept built on top of it.
8y ago

dsr_ 8y ago

Minimum-version-selection is exactly equivalent to giving an opaque cookie to the name of every version. "I need libfoo version ScreamingOrangutan" There's nothing to be done with a numeric string other than match it, so you might as well choose something memorable or with meaning to humans.

infogulch 8y ago
No? An MVS constraint like "1.2.0" will match any "1.X.Y" where X >= 2, and the one that's chosen is the first version that matches all the constraints. Lets say you've walked the dependency graph for a project and somewhere dependency A is requested 3 times: "v1.0.0", "v1.2.0", "v1.2.1" which each mean (respectively): anything at or after v1.0.0 anything at or after v1.2.0 anything at or after v1.2.1 The minimum version that satisfies all of those constraints is simply the last listed version, v1.2.1, so that version is selected. Note, you can manually choose a later version by asking for it directly, e.g. add "A v1.2.10" directly to the dependencies of your root project and it will be chosen. Or add a new dependency that requests "A v1.3.0" or update a different dependency that requests "A v1.3.2" and the last one will be chosen. I'm surprised there is still so much confusion about what MVS actually does. The actual algorithm is just a sort function. I blame the name, it doesn't describe what it does very well.
- dsr_ 8y ago
  This is perfectly reasonable -- and not what was described to me when I asked about it, which also fit the name better than this does. That is a terrible name.
kardianos 8y ago
Nope. MVS is designed so that each version can be explicitly ordered within a major version. Non versioned commits are assigned to the "v0" major version and compared by date of commit I believe.

mfer 8y ago

I'm amused... While this was on the homepage or hacker news, vgo was marked as accepted. It's the timing I find amusing. https://github.com/golang/go/issues/24301#issuecomment-390766926 https://github.com/golang/go/issues/24301#issuecomment-39076...

kardianos 8y ago
Yes, it is amusing that this objection to the vgo proposal are only now coming forward when it was proposed 2018-03-20, 2 months ago, and the design document a month before then.
- sagichmal 8y ago
  Russ and the core team ignored the dependency management issue for years, and only responded to the community-led `dep` initiative after a year and a half. The `vgo` proposal was something like a hundred pages of dense technical details, and Sam has a full-time job. I think the sense of urgency you're implying is both artificial and unfair. It's been years already; we can wait another month or two for the details to shake out.
  - nemo1618 8y ago
    >The `vgo` proposal was something like a hundred pages of dense technical details Which document are you referring to? The one linked in https://github.com/golang/go/issues/24301 https://github.com/golang/go/issues/24301 is only about 600 lines: https://github.com/golang/proposal/blob/master/design/24301-versioned-go.md https://github.com/golang/proposal/blob/master/design/24301-...
  - geodel 8y ago
    > The `vgo` proposal was something like a hundred pages of dense technical details, and Sam has a full-time job Does that mean 2-3 months are not enough to read 100 pages document and respond specially from people who are most concerned about package management? Or that Sam should be the only one to respond to this?

rwcarlsen 8y ago

For the uninitiated, vgo has a way to blacklist/exclude certain versions of dependency modules, but it only works for the top-level module being built. Exclusions from deeper dependencies are not honored in order to avoid an np-complete satisfiability problem (https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs).

mfer 8y ago
Two quick thoughts... 1. By not automatically hoisting that information there is a requirement for the developer to know those details for all transitive dependencies and handle those issues themselves. A task that was automated under dep (and in other languages) is now a manual task in vgo. I do not look forward to doing that when I import Kubernetes into a project as a dependency. 2. Writing a solver that can work over a rather large codebase (e.g., kubernetes) in a timely manner (as fast as those for other languages) can and has been done. Why are we trying to avoid satisfaction problems, whom do they benefit, and why?
infogulch 8y ago
Dependencies' exclusions are not honored, but what if an exclusion produces a warning or notice if MVS happens to select that exclusion? That would help solve the problem of you not knowing that a later version inadvertently breaks a dependency without taking away your (as the top-level module maintainer) choice.

orivej 8y ago

vgo was explicitly designed to balance out two needs: (1) the need to use known good versions of the dependencies, and (2) the need not to burden dependency consumers with meaningless constraints (especially with an upper limit on the version). This article shows* why the first need is important, but it does not give vgo the credit for satisfying it with its minimal version selection (and, indeed, for providing a more stable and hence reliable solution than maximum version selection), and it misses the value of the second need. In my experience, the upper limit on the minor version is most often arbitrary, and in some cases when it is not, a future minor version reverts the mistakenly introduced incompatibility. Therefore vgo approach has unique advantages over other version selection methods, and it should not be discounted for the lack of a feature necessary to provide them. * The article says that "Prior to 1.4.0 there was one function of MaxMsgSize" which "had previously set the size on both send and receive" but it does not substantiate this claim, and it may be false since go-grpc 1.3.0 documents that "MaxMsgSize returns a ServerOption to set the max message size in bytes for inbound mesages" https://github.com/grpc/grpc-go/blob/v1.3.0/server.go#L166 https://github.com/grpc/grpc-go/blob/v1.3.0/server.go#L166, and it has not changed in go-grpc 1.12.0 https://github.com/grpc/grpc-go/blob/v1.12.0/server.go#L228 https://github.com/grpc/grpc-go/blob/v1.12.0/server.go#L228 which strongly suggests that this is not a bug.

mfer 8y ago
> it does not give vgo the credit for satisfying it with its minimal version selection (and, indeed, for providing a more stable and hence reliable solution than maximum version selection) I did not argue that because it doesn't do that. In an effort to not spoil things, because Sam Boyer has done a lot of work on this and wants to write about it, I won't say to much. MVS does use a maximum. It's the major version number in SemVer. It's implicit. You can't override it when dealing with transitive dependencies. For it to always be a safe value people have to always follow SemVer. Unfortunately, they don't. In a perfect world this would work. Unfortunately, people are fallible and we need a system that works in light of that.
- orivej 8y ago
  vgo trades some safety (by not supporting upper bounds) for some utility (by not artificially limiting the lifetime of a released library). Yet package managers that do support upper bounds do not guarantee safety, because libraries may not specify upper bounds or they may specify too broad bounds. This is a trade-off, and I have not seen anything convincing about why vgo position on this trade-off is unreasonable. Can you give an example where vgo prevents use of a library where another approach does not? The main difference in the expressive power between vgo and traditional approaches is that the latter can restrict your use of libraries together more. vgo does not need a perfect world: it is practical in the imperfect one. > MVS does use a maximum. It's the major version number in SemVer. So, we can not force a library that wants dependency v3 to use dependency v2 (and vice versa), even if the author of the library knows that it works with either v2 or v3. This is a loss of vgo. On the other hand, if another library can only work with v2, and yet another can only work with v3, vgo allows the use of both in the same application. This looks like an acceptable win for the price of that loss.

tyrankh 8y ago

Preface: vgo _specifically_ calls out the fact that maintainers of libraries have to be backwards-compatible within a major version, and that the onus is on users to put their trust into libraries not to break them appropriately: > Modules are assumed to follow the import compatibility rule—packages in any newer version should work as well as older ones—so a dependency requirement gives only a minimum version, never a maximum version or a list of incompatible later versions. (https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs) Back to the article - it seems predicated on this scenario: > “Our project depends on X@v1.5.0 right now, but it doesn’t work with X@v1.7.0 or newer. We want to be good citizens and adapt, but we just don’t have the bandwidth right now.” If your deps + your transitive deps for some package are: - 1.5.0 (you) - 1.5.1 (some transitive dep) - 1.4.7 (some transitive dep) vgo will choose 1.5.1. However, if your deps for some package are are: - 1.5.0 (you) - 1.5.1 (some transitive dep) - 1.7.3 (some transitive dep) vgo will choose 1.7.3 and presumedly your app will break. In other dep managers, you might specify <1.7.0. How would this work? Grab two versions of the package (1.5.1 and 1.7.3), rewrite the import paths of the stuff that requires 1.7.3, and kind of opaquely have two version of the same thing? Or perhaps modify the way the import "xyz" works to be more opaque to solve this problem somehow? There's no nice solution to this. This seems a fairly reasonable tradeoff; on the upside is a _very_ fast, very simple, and very predictable dependency manager. On the downside is that I have to really think about which libraries I trust not to break me instead of relying on my tool to specify ranges and the like. Generally though, the ask from vgo is that folks care about backwards compatibility and think about trust, rather than covering up the issue. It's not going to be great for everyone, but I like the straight forwardness of it.

bennofs 8y ago
Other dep managers could at least tell you about the conflict. Maybe that's what vgo needs, a way to specify a conflict bound so that any downstream user is notified if the depgraph has conflicts (ofc conflict bounds would only apply after del resolution as simple checks otherwise it wouldn't be MVS)?
- tyrankh 8y ago
  > Maybe that's what vgo needs, a way to specify a conflict bound This is a valid idea; you should make a proposal or issue for it. Alternatively, it seems like something that would be easy to add separately - I personally like the idea of vgo as more of a simple, unix-like building block that you'd build on top of. Adding a small utility CLI checks incompatibilities with `vgo list` + some incompatibilities.txt file would be pretty easy.

tschellenbach 8y ago

Super confusing that they picked a name that's so similar to VG (the tool to manage your go workspaces, similar to python's virtualenv): https://github.com/getstream/vg https://github.com/getstream/vg

mseepgood 8y ago
The name is not important, it's temporary. It's going to become the regular "go" tool in one of the next releases.

tschellenbach 8y ago

1. You should always fix your versions so things don't automatically upgrade 2. A good test suite will catch when things break so you don't find out in production. 3. Combining the above 2 it becomes easy to occasionally see which packages changed and upgrade to more recent versions. 4. After trying out a ton of dependency management tools (python, ruby, node, go, java) I think Yarn is the nicest solution i've worked with so far.

TheDong 8y ago
> don't automatically upgrade You've missed the point; code neither automatically upgrades with MVS or with a lockfile. In both cases, it's a conscious decision to do so. Helm likely had to update its dependencies (e.g. k8s dependency for k8s version compatibility) to solve a business problem, which makes it difficult to trivially downgrade that dependency (and thus break the business) > A good test suite will catch when things break so you don't find out in production helm has a good test suite; it wasn't good enough. A test suite is rarely, if ever, "good enough" to catch all bugs. That being the case, we must design in such a way that when a bug does occur, it can be easily resolved. vgo reduces how many knobs we can twiddle to resolve a bug. > After trying out a ton of dependency management tools (python, ruby, node, go, java) None of those are dependency management tools. python has easy_install and two versions of pip. ruby has gem, bundler, and the mess that predated that. node has npm and yarn. go has dep, glide, gb, go get, and dozens of others. Java has maven, gradle, and others. > I think Yarn is the nicest solution i've worked with so far Totally irrelevant information without any additional data on that conclusion.

jrs95 8y ago

MVS sucks, except maybe to protect some users of libraries from their own stupidity. I'd rather not have to deal with that. The way Cargo and other tools do it works perfectly fine and is well integrated with SemVer. Default behavior is to upgrade to the latest non-major release, but it's easy to override for libraries that are more volatile or less trusted.

dilap 8y ago

So what's happened here is there's a bug in in gRPC 1.8. What the OP wants is a way to say, "hey, our software is broken with this buggy version of gRPC," as part of the dependency requirements. I can see some value in that, but it also feels like a case where just documenting it in the README would not be super-unreasonable. As a practical matter, any time a dependency A says it wants some other dependency B at version V, if you are using B at a version U>V, you are running in a untested configuration, and may encounter problems. (But! this will only happen if you are specifically requesting U>V, or if you have some other requirement A2 requesting B@U. In the latter case, again, you are running a new, untested combination of software, and there might be problems.) An alternative to this would just be allowing multiple copies of B at the same major revision, B.U and B.V. But due to Go's use of package-level global variables, that is just as likely, if not more so, to cause bugs, since not all packages will see the same set of shared variables. It does seem like extending the vgo to allow != version requirements might be reasonable. Not full-on <=, which breaks builds or leaves them unsat forever, but just a != to say "hey this version is buggy and we know we don't work with it."

RVuRnvbM2e 8y ago
> So what's happened here is there's a bug in in gRPC 1.8. Yeah I don't understand how this is vgo's problem. In the follup post it shows that they are importing the buggy gRPC version directly: https://codeengineered.com/blog/2018/golang-vgo-broken-dep-tree/ https://codeengineered.com/blog/2018/golang-vgo-broken-dep-t... So doesn't that mean that they can just file a bug report upstream to get the regression fixed in 1.8.1, and exclude 1.8.0 in the meantime?
- dolmen 8y ago
  I agree. I don't see a vgo issue. The behavior in the example was broken by a release, but could have as well been restored by a patch release as restoring behaviour would not have broken the API. See https://news.ycombinator.com/item?id=17123583 https://news.ycombinator.com/item?id=17123583
mzl 8y ago
For vgo there is an explicit goal is to avoid NP-completeness for some reason. Systems of not-equals are NP-hard (trivially models 3-colouring), so allowing not-equals constraints on package versions is a non-starter if the stated goals of vgo are to be maintained. Personally I think that some kind !=-requirements are needed for package managers just for this case, and thus the choice of dependency resolution approach vgo has is wrong IMHO.