10 ms·

An Analysis of vgo

184 points by bpineau 8y ago

pcx 8y ago
I think the current approach of announcing the death of `dep` before even having a stable alternative (`vgo`) is deeply flawed. With `dep` the ecosystem finally had something most of the developers agreed on, despite whatever shortcomings it had. The model was working. They could have adopted dep and improved on it. The packaging story for Golang has gone from worse to OK to worse again. After all this progress, we still don't have a stable alternative and no clear date as to when it might be ready or even worse, the community is still not sure vgo will be the final one. Golang should be about pragmatic, practical decisions. The packaging situation is just the opposite.
- lobster_johnson 8y ago
  I don't disagree — dep has been a huge improvement on Go's package management story. Until dep was stable, we were using Glide, which is extremely buggy. That said, if you ignore the doubts and arguments about the sanity of its dependency resolution algorithm, I think vgo's introduction of modules is its real contribution. Go packages have been problematic since the start since their design, naively, conflates a bunch of concepts that most developers keep separate: File location, file structure and source repository. That is, when you do 'import "github.com/foo/bar"', there's a whole bunch of conventions at play that dictate where it can be fetched from and where it should be in the file system. Aside from the fact that the $GOPATH convention is maddeningly annoying to many developers, the design has several issues, such as that an entire git repository has to be fetched in order to use a nested package, or that a package can have multiple import paths. But getting rid of $GOPATH is in itself a huge deal. I'm personally less concerned with the new MVS algorithm (though I personally prefer the more traditional lock file approach such as that used by dep, Cargo, Bundler, NPM, etc.), as long as I can get modules. While getting modules retrofitted into depwithout all the other stuff in the proposal would be lovely, I don't know if the module design is too dependent (heh) on the rest of vgo's versioning thesis to be separated out.
  - endymi0n 8y ago
    All of this feels like a classic case of "the perfect is the enemy of the good". Package and dependency management in Go is honestly a total mess today and with vgo challenging all current assumptions about how to generally approach packagement, it brings complete paralysis to the simple question of "how should I organize my dependencies right now". As much as I loathe JavaScript or Ruby these days for their language, using npm or bundler simply feels sane despite all of their warts. Yes, `left-pad`. Yes, they do have their issues. But seriously, they just work and all open source packages comply to their specifications. And they can and do evolve gradually, like yarn shows. It's just so funny how careful the Go community seems to tread about not breaking their language with all these Go 2 proposals and general resistance to new features — and then completely ignore this spirit to "move fast and break things" when it comes to package management.
    - pknopf 8y ago
      > using npm or bundler simply feels sane I feel the opposite. Lock files are awesome when your developing a package, but after you release it, you are at the mercy of all your dependencies in making sure that they don't break semvar. Each "npm install" will ignore any lock file your package was using during development. It would be awesome if I could distribute a package-json.lock file with my npm package and have yarn/npm use it when resolving packages on fresh installations. Yeah, this would mean that there may be duplicated libraries in your node_modules tree, each with a slightly different version, even if semver compat. However, people break semver to often anyway. I want to deploy a package to npm and be sure that it will never break, ever.
      - regularfry 8y ago
        Can't you do this right now by specifying exact dependency versions in package.json? Lock files should be for applications, not libraries.
        pknopf 8y ago
        Yes, but "npm install" will by default add a semver-compat package version. The vast majority of packages out there use what "npm install" gives you be default. Furthermore, even if I use exact versions in my package.json, that doesn't stop my referenced packages from internally referencing semver-compat versions. At the very least, you should reference exact versions at the top level. It is better than nothing.
- ngrilly 8y ago
  > They could have adopted dep and improved on it. Not really because: - The scope is different. dep is a package management tool. vgo is more than that. vgo intends to fully replace the go command. This is how we deprecate GOPATH. - The underlying principles are different enough to make it difficult to "improve" on dep. It looked easier to start from scratch. References: "Go += Package Versioning", Russ Cox, https://research.swtch.com/vgo-intro https://research.swtch.com/vgo-intro "FAQ: Why is the proposal not “use Dep”?", Russ Cox, https://github.com/golang/go/issues/24301#issuecomment-371228664 https://github.com/golang/go/issues/24301#issuecomment-37122...
  - sagichmal 8y ago
    dep was always written with the intention that it would be merged into or replace to go tool. And it's still not settled that the underlying principles of vgo are valid -- which this article begins to unpack.
- skywhopper 8y ago
  Yes! Speaking as a sysadmin who writes Go tools from time to time along with a lot of Bash, Ruby, etc, I’m no expert in the land of Go. But in terms of designing my code to be shareable and figuring out how to integrate it with other source code I deal with every day is a mess. dep had seemed a very promising direction, and so I’m really frustrated to see it’s been deprecated and we are still seemingly years out from a solution which may not even work well in the end. Quite honestly the situation is actively holding back Go, as it’s very hard to convince developers used to lock files and well defined packaging to build anything in a language with no official story on this, even after a decade. Go is rapidly losing the momentum it had for a while now that Rust is starting to become a more stable target and given Go’s inability to address package versioning.
  - Karrot_Kream 8y ago
    From what I read `dep` isn't deprecated, and is right now the recommended way to track dependencies. I used `dep` in a project I started a bit ago (around the time of the announcement of `vgo`) and it's been working out just great. That said I agree it's super frustrating, and the vgo announcement just had zero finesse. This isn't a language of experimenters hacking away, there are large production systems that run on solely Go code. Upending the main tool (`go` itself) and the versioning strategy is not to be taken lightly. > Go is rapidly losing the momentum it had for a while now that Rust is starting to become a more stable target and given Go’s inability to address package versioning. Here I disagree. Rust is fine for small utilities right now, but if you start building larger apps, developing with futures, tokio, and some of the other async libraries is a nightmare. The APIs are moving really quickly (`futures` hit 0.2, `tokio` is on 0.1 so I don't blame them really) and the ecosystem is still extremely immature. I'm excited for Rust, but it's not yet at the point that I want to build a complex application.
jstarks 8y ago
Could anyone distill this down for the casual reader? The author has a fairly laborious style and seems to be writing for an audience that is intimately familiar with the internals of dep and vgo.
- lobster_johnson 8y ago
  The article is about vgo, a proposal (with working prototype) to build package management into Go itself. vgo bundles several improvements, a major one being the introduction of "modules", which are versioned collections of packages. Modules, among other features, will let us decouple the import path from the file system structure and source repository, making the awkward and much-maligned $GOPATH finally obsolete. vgo also introduces a new dependency resolution algorithm, minimal version selection (MVS), which is different from traditional dependency resolution algorithms in that it will at any time choose the oldest allowed version that satisfy the minimum version constraints specified in the list of imported packages, where other systems will choose the newest. As MVS does not require (as with dep) a complex boolean SAT solver with potentially pathological, NP-complete cases, it is much simpler and faster to compute. However, it also has downsides. (Edited, thanks to pa7ch for correction.) The author likes pretty much everything about vgo except the MVS algorithm, which has been deeply controversial since the proposal was first published. The author goes into explanations of why he thinks MVS is a bad fit, but this is a complicated topic, so you really have read both the vgo proposal and this rebuttal to understand what it's all about. The article was written by Sam Boyer, one of the designers/authors of dep. While he has been courteous about it, there are obviously emotions at play; during its development, dep seemed to have the blessing of the official Go team, and this proposal came out of the blue and probably felt like an ambush.
  - pa7ch 8y ago
    Minor correction: MVS requires modules to specify the minimum version to use of a module. The algorithm will then select the newest among those based on the idea that semver is forward compatible among minor versions. The name minimal version selection throws people off. Other algorithms would select the newest release within a major version even if no module has ever tested it and it was released 5 seconds ago. When authors do make incompatible changes and MVS selects a broken dependency, as expected, there are manual escape hatches. More complicated NP complete algo's would have more of an automated answer here by allowing dependencies to have boolean constraints (greater then X, less then Y, not Z etc.) on version so that there is this collaborative summation of constraints to work around authors violating semver or bugs. To summerize poorly: It seems Sam Boyer regrets that dep didn't do what vgo does by default, but believes that a SAT solver could kinda do MVS for the happy path but still solve against the introduction of boolean constraints. Personally, I'd like to let vgo play out a bit, I'm not convinced boolean constraints are something I really want imposed on me by my dependencies. More info for those who are looking for more context: https://research.swtch.com/vgo https://research.swtch.com/vgo https://github.com/golang/go/issues/24301 https://github.com/golang/go/issues/24301 Whether you agree or disagree, its novel research in this field and excellent technical writing worth reading in its entirety over a cup of coffee.
pcwalton 8y ago
The basic upshot of all this is that vgo's attempt to avoid NP-completeness in the dependency resolution algorithm is solving a problem that doesn't matter in practice, and is simultaneously creating downsides that do matter in practice. This is consistent with what I've seen with systems like Cargo. If I had to make a list of top 10 issues I run into with Cargo, theoretical scalability of the core dependency resolution algorithm wouldn't make the cut.
- enqk 8y ago
  On the other hand, git became successful by being fast and by not solving certain of the hard theoretical problems that specialists were obsessed with solving. It's hard to know before hand what the healthy compromise might be.
  - lobster_johnson 8y ago
    git may have become successful, but that doesn't mean it's right. I'd argue that most developers still don't really understand git. I know seasoned, senior developers who struggle when git gets into an unfamiliar state, or who are completely mystified as to how it actually works. And don't forget that git spent years improving its initially horrific UI to make it easier and more palatable to people who aren't kernel developers. If you compare git to rival tools such as Mercurial and Fossil or even Darcs, it's pretty evident that git didn't really win on technical merit, user friendliness or indeed hardly any other metric. It was fast, and it had the dubious advantage of being better than a lot of crappy alternatives at the time, and of being the product of a famous developer. But if technical merit was how we chose software, we might all be using Mercurial today. It's quite possible we'd be better off, too. git's punting of "certain of the hard theoretical problems" lead to major shortcomings in its design. For example, git does not track branching histories. Once you merge your changes back into a branch and delete the origin branch, the path that your commits took are lost. Another example: Since git's data model works entirely on repository snapshots (there are no "patches" or logical operations, only snapshots), it doesn't know about file renames. Some commands, like log and diff, do understand how to detect renames (but not out of the box!), but it's fundamentally a fuzzy-matching operation that can only work perfectly if you separate your rename commits from modification commits, which is rarely feasible. Don't get me wrong, git has overall been a force for good, but I think it's a pretty terrible example in this particular case. In fact, I'm not sure I can think of many cases where the "worse is better" philosophy was ultimately a good idea. In every instance that comes to mind (MySQL, MongoDB, PHP), the fast-and-loose attitude ultimately came back to hurt everyone, and years were spent paying for those mistakes, plugging one hole at a time. The original decision by the Go team to build package management into the language ("go get") was made without really understanding the problem, and this decision has been haunting us pretty much since day one, with developers having to chase unofficial package management tools of the week in order to get their work done. I really want it to be done right, even if takes a bit longer. It's a problem worth solving well.
IshKebab 8y ago
This badly needs a TL;DR. If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words.
- TheDong 8y ago
  > If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words. That is a very lazy argument. It's perfectly reasonable to claim something is flawed for numerous complex reasons. It's equally reasonable to claim that a complex flaw may be difficult to explain concisely. Most importantly though, I don't think Sam is claiming that there's an obvious fatal flaw in vgo, but rather that it is not the ideal choice given the problem space. To make that more subtle argument, it is necessary to fully define the alternatives and problem space, which is what a lot of the words are attempting to do. The tl;dr would be along the lines of "vgo knowingly makes a set of tradeoffs which I think are worse than the tradeoffs that are possible with another hypothetical option, which I am also proposing"... but that tl;dr is not really a very useful one, and attempting to condense a more meaningful one will lose a bit too much nuance I think.
  - closeparen 8y ago
    The post could stand to have an introduction that lays out its arguments at a high level, as is hammered into students at every level of English class.
- geodel 8y ago
  I am afraid it is because lot of expectations were built that dep author would reveal some rather serious flaws with vgo. But since he has nothing explicit, many thousand words are required to cover that up.
red_admiral 8y ago
> If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. Problems can be NP-complete. Algorithms cannot - they can be exponential time (and if your choice is between exponential and double exponential, you pick the former!) In practice, and especially for SAT solving, as far as I understand the state of the art is that the general case is NP-hard but we have tools that work surprisingly well in those cases that reality tends to produce. In some more academic cases, we even have heuristic solvers that are not even guaranteed to terminate, but usually do so within a couple of seconds, so you can get away with the "beginner's solution to the halting problem".
- munificent 8y ago
  > In practice, and especially for SAT solving, as far as I understand the state of the art is that the general case is NP-hard but we have tools that work surprisingly well in those cases that reality tends to produce. You're correct. Modern version solvers need fairly sophisticated algorithms, but it's not rocket science. It's mostly a non-problem for most users of most package managers most of the time.
  - bennofs 8y ago
    The thing that worries me about this line of thinking is that as far as I know, we don't know much about why SAT solvers tend to work well in practice. So if I ever happen to hit a problem that the solver cannot solve, all we can say is "well bad luck". If the algorithm fails, I won't have any way to know why. This strikes me as a bad situation to be in.
rosshemsley 8y ago
Anyone have a bullet-pointed summary of the "crux" of what the author dislikes about MVS? I find MVS to be a very natural and simple solution to versioning - I feel like it's much closer to how people actually manage their dependencies in practice. After an admittedly low-effort skim of this article, most of the arguments seem to be that the author doesn't think it "feels right". Is there something more concrete hidden in here?
- cdoxsey 8y ago
  Its a long post, but I don't think he actually gives the reasons, rather he intends to write several additional posts that will.
  - ngrilly 8y ago
    Honestly, Russ Cox published his first post about vgo on February 20 (https://research.swtch.com/vgo https://research.swtch.com/vgo). It's been almost 3 months then. It's not fair to ask the community to wait any longer. If Sam Boyer has good reasons to oppose vgo, then he should publish them now.
    - sagichmal 8y ago
      Russ waited over a year and a half before engaging meaningfully with the community on package management, and waited another nine months before coming up with this greenfield proposal. There's no reason to rush things now.
      - ngrilly 8y ago
        > There's no reason to rush things now. I disagree. I want things to move forward as soon as possible. I've been waiting so long for a dependency management solution and for GOPATH deprecation! Russ came with his "greenfield" proposal because he was not convinced that dep could become the long term solution to dependency management in Go. Sam and the whole dep team have been working on package management for more than a year now. If they see important issues with vgo, I think they should be able to explain them succinctly in a post, at least to convince other gophers to hold on.
eknkc 8y ago
For an introduction to vgo and to see the other side of argument, watch the presentation from Russ Cox: https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg I was sceptical but now kind of sold on the idea after wathing the presentation.
nicpottier 8y ago
Though this is well written and I'm sure Sam is trying to set a tone of respect and appreciation of what is good in vgo so as to start on the right foot, he is also "burying the lead" in that he never gets to the point. I think Russ did an unusually good job of explaining the ideas around vgo, grounded by using real world examples. He set a high bar there, but hopefully Sam adopts some of the same methods of very concrete examples to explain his criticisms. As it stands this is an interesting piece of reading but is too high level and abstract in its criticism to be effective. Looking forward to future installments that hopefully that will make Sam's criticisms clear. From what I can tell one piece of criticism is the inability of vgo to define incompatibilities. That is that if you know these versions won't work, you can say never allow that version in your dependency graph. I'd love to hear more examples of this in the real world. Since vgo dependencies are basically always "pinned" this seems like that not big a deal to me, but maybe I'm missing something. Perhaps this is something that occurs with having two of your dependencies having different dependencies on a third package and one of those works only with dep3@v1.2 and the other only works with dep3@v1.3 or somesuch. Maybe there are ways around that or maybe that's not the actual problem (again, hope Sam explains more in the future). I think there is a really interesting element to MSV that will actually drive Go culture though (and I'm a big believer in the importance of the culture of a language) which I think may minimize the problems that Sam is talking about and for the betterment of the community. As others have noted, I hope that Sam comes out with the rest of this series quickly so that the go community can commit do a direction sooner rather than later and move on to actually building this new ecosystem.
- 4ad 8y ago
  > he is also "burying the lead" The expression is burying the lede, not lead.
  - nicpottier 8y ago
    Hah, thanks, learn something every day! https://www.merriam-webster.com/words-at-play/bury-the-lede-versus-lead https://www.merriam-webster.com/words-at-play/bury-the-lede-...
- 8y ago
abiox 8y ago
i think one fundamental problem is that for most ecosystems, versioning is almost entirely arbitrary, manual and dissociated from language semantics. the elm package manager seems to take a step in the right direction where changing a function signature will force a version bump if you want to publish the update. (iirc) are there other languages/ecosystems that do something interesting with versioning?
- steveklabnik 8y ago
  We're working on similar tooling to that in Rust https://github.com/rust-lang-nursery/rust-semverver https://github.com/rust-lang-nursery/rust-semverver

184 points by bpineau 8y ago

pcx 8y ago

I think the current approach of announcing the death of `dep` before even having a stable alternative (`vgo`) is deeply flawed. With `dep` the ecosystem finally had something most of the developers agreed on, despite whatever shortcomings it had. The model was working. They could have adopted dep and improved on it. The packaging story for Golang has gone from worse to OK to worse again. After all this progress, we still don't have a stable alternative and no clear date as to when it might be ready or even worse, the community is still not sure vgo will be the final one. Golang should be about pragmatic, practical decisions. The packaging situation is just the opposite.

lobster_johnson 8y ago
I don't disagree — dep has been a huge improvement on Go's package management story. Until dep was stable, we were using Glide, which is extremely buggy. That said, if you ignore the doubts and arguments about the sanity of its dependency resolution algorithm, I think vgo's introduction of modules is its real contribution. Go packages have been problematic since the start since their design, naively, conflates a bunch of concepts that most developers keep separate: File location, file structure and source repository. That is, when you do 'import "github.com/foo/bar"', there's a whole bunch of conventions at play that dictate where it can be fetched from and where it should be in the file system. Aside from the fact that the $GOPATH convention is maddeningly annoying to many developers, the design has several issues, such as that an entire git repository has to be fetched in order to use a nested package, or that a package can have multiple import paths. But getting rid of $GOPATH is in itself a huge deal. I'm personally less concerned with the new MVS algorithm (though I personally prefer the more traditional lock file approach such as that used by dep, Cargo, Bundler, NPM, etc.), as long as I can get modules. While getting modules retrofitted into depwithout all the other stuff in the proposal would be lovely, I don't know if the module design is too dependent (heh) on the rest of vgo's versioning thesis to be separated out.
- endymi0n 8y ago
  All of this feels like a classic case of "the perfect is the enemy of the good". Package and dependency management in Go is honestly a total mess today and with vgo challenging all current assumptions about how to generally approach packagement, it brings complete paralysis to the simple question of "how should I organize my dependencies right now". As much as I loathe JavaScript or Ruby these days for their language, using npm or bundler simply feels sane despite all of their warts. Yes, `left-pad`. Yes, they do have their issues. But seriously, they just work and all open source packages comply to their specifications. And they can and do evolve gradually, like yarn shows. It's just so funny how careful the Go community seems to tread about not breaking their language with all these Go 2 proposals and general resistance to new features — and then completely ignore this spirit to "move fast and break things" when it comes to package management.
  - pknopf 8y ago
    > using npm or bundler simply feels sane I feel the opposite. Lock files are awesome when your developing a package, but after you release it, you are at the mercy of all your dependencies in making sure that they don't break semvar. Each "npm install" will ignore any lock file your package was using during development. It would be awesome if I could distribute a package-json.lock file with my npm package and have yarn/npm use it when resolving packages on fresh installations. Yeah, this would mean that there may be duplicated libraries in your node_modules tree, each with a slightly different version, even if semver compat. However, people break semver to often anyway. I want to deploy a package to npm and be sure that it will never break, ever.
    - regularfry 8y ago
      Can't you do this right now by specifying exact dependency versions in package.json? Lock files should be for applications, not libraries.
      - pknopf 8y ago
        Yes, but "npm install" will by default add a semver-compat package version. The vast majority of packages out there use what "npm install" gives you be default. Furthermore, even if I use exact versions in my package.json, that doesn't stop my referenced packages from internally referencing semver-compat versions. At the very least, you should reference exact versions at the top level. It is better than nothing.
ngrilly 8y ago
> They could have adopted dep and improved on it. Not really because: - The scope is different. dep is a package management tool. vgo is more than that. vgo intends to fully replace the go command. This is how we deprecate GOPATH. - The underlying principles are different enough to make it difficult to "improve" on dep. It looked easier to start from scratch. References: "Go += Package Versioning", Russ Cox, https://research.swtch.com/vgo-intro https://research.swtch.com/vgo-intro "FAQ: Why is the proposal not “use Dep”?", Russ Cox, https://github.com/golang/go/issues/24301#issuecomment-371228664 https://github.com/golang/go/issues/24301#issuecomment-37122...
- sagichmal 8y ago
  dep was always written with the intention that it would be merged into or replace to go tool. And it's still not settled that the underlying principles of vgo are valid -- which this article begins to unpack.
skywhopper 8y ago
Yes! Speaking as a sysadmin who writes Go tools from time to time along with a lot of Bash, Ruby, etc, I’m no expert in the land of Go. But in terms of designing my code to be shareable and figuring out how to integrate it with other source code I deal with every day is a mess. dep had seemed a very promising direction, and so I’m really frustrated to see it’s been deprecated and we are still seemingly years out from a solution which may not even work well in the end. Quite honestly the situation is actively holding back Go, as it’s very hard to convince developers used to lock files and well defined packaging to build anything in a language with no official story on this, even after a decade. Go is rapidly losing the momentum it had for a while now that Rust is starting to become a more stable target and given Go’s inability to address package versioning.
- Karrot_Kream 8y ago
  From what I read `dep` isn't deprecated, and is right now the recommended way to track dependencies. I used `dep` in a project I started a bit ago (around the time of the announcement of `vgo`) and it's been working out just great. That said I agree it's super frustrating, and the vgo announcement just had zero finesse. This isn't a language of experimenters hacking away, there are large production systems that run on solely Go code. Upending the main tool (`go` itself) and the versioning strategy is not to be taken lightly. > Go is rapidly losing the momentum it had for a while now that Rust is starting to become a more stable target and given Go’s inability to address package versioning. Here I disagree. Rust is fine for small utilities right now, but if you start building larger apps, developing with futures, tokio, and some of the other async libraries is a nightmare. The APIs are moving really quickly (`futures` hit 0.2, `tokio` is on 0.1 so I don't blame them really) and the ecosystem is still extremely immature. I'm excited for Rust, but it's not yet at the point that I want to build a complex application.

jstarks 8y ago

Could anyone distill this down for the casual reader? The author has a fairly laborious style and seems to be writing for an audience that is intimately familiar with the internals of dep and vgo.

lobster_johnson 8y ago
The article is about vgo, a proposal (with working prototype) to build package management into Go itself. vgo bundles several improvements, a major one being the introduction of "modules", which are versioned collections of packages. Modules, among other features, will let us decouple the import path from the file system structure and source repository, making the awkward and much-maligned $GOPATH finally obsolete. vgo also introduces a new dependency resolution algorithm, minimal version selection (MVS), which is different from traditional dependency resolution algorithms in that it will at any time choose the oldest allowed version that satisfy the minimum version constraints specified in the list of imported packages, where other systems will choose the newest. As MVS does not require (as with dep) a complex boolean SAT solver with potentially pathological, NP-complete cases, it is much simpler and faster to compute. However, it also has downsides. (Edited, thanks to pa7ch for correction.) The author likes pretty much everything about vgo except the MVS algorithm, which has been deeply controversial since the proposal was first published. The author goes into explanations of why he thinks MVS is a bad fit, but this is a complicated topic, so you really have read both the vgo proposal and this rebuttal to understand what it's all about. The article was written by Sam Boyer, one of the designers/authors of dep. While he has been courteous about it, there are obviously emotions at play; during its development, dep seemed to have the blessing of the official Go team, and this proposal came out of the blue and probably felt like an ambush.
- pa7ch 8y ago
  Minor correction: MVS requires modules to specify the minimum version to use of a module. The algorithm will then select the newest among those based on the idea that semver is forward compatible among minor versions. The name minimal version selection throws people off. Other algorithms would select the newest release within a major version even if no module has ever tested it and it was released 5 seconds ago. When authors do make incompatible changes and MVS selects a broken dependency, as expected, there are manual escape hatches. More complicated NP complete algo's would have more of an automated answer here by allowing dependencies to have boolean constraints (greater then X, less then Y, not Z etc.) on version so that there is this collaborative summation of constraints to work around authors violating semver or bugs. To summerize poorly: It seems Sam Boyer regrets that dep didn't do what vgo does by default, but believes that a SAT solver could kinda do MVS for the happy path but still solve against the introduction of boolean constraints. Personally, I'd like to let vgo play out a bit, I'm not convinced boolean constraints are something I really want imposed on me by my dependencies. More info for those who are looking for more context: https://research.swtch.com/vgo https://research.swtch.com/vgo https://github.com/golang/go/issues/24301 https://github.com/golang/go/issues/24301 Whether you agree or disagree, its novel research in this field and excellent technical writing worth reading in its entirety over a cup of coffee.

pcwalton 8y ago

The basic upshot of all this is that vgo's attempt to avoid NP-completeness in the dependency resolution algorithm is solving a problem that doesn't matter in practice, and is simultaneously creating downsides that do matter in practice. This is consistent with what I've seen with systems like Cargo. If I had to make a list of top 10 issues I run into with Cargo, theoretical scalability of the core dependency resolution algorithm wouldn't make the cut.

enqk 8y ago
On the other hand, git became successful by being fast and by not solving certain of the hard theoretical problems that specialists were obsessed with solving. It's hard to know before hand what the healthy compromise might be.
- lobster_johnson 8y ago
  git may have become successful, but that doesn't mean it's right. I'd argue that most developers still don't really understand git. I know seasoned, senior developers who struggle when git gets into an unfamiliar state, or who are completely mystified as to how it actually works. And don't forget that git spent years improving its initially horrific UI to make it easier and more palatable to people who aren't kernel developers. If you compare git to rival tools such as Mercurial and Fossil or even Darcs, it's pretty evident that git didn't really win on technical merit, user friendliness or indeed hardly any other metric. It was fast, and it had the dubious advantage of being better than a lot of crappy alternatives at the time, and of being the product of a famous developer. But if technical merit was how we chose software, we might all be using Mercurial today. It's quite possible we'd be better off, too. git's punting of "certain of the hard theoretical problems" lead to major shortcomings in its design. For example, git does not track branching histories. Once you merge your changes back into a branch and delete the origin branch, the path that your commits took are lost. Another example: Since git's data model works entirely on repository snapshots (there are no "patches" or logical operations, only snapshots), it doesn't know about file renames. Some commands, like log and diff, do understand how to detect renames (but not out of the box!), but it's fundamentally a fuzzy-matching operation that can only work perfectly if you separate your rename commits from modification commits, which is rarely feasible. Don't get me wrong, git has overall been a force for good, but I think it's a pretty terrible example in this particular case. In fact, I'm not sure I can think of many cases where the "worse is better" philosophy was ultimately a good idea. In every instance that comes to mind (MySQL, MongoDB, PHP), the fast-and-loose attitude ultimately came back to hurt everyone, and years were spent paying for those mistakes, plugging one hole at a time. The original decision by the Go team to build package management into the language ("go get") was made without really understanding the problem, and this decision has been haunting us pretty much since day one, with developers having to chase unofficial package management tools of the week in order to get their work done. I really want it to be done right, even if takes a bit longer. It's a problem worth solving well.

IshKebab 8y ago

This badly needs a TL;DR. If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words.

TheDong 8y ago
> If there's something seriously wrong with vgo's dependency versioning then it should be possible to explain it in less than 100k words. That is a very lazy argument. It's perfectly reasonable to claim something is flawed for numerous complex reasons. It's equally reasonable to claim that a complex flaw may be difficult to explain concisely. Most importantly though, I don't think Sam is claiming that there's an obvious fatal flaw in vgo, but rather that it is not the ideal choice given the problem space. To make that more subtle argument, it is necessary to fully define the alternatives and problem space, which is what a lot of the words are attempting to do. The tl;dr would be along the lines of "vgo knowingly makes a set of tradeoffs which I think are worse than the tradeoffs that are possible with another hypothetical option, which I am also proposing"... but that tl;dr is not really a very useful one, and attempting to condense a more meaningful one will lose a bit too much nuance I think.
- closeparen 8y ago
  The post could stand to have an introduction that lays out its arguments at a high level, as is hammered into students at every level of English class.
geodel 8y ago
I am afraid it is because lot of expectations were built that dep author would reveal some rather serious flaws with vgo. But since he has nothing explicit, many thousand words are required to cover that up.

red_admiral 8y ago

> If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. Problems can be NP-complete. Algorithms cannot - they can be exponential time (and if your choice is between exponential and double exponential, you pick the former!) In practice, and especially for SAT solving, as far as I understand the state of the art is that the general case is NP-hard but we have tools that work surprisingly well in those cases that reality tends to produce. In some more academic cases, we even have heuristic solvers that are not even guaranteed to terminate, but usually do so within a couple of seconds, so you can get away with the "beginner's solution to the halting problem".

munificent 8y ago
> In practice, and especially for SAT solving, as far as I understand the state of the art is that the general case is NP-hard but we have tools that work surprisingly well in those cases that reality tends to produce. You're correct. Modern version solvers need fairly sophisticated algorithms, but it's not rocket science. It's mostly a non-problem for most users of most package managers most of the time.
- bennofs 8y ago
  The thing that worries me about this line of thinking is that as far as I know, we don't know much about why SAT solvers tend to work well in practice. So if I ever happen to hit a problem that the solver cannot solve, all we can say is "well bad luck". If the algorithm fails, I won't have any way to know why. This strikes me as a bad situation to be in.

rosshemsley 8y ago

Anyone have a bullet-pointed summary of the "crux" of what the author dislikes about MVS? I find MVS to be a very natural and simple solution to versioning - I feel like it's much closer to how people actually manage their dependencies in practice. After an admittedly low-effort skim of this article, most of the arguments seem to be that the author doesn't think it "feels right". Is there something more concrete hidden in here?

cdoxsey 8y ago
Its a long post, but I don't think he actually gives the reasons, rather he intends to write several additional posts that will.
- ngrilly 8y ago
  Honestly, Russ Cox published his first post about vgo on February 20 (https://research.swtch.com/vgo https://research.swtch.com/vgo). It's been almost 3 months then. It's not fair to ask the community to wait any longer. If Sam Boyer has good reasons to oppose vgo, then he should publish them now.
  - sagichmal 8y ago
    Russ waited over a year and a half before engaging meaningfully with the community on package management, and waited another nine months before coming up with this greenfield proposal. There's no reason to rush things now.
    - ngrilly 8y ago
      > There's no reason to rush things now. I disagree. I want things to move forward as soon as possible. I've been waiting so long for a dependency management solution and for GOPATH deprecation! Russ came with his "greenfield" proposal because he was not convinced that dep could become the long term solution to dependency management in Go. Sam and the whole dep team have been working on package management for more than a year now. If they see important issues with vgo, I think they should be able to explain them succinctly in a post, at least to convince other gophers to hold on.

eknkc 8y ago

For an introduction to vgo and to see the other side of argument, watch the presentation from Russ Cox: https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg I was sceptical but now kind of sold on the idea after wathing the presentation.

nicpottier 8y ago

Though this is well written and I'm sure Sam is trying to set a tone of respect and appreciation of what is good in vgo so as to start on the right foot, he is also "burying the lead" in that he never gets to the point. I think Russ did an unusually good job of explaining the ideas around vgo, grounded by using real world examples. He set a high bar there, but hopefully Sam adopts some of the same methods of very concrete examples to explain his criticisms. As it stands this is an interesting piece of reading but is too high level and abstract in its criticism to be effective. Looking forward to future installments that hopefully that will make Sam's criticisms clear. From what I can tell one piece of criticism is the inability of vgo to define incompatibilities. That is that if you know these versions won't work, you can say never allow that version in your dependency graph. I'd love to hear more examples of this in the real world. Since vgo dependencies are basically always "pinned" this seems like that not big a deal to me, but maybe I'm missing something. Perhaps this is something that occurs with having two of your dependencies having different dependencies on a third package and one of those works only with dep3@v1.2 and the other only works with dep3@v1.3 or somesuch. Maybe there are ways around that or maybe that's not the actual problem (again, hope Sam explains more in the future). I think there is a really interesting element to MSV that will actually drive Go culture though (and I'm a big believer in the importance of the culture of a language) which I think may minimize the problems that Sam is talking about and for the betterment of the community. As others have noted, I hope that Sam comes out with the rest of this series quickly so that the go community can commit do a direction sooner rather than later and move on to actually building this new ecosystem.

4ad 8y ago
> he is also "burying the lead" The expression is burying the lede, not lead.
- nicpottier 8y ago
  Hah, thanks, learn something every day! https://www.merriam-webster.com/words-at-play/bury-the-lede-versus-lead https://www.merriam-webster.com/words-at-play/bury-the-lede-...
8y ago

abiox 8y ago

i think one fundamental problem is that for most ecosystems, versioning is almost entirely arbitrary, manual and dissociated from language semantics. the elm package manager seems to take a step in the right direction where changing a function signature will force a version bump if you want to publish the update. (iirc) are there other languages/ecosystems that do something interesting with versioning?

steveklabnik 8y ago
We're working on similar tooling to that in Rust https://github.com/rust-lang-nursery/rust-semverver https://github.com/rust-lang-nursery/rust-semverver