6 ms·

Vgo has certainly been a disaster from a communication and community point of view. From an outsider perspective, it looks like a case of maintainer arrogance,

by Athas 8y ago

Vgo has certainly been a disaster from a communication and community point of view. From an outsider perspective, it looks like a case of maintainer arrogance, and it is hard to believe that Go really has needs that are significantly different from those of e.g. Rust or Java. However, I am also in the unusual position that I had to design and implement a package manager for my own little language recently. My original hunch was to just create a Cargo clone, but after reading Russ Cox's writings, I ended up cloning vgo (with a few simplifications). The implementation was wonderfully simple - everything just fit together. The algorithms are trivial. Operationally, it is easy to understand what the package manager is doing (even more so than for vgo, as vgo has do deal with various Go idiosyncracies and backwards compatibility). This is in stark contrast to my experience with other package managers, which are temperamental beasts at the best of times. When they work, everything's groovy, but their error modes are easily incomprehensible. I think vgo's approach of restricting expressivity and streamlining processes is the right one. But since nobody has used such a package manager before, it remains to be seen whether it works in practice. For my own experiment, I do have one data point: I showed people the (rather brief) documentation for my vgo-inspired package manager, and they all felt it was very simple to follow and easy understand what it did, and how.

pas 8y ago
Could you contrast Cargo and vgo for those who only know the former?
- Athas 8y ago
  There are lots of small differences, but I would say there are two major: * Semantic Import Versioning. A package is identified by some name. I think vgo calls it an "import path", but I have also seen "package path" used. Every version of that package must remain compatible with previous versions. You may not break compatibility (i.e. increment the major semver number) without also renaming the package. There is some syntactic sugar that makes it clear that a version 2 is closely related to the original version 1, but from the point of view of vgo, the two versions are completely distinct packages. This also means a program may depend on several major versions of the same package (which Russ claims is necessary when doing gradual migrations of large programs). * Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree. Upper bounds are not supported. This both makes the solving algorithm trivial (which probably doesn't matter much), but also gives you reproducible builds without using a lockfile (which is nice), and it gives you a very simple operational model for what the package manager is doing (which I think is crucial). All package managers suck and all package managers break, but I think when vgo breaks, it will be more obvious what is wrong. Time will tell whether it works in practice! (And I'm not even a Go programmer; I just like the thoughtfulness that goes into the tooling.)
  - jessaustin 8y ago
    Minimal Version Selection. That's really interesting. Obviously this would lead to fewer regressions, but it seems opposed to the "security problems fixed for free" scenario you hear about with other package dependency schemes. Were we all just imagining that scenario? Is there another way we should be handling vulnerabilities, i.e. by "repudiating" old versions rather than simply releasing new ones? Then you could have something like "Minimal Unrepudiated Version Selection".
    - Athas 8y ago
      I think reproducible builds are more important than having security problems fixed behind your back. It's not like vgo makes it hard to upgrade all dependencies to their most recent version - it's done automatically with a single command, if you want it.
      - jessaustin 8y ago
        What you're saying makes perfect sense. This is what makes HN great. Incidentally, there's nothing about e.g. npm that would prevent this scheme: just set literal version numbers instead of using operators. The two systems definitely have different "best practices".
      - necubi 8y ago
        Other package managers (cargo, gemfile, etc.) also give reproducible builds by using a lockfile. Upgrading dependencies to their maximum version is an explicit action (cargo update). It's not clear to me why the go team things lockfiles are a huge issue.
        kjksf 8y ago
        Lock files also remove the "silent security fix" that was touted as the benefit of other systems. You can't have it both ways. You either get reproducible builds (by default with vgo, by adding lock files in other systems) or you get silent upgrades that potentially fix security issues (but also potentially introduce security issues). You also mis-characterize the position of vgo creator. He doesn't think that lock files are a huge issue per se. The difference is in default behavior of the system: vgo by default picks predictable, consistent version of dependencies. That version doesn't change if dependencies release new versions. In fact, it's not just the default behavior but the only behavior. Other systems allow specifying complex rules for which version of dependency to pick and they all allow for a scenario where you run the same algorithm over the same rules but pick different version because in the meantime some dependency has released a new version. It's such a big problem that all those system end up introducing lock files, which is a tacit admission that what vgo does by default is a good idea. vgo doesn't need lock files to get the benefit of lock files, which is a nice cherry on top but the real advancement is in changing the default behavior of how resolving versions of dependencies work.
zeeboo 8y ago
For what it's worth, I analyzed as many Gopkg.{lock,toml} files I could find in the wild, and found that vgo's algorithms would service all of them. I've done similar analysis for a smaller number of Rust projects using Cargo and had the same findings. I've found it hard to find projects in the wild that actually do any non-trivial version selection. I think this is some good evidence that it will work out in practice. I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are. https://github.com/zeebo/dep-analysis https://github.com/zeebo/dep-analysis
- Athas 8y ago
  > I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are. I think you are right. And I doubt it hurts that SAT solving is a fun problem! My main package management experience has been with Haskell, which has used the cabal tool for many years. Cabal was a traditional solver-based tool (with the added pain of a global mutable package database, although that is going away), and it frequently broke down in confusing ways. Cabal hell was a widely used term. A few years ago, another tool arrived on the scene, Stack, which used the same package format and such as cabal, but snapshotted the central package list (Hackage) by gathering subsets of packages and versions that were guaranteed to work together (or at least do not have conflicting bounds). It works well[0], and although it does in principle result in a major loss in flexibility, it's rarely something I miss. Importantly, the improvement in reliability was nothing short of astounding. That certainly helped convince me that flexibility may not be a needed feature for a (language) package manager. [0]: There are all sorts of socio-political stack/cabal conflicts in the Haskell community now, but I'm not sure they are founded in technical issues.
- pcwalton 8y ago
  The problem with minimal version selection isn't that it can't support nontrivial version selection but rather that it doesn't automatically select the newest versions of packages.
  - kaeshiwaza 8y ago
faitswulff 8y ago
That's just the thing, right? It seems to be a clearly superior technical solution, but it has thoroughly failed from the perspective of human relationships.
pcwalton 8y ago
I've never seen Cargo errors be incomprehensible. Solving versions just isn't a problem in practice. Minimal version selection optimizes for convenience of the package manager implementer at the expense of the users. To me, that's the wrong trade.

by Athas 8y ago

pas 8y ago

Could you contrast Cargo and vgo for those who only know the former?

Athas 8y ago
There are lots of small differences, but I would say there are two major: * Semantic Import Versioning. A package is identified by some name. I think vgo calls it an "import path", but I have also seen "package path" used. Every version of that package must remain compatible with previous versions. You may not break compatibility (i.e. increment the major semver number) without also renaming the package. There is some syntactic sugar that makes it clear that a version 2 is closely related to the original version 1, but from the point of view of vgo, the two versions are completely distinct packages. This also means a program may depend on several major versions of the same package (which Russ claims is necessary when doing gradual migrations of large programs). * Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree. Upper bounds are not supported. This both makes the solving algorithm trivial (which probably doesn't matter much), but also gives you reproducible builds without using a lockfile (which is nice), and it gives you a very simple operational model for what the package manager is doing (which I think is crucial). All package managers suck and all package managers break, but I think when vgo breaks, it will be more obvious what is wrong. Time will tell whether it works in practice! (And I'm not even a Go programmer; I just like the thoughtfulness that goes into the tooling.)
- jessaustin 8y ago
  Minimal Version Selection. That's really interesting. Obviously this would lead to fewer regressions, but it seems opposed to the "security problems fixed for free" scenario you hear about with other package dependency schemes. Were we all just imagining that scenario? Is there another way we should be handling vulnerabilities, i.e. by "repudiating" old versions rather than simply releasing new ones? Then you could have something like "Minimal Unrepudiated Version Selection".
  - Athas 8y ago
    I think reproducible builds are more important than having security problems fixed behind your back. It's not like vgo makes it hard to upgrade all dependencies to their most recent version - it's done automatically with a single command, if you want it.
    - jessaustin 8y ago
      What you're saying makes perfect sense. This is what makes HN great. Incidentally, there's nothing about e.g. npm that would prevent this scheme: just set literal version numbers instead of using operators. The two systems definitely have different "best practices".
    - necubi 8y ago
      Other package managers (cargo, gemfile, etc.) also give reproducible builds by using a lockfile. Upgrading dependencies to their maximum version is an explicit action (cargo update). It's not clear to me why the go team things lockfiles are a huge issue.
      - kjksf 8y ago
        Lock files also remove the "silent security fix" that was touted as the benefit of other systems. You can't have it both ways. You either get reproducible builds (by default with vgo, by adding lock files in other systems) or you get silent upgrades that potentially fix security issues (but also potentially introduce security issues). You also mis-characterize the position of vgo creator. He doesn't think that lock files are a huge issue per se. The difference is in default behavior of the system: vgo by default picks predictable, consistent version of dependencies. That version doesn't change if dependencies release new versions. In fact, it's not just the default behavior but the only behavior. Other systems allow specifying complex rules for which version of dependency to pick and they all allow for a scenario where you run the same algorithm over the same rules but pick different version because in the meantime some dependency has released a new version. It's such a big problem that all those system end up introducing lock files, which is a tacit admission that what vgo does by default is a good idea. vgo doesn't need lock files to get the benefit of lock files, which is a nice cherry on top but the real advancement is in changing the default behavior of how resolving versions of dependencies work.

zeeboo 8y ago

For what it's worth, I analyzed as many Gopkg.{lock,toml} files I could find in the wild, and found that vgo's algorithms would service all of them. I've done similar analysis for a smaller number of Rust projects using Cargo and had the same findings. I've found it hard to find projects in the wild that actually do any non-trivial version selection. I think this is some good evidence that it will work out in practice. I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are. https://github.com/zeebo/dep-analysis https://github.com/zeebo/dep-analysis

Athas 8y ago
> I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are. I think you are right. And I doubt it hurts that SAT solving is a fun problem! My main package management experience has been with Haskell, which has used the cabal tool for many years. Cabal was a traditional solver-based tool (with the added pain of a global mutable package database, although that is going away), and it frequently broke down in confusing ways. Cabal hell was a widely used term. A few years ago, another tool arrived on the scene, Stack, which used the same package format and such as cabal, but snapshotted the central package list (Hackage) by gathering subsets of packages and versions that were guaranteed to work together (or at least do not have conflicting bounds). It works well[0], and although it does in principle result in a major loss in flexibility, it's rarely something I miss. Importantly, the improvement in reliability was nothing short of astounding. That certainly helped convince me that flexibility may not be a needed feature for a (language) package manager. [0]: There are all sorts of socio-political stack/cabal conflicts in the Haskell community now, but I'm not sure they are founded in technical issues.
pcwalton 8y ago
The problem with minimal version selection isn't that it can't support nontrivial version selection but rather that it doesn't automatically select the newest versions of packages.
- kaeshiwaza 8y ago

faitswulff 8y ago

That's just the thing, right? It seems to be a clearly superior technical solution, but it has thoroughly failed from the perspective of human relationships.

pcwalton 8y ago

I've never seen Cargo errors be incomprehensible. Solving versions just isn't a problem in practice. Minimal version selection optimizes for convenience of the package manager implementer at the expense of the users. To me, that's the wrong trade.