8 ms·

There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team

by amasad 8y ago

There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team is not taking for granted. One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy (https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro). Another is shared (or recursive) dependencies. The semver answer to this is that if the versions match then they should be shared if not then they should be duplicated. But what do you do if it's a singleton, listens on a port, or exposes a port. Right now, with npm for example, good luck with that, you're in for a world of pain. On the other hand with vgo's "semantic import versioning" they're trying to make version interop more explicit. Just because 90% of the time mainstream package mangers work doesn't mean it's a solved problem. Kudos for the Go team for trying to advance the state of the art.

kibwen 8y ago
> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?
- amasad 8y ago
  More things, more state = bloat. Don't you agree that if you can do something without holding state it's invariably simpler? That's what MVS does. There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100 https://github.com/rust-lang/cargo/issues/4100
  - kibwen 8y ago
    > Don't you agree that if you can do something without holding state it's invariably simpler? Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS: "If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/ https://sdboyer.io/vgo/intro/
- jasonwatkinspdx 8y ago
  There are situations where lock files, as implemented in some languages, can generate conflicts. Honestly I consider this a benefit, because I'd like to know about different expectations as early and explicitly as possible. The bloat argument seems frankly absurd to me.
- blaisio 8y ago
  It's pretty easy to get a merge conflict from a lock file. If two developers add a dependency and then try to merge their changes together, it can happen easily.
  - kibwen 8y ago
    In that scenario, you're going to get a merge conflict anyway in your manifest (e.g. in go.mod).
    - grahamedgecombe 8y ago
      It's probably easier to manually merge the manifest (as they're much more human-readable) than a lock file though.
      - steveklabnik 8y ago
        Lock files are easy enough to resolve that npm, in newer versions, will do it for you automatically, if you ask it to.
  - Groxx 8y ago
    That's correct behavior though - you can't merge "upgrade A" and "upgrade B" in isolation with literally any confidence that it works.
eridius 8y ago
The "minimal versions" strategy basically just replaces the lockfile with your dependency list. Two branches want to update the same dependency? With a lockfile, they both update the lockfile and will have a merge conflict (assuming the updates aren't identical). With "minimal versions", they both update the file that declares the dependency, and, you guessed it, have a merge conflict.
- petre 8y ago
  Minimal versions put the burden of background compatibility on the maintainer of the dependency. If such is the case, library maintainers are constrained to refrain from breaking background compatibility at minor version changes, like Go itself does. In this case, you can merge, resolve the conflict by incrementing the dependency's version number, rebuild and retest and if all goes well commit. With lockfiles if branch X requires 1.4, branch Y requires 1.5 and 1.5 breaks 1.4, good luck with that. You'll need to update branch X to support 1.5.
  - eridius 8y ago
    SemVer already solves that problem.
- sseth 8y ago
  Most package managers ignore the lock files of the dependencies when picking up versions - they only use the lockfile of the topmost project for picking up versions. The big difference with the MVS strategy in vgo is that the dependency list of a dependency is actually used to determine the version. If package A uses B, which was tested with v1.2 of package C, you will get v1.2 of C, even if there is a later version of C available. In the typical package manager scenario, the dependency list of B may be pointing to an old version of C which does not even work with B - there is generally no way to ask : give me the latest version of C which was tested with B.
  - eridius 8y ago
    I think package managers should have an option to run a build of your package using the oldest versions instead of newest versions, just to validate that the build works, but that this shouldn't be the normal way to resolve dependencies.
oceanswave 8y ago
>One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. You can’t have your cake and eat it too

by amasad 8y ago

kibwen 8y ago

> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?

amasad 8y ago
More things, more state = bloat. Don't you agree that if you can do something without holding state it's invariably simpler? That's what MVS does. There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100 https://github.com/rust-lang/cargo/issues/4100
- kibwen 8y ago
  > Don't you agree that if you can do something without holding state it's invariably simpler? Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS: "If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/ https://sdboyer.io/vgo/intro/
jasonwatkinspdx 8y ago
There are situations where lock files, as implemented in some languages, can generate conflicts. Honestly I consider this a benefit, because I'd like to know about different expectations as early and explicitly as possible. The bloat argument seems frankly absurd to me.
blaisio 8y ago
It's pretty easy to get a merge conflict from a lock file. If two developers add a dependency and then try to merge their changes together, it can happen easily.
- kibwen 8y ago
  In that scenario, you're going to get a merge conflict anyway in your manifest (e.g. in go.mod).
  - grahamedgecombe 8y ago
    It's probably easier to manually merge the manifest (as they're much more human-readable) than a lock file though.
    - steveklabnik 8y ago
      Lock files are easy enough to resolve that npm, in newer versions, will do it for you automatically, if you ask it to.
- Groxx 8y ago
  That's correct behavior though - you can't merge "upgrade A" and "upgrade B" in isolation with literally any confidence that it works.

eridius 8y ago

The "minimal versions" strategy basically just replaces the lockfile with your dependency list. Two branches want to update the same dependency? With a lockfile, they both update the lockfile and will have a merge conflict (assuming the updates aren't identical). With "minimal versions", they both update the file that declares the dependency, and, you guessed it, have a merge conflict.

petre 8y ago
Minimal versions put the burden of background compatibility on the maintainer of the dependency. If such is the case, library maintainers are constrained to refrain from breaking background compatibility at minor version changes, like Go itself does. In this case, you can merge, resolve the conflict by incrementing the dependency's version number, rebuild and retest and if all goes well commit. With lockfiles if branch X requires 1.4, branch Y requires 1.5 and 1.5 breaks 1.4, good luck with that. You'll need to update branch X to support 1.5.
- eridius 8y ago
  SemVer already solves that problem.
sseth 8y ago
Most package managers ignore the lock files of the dependencies when picking up versions - they only use the lockfile of the topmost project for picking up versions. The big difference with the MVS strategy in vgo is that the dependency list of a dependency is actually used to determine the version. If package A uses B, which was tested with v1.2 of package C, you will get v1.2 of C, even if there is a later version of C available. In the typical package manager scenario, the dependency list of B may be pointing to an old version of C which does not even work with B - there is generally no way to ask : give me the latest version of C which was tested with B.
- eridius 8y ago
  I think package managers should have an option to run a build of your package using the oldest versions instead of newest versions, just to validate that the build works, but that this shouldn't be the normal way to resolve dependencies.

oceanswave 8y ago

>One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. You can’t have your cake and eat it too