16 ms·

I wish they had just copied Rust/Cargo. I remember reading a comment on GitHub somewhere from one of the Go maintainers who responded to someone expressing a si

I wish they had just copied Rust/Cargo. I remember reading a comment on GitHub somewhere from one of the Go maintainers who responded to someone expressing a similar sentiment and his reply was basically that Go is somehow different than every other language and they need to explore and find a unique custom solution for their particular use case. Has it ever been addressed anywhere why the tried and true "list of packages" + "lock file" paradigm is not good enough for Go?

TheSwordsman 8y ago
I wish it was as simple as that. Instead it's that that Russ Cox is fundamentally against SAT solvers, and has opted take this alternative approach instead. I think they would have been better off iterating on the things that have worked for other languages and tools, and arguably a SAT solver is one of them.
lobster_johnson 8y ago
Rust/Cargo had the luxury of being a greenfield project that could adopt semver from the beginning, whereas Go made the mistake of starting out, and then going years, without any official package management solution. As a result, Go has a swathe of applications and libraries that use specific workflows as well as a mélange of community-developed package management tools such as godep, Glide and dep. The semver standard, in particular, has been inconsistently adopted by the Go community. In other words, any new Go tool either has to support/import existing code, or to wipe the slate clean and say that for a package to be importable it has to follow a new spec. dep decided on the former, and my impression is that this has had unfortunate consequences, because that inherits a lot of historical baggage. We've been using dep for a while (having escaped the bugfest that is Glide, which used a very similar approach), and it's pretty evident that the solver is buggy and slow and also complicated enough that fixing issues like [1] can only be done by a select few that already understand the codebase. I'm not in a position to judge what the causes of all of these issues are, though I'd wager they're not entirely unrelated to the inherent complexity of SAT solving. The current dep issue tracker is full [2] of reports mentioning the solver, not to mention that dep currently has problems with known libraries such as the Kubernetes client [3] and Protobuf. (Google-related projects have historically used godep.) Again, possibly related to this specific implementation and not necessarily something that would apply to a hypothetical "Cargo for Go", but I don't know. Any idea how Cargo compares to dep overall? [1] https://github.com/golang/dep/issues/1306 https://github.com/golang/dep/issues/1306 — this one is a nightmare if you work anything related to Kubernetes. [2] https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+solver https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+... [3] https://github.com/golang/dep/issues/1207 https://github.com/golang/dep/issues/1207
- pcwalton 8y ago
  Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indeed the solver that is the problem (which, again, I'm skeptical of—I'd have to see profiling data to believe it), then it could just come down to optimization differences between rustc and Go 6g/8g.
  - lobster_johnson 8y ago
    Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a086501d8f https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...
    - kibwen 8y ago
      That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dirkjan.ochtman.nl/files/servo-graph.svg https://dirkjan.ochtman.nl/files/servo-graph.svg ; the number of transitive dependencies is quite large and yet the runtime of version selection is negligible.
      - TheDong 8y ago
        It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation). This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io). If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep. I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.
jchw 8y ago
I think Cargo-like is the direction things are headed even with vgo. In a way, though, I don't think the answer is that Go is "special," just that they wanted to explore the problem space of package management, dependencies, etc. in more detail before locking things down. It may have ended in a slightly weakened ecosystem temporarily, but I hope it helps to ensure that the semantics and behavior of Go's package management is level headed and simple. Hopefully, by Go 2, a lot of the common concerns with the Go programming language will be answered...
- lobster_johnson 8y ago
  The vgo proposal explicitly rejects the "Cargo way" [1]. There's no lock file, and the MVS algorithm requires, as far as I recall, that the go.mod file is modified whenever the developer wants to update to a new version. [1] https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro
  - jchw 8y ago
    Indeed, but it does seem like a central repository is possibly the direction it's headed, or at least away from source repositories. Personally, I won't be too upset by MVS if it's one of the only deviations, though I do agree with Sam Boyer on the matter, who I actually had the pleasure of speaking with a bit ago.
    - steveklabnik 8y ago
      Cargo doesn’t require a central repository at all, to be clear.
      - dikaiosune 8y ago
        I'd actually be interested to find out how Cargo's performance compares in a situation as TheDong describes where all of the dependencies are being fetched directly from git. Part of me wants to say "well of course Rust's tool is faster" but it would be interesting to see just how much crates.io acts as a performance optimization for running builds, installing deps, etc.
aviau 8y ago
Yes, it has! Russ Cox talks about this, amongst other things, in this talk: https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg I am sure there are written versions of this somewhere...
lobster_johnson 8y ago
Cox discusses Cargo here, and why he doesn't like it: https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro
- eberkund 8y ago
  Thanks for posting this. It's much more fleshed out than my hazy memory of a GitHub comment although I still don't agree that all this effort it is worth the "simplification" of not having a lock file.
  - kibwen 8y ago
    The aversion to lockfiles is especially head-scratching given that vgo has a concept of a go.modverify file (separate from the go.mod file) which is used to store version hashes, which is a function that lockfiles usually fulfill. I think the advantage is supposed to be that modverify files are optional, but in practice I don't see why anybody wouldn't want one given that it provides additional security for free.
- vhbit 8y ago
  > The lock file stops future upgrades; once it is written, your build stays on serde 1.0.27 even when 1.0.28 is released. That means that the only difference is that `vgo` doesn't require lock file for reproducible builds, now the question is what's considered so terrible wrong/dangerous with having a lock file?
  - iambvk 8y ago
    If there was a security fix in a package, MVS allows build infra to automatically upgrade (assuming packages imported are well maintained by their owners) where as lock-files require manual intervention. Imagine the packaging and development ecosystem in internet scale.
    - kibwen 8y ago
      I'm not sure what this is asserting. There's no association between MVS and build infra. If you have a security fix to apply, you'd run `vgo get -u` to upgrade all your modules; systems like Cargo that use lockfiles would run `cargo upgrade`. This is just as automatic. In fact, vgo is less automatic here because systems like Cargo allow users to decide when they want certain upgrades automatically, which means that new downstream users can get the security fix even without needing to explicitly upgrade. And vgo also requires more manual intervention to upgrade than traditional build systems, because if the upgrade requires a major version bump then you also have to edit the import statements in your source files as well. vgo is both less automatic and requires more manual labor; that appears to be an intentional choice on behalf of the vgo manifesto.
amasad 8y ago
There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team is not taking for granted. One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy (https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro). Another is shared (or recursive) dependencies. The semver answer to this is that if the versions match then they should be shared if not then they should be duplicated. But what do you do if it's a singleton, listens on a port, or exposes a port. Right now, with npm for example, good luck with that, you're in for a world of pain. On the other hand with vgo's "semantic import versioning" they're trying to make version interop more explicit. Just because 90% of the time mainstream package mangers work doesn't mean it's a solved problem. Kudos for the Go team for trying to advance the state of the art.
- kibwen 8y ago
  > The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?
  - amasad 8y ago
    More things, more state = bloat. Don't you agree that if you can do something without holding state it's invariably simpler? That's what MVS does. There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100 https://github.com/rust-lang/cargo/issues/4100
    - kibwen 8y ago
      > Don't you agree that if you can do something without holding state it's invariably simpler? Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS: "If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/ https://sdboyer.io/vgo/intro/
mseepgood 8y ago
> I wish they had just copied Rust/Cargo Cargo is really a suboptimal design. With vgo Go advances the state of the art of dependency management.
kaeshiwaza 8y ago
I believe this video of rsc explain some concerns about the Cargo's approach and lock file issues https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg

TheSwordsman 8y ago

I wish it was as simple as that. Instead it's that that Russ Cox is fundamentally against SAT solvers, and has opted take this alternative approach instead. I think they would have been better off iterating on the things that have worked for other languages and tools, and arguably a SAT solver is one of them.

lobster_johnson 8y ago

Rust/Cargo had the luxury of being a greenfield project that could adopt semver from the beginning, whereas Go made the mistake of starting out, and then going years, without any official package management solution. As a result, Go has a swathe of applications and libraries that use specific workflows as well as a mélange of community-developed package management tools such as godep, Glide and dep. The semver standard, in particular, has been inconsistently adopted by the Go community. In other words, any new Go tool either has to support/import existing code, or to wipe the slate clean and say that for a package to be importable it has to follow a new spec. dep decided on the former, and my impression is that this has had unfortunate consequences, because that inherits a lot of historical baggage. We've been using dep for a while (having escaped the bugfest that is Glide, which used a very similar approach), and it's pretty evident that the solver is buggy and slow and also complicated enough that fixing issues like [1] can only be done by a select few that already understand the codebase. I'm not in a position to judge what the causes of all of these issues are, though I'd wager they're not entirely unrelated to the inherent complexity of SAT solving. The current dep issue tracker is full [2] of reports mentioning the solver, not to mention that dep currently has problems with known libraries such as the Kubernetes client [3] and Protobuf. (Google-related projects have historically used godep.) Again, possibly related to this specific implementation and not necessarily something that would apply to a hypothetical "Cargo for Go", but I don't know. Any idea how Cargo compares to dep overall? [1] https://github.com/golang/dep/issues/1306 https://github.com/golang/dep/issues/1306 — this one is a nightmare if you work anything related to Kubernetes. [2] https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+solver https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+... [3] https://github.com/golang/dep/issues/1207 https://github.com/golang/dep/issues/1207

pcwalton 8y ago
Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indeed the solver that is the problem (which, again, I'm skeptical of—I'd have to see profiling data to believe it), then it could just come down to optimization differences between rustc and Go 6g/8g.
- lobster_johnson 8y ago
  Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a086501d8f https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...
  - kibwen 8y ago
    That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dirkjan.ochtman.nl/files/servo-graph.svg https://dirkjan.ochtman.nl/files/servo-graph.svg ; the number of transitive dependencies is quite large and yet the runtime of version selection is negligible.
    - TheDong 8y ago
      It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation). This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io). If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep. I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.

jchw 8y ago

I think Cargo-like is the direction things are headed even with vgo. In a way, though, I don't think the answer is that Go is "special," just that they wanted to explore the problem space of package management, dependencies, etc. in more detail before locking things down. It may have ended in a slightly weakened ecosystem temporarily, but I hope it helps to ensure that the semantics and behavior of Go's package management is level headed and simple. Hopefully, by Go 2, a lot of the common concerns with the Go programming language will be answered...

lobster_johnson 8y ago
The vgo proposal explicitly rejects the "Cargo way" [1]. There's no lock file, and the MVS algorithm requires, as far as I recall, that the go.mod file is modified whenever the developer wants to update to a new version. [1] https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro
- jchw 8y ago
  Indeed, but it does seem like a central repository is possibly the direction it's headed, or at least away from source repositories. Personally, I won't be too upset by MVS if it's one of the only deviations, though I do agree with Sam Boyer on the matter, who I actually had the pleasure of speaking with a bit ago.
  - steveklabnik 8y ago
    Cargo doesn’t require a central repository at all, to be clear.
    - dikaiosune 8y ago
      I'd actually be interested to find out how Cargo's performance compares in a situation as TheDong describes where all of the dependencies are being fetched directly from git. Part of me wants to say "well of course Rust's tool is faster" but it would be interesting to see just how much crates.io acts as a performance optimization for running builds, installing deps, etc.

aviau 8y ago

Yes, it has! Russ Cox talks about this, amongst other things, in this talk: https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg I am sure there are written versions of this somewhere...

lobster_johnson 8y ago

Cox discusses Cargo here, and why he doesn't like it: https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro

eberkund 8y ago
Thanks for posting this. It's much more fleshed out than my hazy memory of a GitHub comment although I still don't agree that all this effort it is worth the "simplification" of not having a lock file.
- kibwen 8y ago
  The aversion to lockfiles is especially head-scratching given that vgo has a concept of a go.modverify file (separate from the go.mod file) which is used to store version hashes, which is a function that lockfiles usually fulfill. I think the advantage is supposed to be that modverify files are optional, but in practice I don't see why anybody wouldn't want one given that it provides additional security for free.
vhbit 8y ago
> The lock file stops future upgrades; once it is written, your build stays on serde 1.0.27 even when 1.0.28 is released. That means that the only difference is that `vgo` doesn't require lock file for reproducible builds, now the question is what's considered so terrible wrong/dangerous with having a lock file?
- iambvk 8y ago
  If there was a security fix in a package, MVS allows build infra to automatically upgrade (assuming packages imported are well maintained by their owners) where as lock-files require manual intervention. Imagine the packaging and development ecosystem in internet scale.
  - kibwen 8y ago
    I'm not sure what this is asserting. There's no association between MVS and build infra. If you have a security fix to apply, you'd run `vgo get -u` to upgrade all your modules; systems like Cargo that use lockfiles would run `cargo upgrade`. This is just as automatic. In fact, vgo is less automatic here because systems like Cargo allow users to decide when they want certain upgrades automatically, which means that new downstream users can get the security fix even without needing to explicitly upgrade. And vgo also requires more manual intervention to upgrade than traditional build systems, because if the upgrade requires a major version bump then you also have to edit the import statements in your source files as well. vgo is both less automatic and requires more manual labor; that appears to be an intentional choice on behalf of the vgo manifesto.

amasad 8y ago

There are a lot of suboptimal design decisions that's shared across the state of the art in package management (cargo, pipenv, npm, yarn, etc) that the Go team is not taking for granted. One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy (https://research.swtch.com/vgo-repro https://research.swtch.com/vgo-repro). Another is shared (or recursive) dependencies. The semver answer to this is that if the versions match then they should be shared if not then they should be duplicated. But what do you do if it's a singleton, listens on a port, or exposes a port. Right now, with npm for example, good luck with that, you're in for a world of pain. On the other hand with vgo's "semantic import versioning" they're trying to make version interop more explicit. Just because 90% of the time mainstream package mangers work doesn't mean it's a solved problem. Kudos for the Go team for trying to advance the state of the art.

kibwen 8y ago
> The standard answer is lock files which is extra bloat and leads to merge conflicts. Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?
- amasad 8y ago
  More things, more state = bloat. Don't you agree that if you can do something without holding state it's invariably simpler? That's what MVS does. There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100 https://github.com/rust-lang/cargo/issues/4100
  - kibwen 8y ago
    > Don't you agree that if you can do something without holding state it's invariably simpler? Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS: "If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/ https://sdboyer.io/vgo/intro/

mseepgood 8y ago

> I wish they had just copied Rust/Cargo Cargo is really a suboptimal design. With vgo Go advances the state of the art of dependency management.

kaeshiwaza 8y ago

I believe this video of rsc explain some concerns about the Cargo's approach and lock file issues https://www.youtube.com/watch?v=F8nrpe0XWRg https://www.youtube.com/watch?v=F8nrpe0XWRg