6 ms·

Rust/Cargo had the luxury of being a greenfield project that could adopt semver from the beginning, whereas Go made the mistake of starting out, and then going

Rust/Cargo had the luxury of being a greenfield project that could adopt semver from the beginning, whereas Go made the mistake of starting out, and then going years, without any official package management solution. As a result, Go has a swathe of applications and libraries that use specific workflows as well as a mélange of community-developed package management tools such as godep, Glide and dep. The semver standard, in particular, has been inconsistently adopted by the Go community. In other words, any new Go tool either has to support/import existing code, or to wipe the slate clean and say that for a package to be importable it has to follow a new spec. dep decided on the former, and my impression is that this has had unfortunate consequences, because that inherits a lot of historical baggage. We've been using dep for a while (having escaped the bugfest that is Glide, which used a very similar approach), and it's pretty evident that the solver is buggy and slow and also complicated enough that fixing issues like [1] can only be done by a select few that already understand the codebase. I'm not in a position to judge what the causes of all of these issues are, though I'd wager they're not entirely unrelated to the inherent complexity of SAT solving. The current dep issue tracker is full [2] of reports mentioning the solver, not to mention that dep currently has problems with known libraries such as the Kubernetes client [3] and Protobuf. (Google-related projects have historically used godep.) Again, possibly related to this specific implementation and not necessarily something that would apply to a hypothetical "Cargo for Go", but I don't know. Any idea how Cargo compares to dep overall? [1] https://github.com/golang/dep/issues/1306 https://github.com/golang/dep/issues/1306 — this one is a nightmare if you work anything related to Kubernetes. [2] https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+solver https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+... [3] https://github.com/golang/dep/issues/1207 https://github.com/golang/dep/issues/1207

pcwalton 8y ago
Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indeed the solver that is the problem (which, again, I'm skeptical of—I'd have to see profiling data to believe it), then it could just come down to optimization differences between rustc and Go 6g/8g.
- lobster_johnson 8y ago
  Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a086501d8f https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...
  - kibwen 8y ago
    That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dirkjan.ochtman.nl/files/servo-graph.svg https://dirkjan.ochtman.nl/files/servo-graph.svg ; the number of transitive dependencies is quite large and yet the runtime of version selection is negligible.
    - TheDong 8y ago
      It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation). This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io). If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep. I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.

pcwalton 8y ago

Rust was actually around for a good while before Cargo was adopted. (In fact, there were two Cargos, the older of which bore very little resemblance to the Cargo of today.) Of course, Go was stable for longer. I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indeed the solver that is the problem (which, again, I'm skeptical of—I'd have to see profiling data to believe it), then it could just come down to optimization differences between rustc and Go 6g/8g.

lobster_johnson 8y ago
Here [1] is the "dep ensure -v" output for a project of mine. It takes almost 12 seconds even when there are no changes to the actual file. I don't know why, or whether it's actually the solver (though the output seems to indicate it). [1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a086501d8f https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...
- kibwen 8y ago
  That's quite weird. When I run `rm Cargo.lock && cargo generate-lockfile` on the Servo repo (test performed on the cheapest VPS that money can buy) it exits near-instantly (after first spending three seconds trying to git-fetch new versions of the dozen custom dependencies that live on Github rather than crates.io). For reference, here's what Servo's dependency graph looked like two years ago (July 2016): https://dirkjan.ochtman.nl/files/servo-graph.svg https://dirkjan.ochtman.nl/files/servo-graph.svg ; the number of transitive dependencies is quite large and yet the runtime of version selection is negligible.
  - TheDong 8y ago
    It's not strange at all for go. Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies. It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation). This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io). If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep. I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.