18 ms·

Go += Package Versioning

336 points by bketelsen 9y ago

deleted 9y ago
[deleted]
eximius 9y ago
> Minimal Version Selection Means no security fixes at the price of, well, minor developer inconvenience? What is the inconvenience, exactly? > ...tomorrow the same sequence of commands you ran today would produce a different result. I mean, I guess this is technically true. But seems like it shouldn't be an issue in practice as the API you're calling shouldn't have changed, just the implementation. If it has changed, then downgrade the dependency?
- bbatha 9y ago
  These issues all seem to be solved by lockfiles, which are a part of the dep tool. Why ditch them here? It seems like rsc seems to be hinting at an argument for simplicity. But the long history of package managers, and my anecdotal experience confirms, that you're going to need a lock file no matter what. Good versioning practices are not enough, npm versions 1-4 prove that you need locking, and it must be on by default to make reproducible software. Minimum versioning will likely be better than the newest version for stability, but its not enough.
  - stormbeta 9y ago
    Not only that, but locking allows for what is IMO the best way to handle this through CI: Always run your builds with maximal version selection in CI (unlocked), and lock the new versions in if it passes all tests, otherwise leave the previous locked versions and flag it for review.
- binarycrusader 9y ago
  As a long-time maintainer of various packaging systems (and co-author of one): I find myself wholly in agreement with the idea that maximal version selection is appropriate for operating systems and general software components, but not necessarily desirable for a build system. When you consider the evaluation of a dependency graph in the context of a SAT solver, you realize that the solver would consider both a minimal and maximal version as potentially satisfying the general constraints found in a dependency graph. Whether you then optimize the solution for a minimal or maximal version becomes a matter of policy, not correctness. The security concern can be appropriately addressed by increasing the minimum version required in the appropriate places. With all of that said, I think that Go's potential decision to optimize for minimal version selection is likely to be considered a bug by many because it will not be the "accepted" behavior that most are accustomed to. In particular, I can already imagine a large number of developers adding a fix, expecting others to automatically pick it up in the next build, and being unpleasantly surprised when that doesn't happen. This is an interesting experiment at the least, but I hope they make the optimizing choice for minimal/maximal version configurable (although I doubt they will).
  - readams 9y ago
    We can only hope they'll at least have a tool that you can run that will bump up all the versions to specify the latest. But then of course that will make solving dependencies a lot harder since it's much more likely you'll get an unsolvable set of deps. It's hard to see how this could ever help anything. Lock files also have the major advantage that you have a hash of the thing you're downloading so you're not depending on the maintainer not e.g. force pushing something.
    - mook 9y ago
      Because only the minimum version can be specified, you will always get a solvable set of deps. Those deps might not build but they will be solvable.
  - munificent 9y ago
    > When you consider the evaluation of a dependency graph in the context of a SAT solver, you realize that the solver would consider both a minimal and maximal version as potentially satisfying the general constraints found in a dependency graph. Whether you then optimize the solution for a minimal or maximal version becomes a matter of policy, not correctness. I believe rsc is hoping they can avoid the need for a SAT solver entirely by going with minimum versions (and, implied, not allowing a newer package to be published with a lower version number).
    - uluyol 9y ago
      Actually, I think the SAT solver is avoided by making the only version constraints of the form <=. Using the min version appears to eschew the need for lock files. Want to upgrade? Bump your min version.
      - binarycrusader 9y ago
        One potential problem with dependency resolution that focuses solely on minimal or maximal-bound resolution is that it usually ignores the untested version combination problems. That is, a given version that satisfies a version bound may not have necessarily been tested with all of the different combinations of versions of components it is used with. It will be interesting to see how minimal version selection interacts with that particular challenge. For a build system it may matter much less than an operating system.
- wtetzner 9y ago
  > I mean, I guess this is technically true. But seems like it shouldn't be an issue in practice as the API you're calling shouldn't have changed, just the implementation. If it has changed, then downgrade the dependency? And if the new implementation has a new bug, you might be screwed. It worked last week, but not this week. How do I get back to the working version?
  - lsaferite 9y ago
    Use 'git bisect' or some similar tool to walk back and find the point where the lock file change triggered the bug?
    - wtetzner 9y ago
      If you have a lock file, there's no problem. I was arguing about what happens when there isn't a lock file. I may have inferred something that wasn't in the original comment, by reading too many of the other comments on this page.
      - makapuf 9y ago
        Sort but given that we're relying on git mostly, what are the advantages of lock files over submodules/subtrees ?
        wtetzner 9y ago
        Well, that would lock all of the projects into git. What if some libraries are in git, and others are in mercurial and darcs? I think it’s better to have a package manager that’s not reliant on a particular source control system.
Sir_Cmpwn 9y ago
>End of GOPATH Yes yes yes yes finally! This has been the #1 reason I've avoided Go for years and even my recent forays back into the language were only after I figured out some hacks that I could use to avoid it.
- jadeklerk 9y ago
  Because you didn't want to set an env var?
  - merb 9y ago
    GOPATH has a default. Unix: $HOME/go Windows: %USERPATH%/go but on ci servers it could be akward.
  - chimeracoder 9y ago
    > Because you didn't want to set an env var? You haven't needed to set a GOPATH since 1.8, which was released over a year ago (we're now at 1.10). Since 1.8, the Go toolchain will use a default GOPATH; the environment variable is only needed as an override.
  - lloeki 9y ago
    The problem is not the env var, because whether or not it was set or with a default value, you have to clone the project https://github.com/foo/bar.git https://github.com/foo/bar.git at ~/go/src/github.com/foo/bar and name the imports github.com/foo/bar accordingly and run the go commands from there. I used to (automatically) just export GOPATH="${PWD}/.gopath" but this doesn't work anymore since 1.8 or 1.9. People† at large just want to clone https://github.com/foo/bar.git https://github.com/foo/bar.git wherever they see fit, like directly at ~/Workspace/bar, ~/work/bar, ~/projects/contrib/bar or even /tmp/bar. † "People" includes CI systems that expect the typical "clone and cd straight into" thing to work, resulting in a lot of boilerplate and/or symlink hacks to work around such expectations.
- ovao 9y ago
  For what reasons did you need to avoid it?
deleted 9y ago
[deleted]
readams 9y ago
Damn it. Just when dep seemed like it was going to finally end the horrible nightmare that is go dependency management, this comes along. There's this great survey of all the progress that had been made and then "but we're doing it in this bizarre terrible way because..."
howeyc 9y ago
I like the removal of GOPATH, even though I don't mind it at all, it can be an annoyance, especially when I don't want to co-mingle work/personal stuff. Now I don't need to worry about setting GOPATH based on what I'm working on. Plus sometimes I'd wonder, is it all in vendor, or is it using GOPATH for something. Plus now libraries are "modules" so libraries can have dependencies for specific versions, before the question is what do you do if multiple dependencies have the same dependencies in their own separate vendor directories. This change removes that, as it's handled by the vgo tool for all modules in that build. go.mod is kind of a cross between lock-file and dependency listing. I think it will work alright. All together, it seems to be a cross between gb and dep, while also attempting to solve library-packages tracking dependencies too.
fiatjaf 9y ago
> we also must not remove the best parts of the current go command: its simplicity, speed, and understandability. Then you propose to introduce the concept of modules and make the world a hell? I believe versioning should be left of to third-party tools. For example, no one is complaining about the lack of versioning built in Node.js.
merb 9y ago
this looks damn complex.
steveklabnik 9y ago
Glad to see this! I'm still digesting the details, but to comment on https://research.swtch.com/cargo-newest.html https://research.swtch.com/cargo-newest.html Cargo does not use the latest version. Saying toml = "0.4.1" is the same as saying toml = "^0.4.1" NOT saying toml = "=0.4.1" which is what rsc would guess. This decision was made because ^ is the most reasonable default; over-use of `=` destroy's Cargo's ability to help you upgrade, given that you basically asked it to not think about it at all. Further, it would significantly bloat binaries, as this would mean many more duplicate versions of packages in your project. Basically, idiomatic use of = is only in very specific situations. We could have required that you always specify some operator, but we also like reasonable defaults. Writing `^` everywhere is just more typing. The transitive dependency issue isn't any different, really: that's due to said packages also not using =.
- rsc 9y ago
  Hi. As I said in that page, I do know that cargo is working as designed, and that "0.4.1" is the same as "^0.4.1". My point is maybe a little more subtle, that cargo takes the newest allowed under the constraints, so given "^0.4.1", it has a choice between 0.4.1, 0.4.2, 0.4.3, 0.4.4, and 0.4.5, and it takes the last. When you're adding a new direct dependency, taking the latest is almost certainly right. But when you're pulling in a new transitive dependency, I think taking the last is problematic: it seems much safer to me to take the one closest to what the author of that code tested with. I'll elaborate more on this in a post tomorrow. For what it's worth, I think I do understand what cargo is doing, and why, and I completely respect that approach. I just think this other approach might have some different properties worth exploring. As I've been working on this I've been using cargo as the gold standard, really. If the end of this all is that people say Go's package management is as nice as cargo, then I will be very very happy. A long way to go yet for sure.
  - steveklabnik 9y ago
    To be clear, I don't think you don't understand it; I'm afraid that the way that you've worded it means that others won't understand it. I also very much don't think you should cargo cult! As munificient said above, this is a Hard Problem, and there are a lot of options. In some senses, I'm glad you're not just copying things, as well, that's how we all learn.
Animats 9y ago
I have misgivings about all this version pinning. At first, it seems to make things easier. Programs don't break because some imported package changed. So it looks like a win. At first. Over time, though, version pinning builds up technical debt. You have software locked to old unmaintained versions of packages. If you later try to bring some package up to date, it can break the fragile lace of dependencies locked in through past version pin decisions. Some version pinning systems let you import multiple versions of the same package into the same build to accommodate dependencies on different version. That bloats executable code and may cause problems with multiple copies of the same data. On the other side, once package users are using version pinning, it's easier for package maintainers to make changes that aren't backwards compatible. This makes it harder for package users to upgrade to the new package and catch up. Three years after you start version pinning, it's going to be a major effort to clean up the mess. The Go crowd tries to avoid churn. Their language and libraries don't change too much. That's a good, practical decision. Go should reject version pinning as incompatible with the goals of Go.
- condescendence 9y ago
  Technical debt is definitely something to worry about, however I think most companies' first priority is a working product. Technical debt is to be handled during the development cycle, not in source code and most of the issues you bring up (bloated software, version pinning) are solved in build systems and again should not be handled by source code. Anyone working in an enterprise enviorment, especially that has clients on different systems, would instantly crumble without being able to target specfic builds. Legacy and long term support software is everything. >Go should reject version pinning as incompatible with the goals of Go. This would instantly make go a 'no-go' in any enterprise enviorment.
- rsc 9y ago
  We do reject version pinning. There is no way to pin a particular version. The only constraint you can express is ">= this specific version". If nothing else pushes it forward, though, you'll keep getting that specific version. But it's not pinning, it's just stating a minimum, and the system uses the single oldest version that satisfies all the stated minimums (the max of the mins).
pknopf 9y ago
Generics first please, I can live with using tools like vndr and deep.
- sctb 9y ago
  We've heard this one before, and uppercase doesn't summon it any more effectively. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - pknopf 9y ago
    I was being cheeky. Thank you for your attention.
- rsc 9y ago
  Sorry, maybe generics next.
  - pknopf 9y ago
    Thanks! Vendoring needs to be solved and unified, I get it. But to me, the biggest thorn is generics. I have ran into too many cases where generics would have made my life a lot easier.
leshow 9y ago
How does Go not have package versioning? What do you guys use to get the appropriate version of a dep?
- abhia 9y ago
  There's a bunch of external packages. One by the go team: https://github.com/golang/dep https://github.com/golang/dep
  - iends 9y ago
    None of the top 10 of contributors to dep are from the Go core team, dep is a community project that eventually clawed their way into official recognition by sheer force of will by some great people.
deleted 9y ago
[deleted]
munificent 9y ago
I'm going to comment mostly on the parts of the proposal that I think are wrong, but don't take this to be an overall negative response. I'm excited to see smart folks working on this, and package management is a really hard problem. There are no silver bullets to code reuse. Context for those who don't know: I along with Natalie Weizenbaum wrote pub[1], the package manager used for Dart. > Instead of concluding from Hyrum's law that semantic versioning is impossible, I conclude that builds should be careful to use exactly the same versions of each dependency that the author did, unless forced to do otherwise. That is, builds should default to being as reproducible as possible. Right on. Another way to state this is: Changing the version of a dependency should be an explicit user action, and not an implicit side effect of installing dependencies. import "github.com/go-yaml/yaml/v2" > Creating v2.0.0, which in semantic versioning denotes a major break, therefore creates a new package with a new import path, as required by import compatibility. Because each major version has a different import path, a given Go executable might contain one of each major version. This is expected and desirable. It keeps programs building and allows parts of a very large program to update from v1 to v2 independently. It took me several readings to realize that you encode the major version requirement both in the import string and in the module requirements. The former lets you have multiple copies of the "same" module in your app at different major versions. The latter lets you express more precise version requirements like "I need at least 2.3, not just 2.anything". I think it's really going to confuse users to have the major version in both places. What does it mean if I my code has: import "github.com/go-yaml/yaml/v2" But my go.mod has: require ( "github.com/go-yaml/yaml" v1.5.2 ) I don't know if the goal of this is to avoid lockfiles, or to allow multiple versions of the same package to co-exist, but I think it's going to end up a confusing solution that doesn't cleanly solve any problem. For what it's worth, Dart does not let you have two versions of the same package in your application, even different major versions. This restriction does cause real pain, but it doesn't appear to be insurmountable. Most of the pain seems to be in the performance issues over-constrained dependencies caused in our old version solver and not in the user's code itself. In almost all cases, I think there is a single version of a given package that would work in practice, and I think it's confusing for users to have an application that has multiple versions of what they think of as the "same" package inside it. This may be less of an issue in Go because it's structurally typed, but in Dart you could get weird errors like "Expected a Foo but got a Foo" because those "Foo"s are actually from different versions of "foo". Requiring a single version avoids that. > I believe this is the wrong default, for two important reasons. First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. Maybe tonight someone will introduce a new version of some dependency, and then tomorrow the same sequence of commands you ran today would produce a different result. No, I think newest (stable version) is the right default. Every package manager in the world works this way and the odds that they all got this wrong are slim at this point. At the point in time that the user is explicitly choosing to mess with their dependencies, picking the current state of the art right then is likely what the user wants. If I'm starting a brand new from scratch Ruby on Rails application today, in 2017, there is no reason it should default to having me use Rails 1.0 from 2005. Every version of the package is new to me because I'm changing my dependencies right now. Might as well give me the version that gets me as up-to-date as possible because once I start building on top of it, it gets increasingly hard to change it. Encouraging me to build my app in terms of an API that may already be quite out of date seems perverse. > This proposal takes a different approach, which I call minimal version selection. It defaults to using the oldest allowed version of every package involved in the build. This decision does not change from today to tomorrow, because no older version will be published. I think this is confusing older versions and lower. You could, I suppose, build a package manager that forbids publishing a version number lower than any previously published version of the package and thus declare this to be true by fiat. But, in practice, I don't think most package managers do this. In particular, it's fairly common for a package to have multiple simultaneously supported major or minor versions. For example, Python supports both the 2.x and 3.x lines. 2.7 was released two years after 3.0. When a security issue is found in a package, it's common to see point releases get released for older major/minor versions. So if foo has 1.1.0 and 1.2.0 out today and a security bug that affects both is found, the maintainers will likely release 1.1.1 and 1.2.1. This means 1.1.1 is released later than 1.2.0. I think preferring minimum versions also has negative practical consequences. Package maintainers have an easier job if most of their users are on similar, recent versions of the package's own dependencies. It's no fun getting bug reports from users who are using your code with ancient versions of its dependencies. As a maintainer, you're spending most of your time ensuring your code still works with the latest so have your users in a different universe makes it harder to be in sync with them. Look at, for example, how much more painful Android development is compared to iOS because Android has such a longer tail of versions still in the wild that app developers need to deal with. If you do minimum version selection, my hunch is that package maintainers will just constantly ship new versions of their packages that bump the minimum dependencies to forcibly drag their users forword. Or they'll simply state that they don't support older versions beyond some point in time even when the package's own manifest states that it technically does. There is a real fundamental tension here. Users — once they have their app working — generally want stability and reproducibility. No surprises when they aren't opting into them. But the maintainers of the packages those users rely on want all of their users in the same bucket on the latest and greatest, not smeared out over a long list of configurations to support. A good package manager will balance those competing aims to foster a healthy ecosystem, not just pick one or the other. [1]: https://pub.dartlang.org/ https://pub.dartlang.org/
tetraodonpuffer 9y ago
How would you specify dependencies on a commit/tag in a particular branch (say, during development I always want to build with the HEAD of the development branch for this module)? I know I could clone what I want somewhere and use a replace directive inside the go.mod file, but I was wondering if there's any way to do this using only vgo.
- teacpde 9y ago
  To name untagged commits, the pseudo-version v0.0.0- yyyymmddhhmmss-commit identifies a specific commit made on the given date.
marcrosoft 9y ago
Please don't remove vendoring...
- LaGrange 9y ago
  Yeah, this. You want reproducible builds? Vendor your crap. Check it into your repository. Live free.
  - hota_mazi 9y ago
    Or adopt a system with a sane versioning system and immutable artifacts, e.g. Maven Central.
    - merb 9y ago
      which still means that your ci needs to download this stuff. with vendor it is already checked it. consider builds on docker etc, where build caching is hard. go is a breeze. build takes like half a second because everything is already there. mvn will first download the world. I'm a scala developer and sbt will probably take like 10 minutes just downloading stuff, if a cache missed (even with a local proxy).
      - pjmlp 9y ago
        I wonder how it is possible to take 10 minutes, unless it is downloading directly from internet. We always have internal Plexus mirrors, and the Jenkins servers have their global .m2 local repository. Even big JEE applications barely take more than 5 minutes to build. The only build system I really dislike are Android builds with Gradle, trying to beat C++ compilation times.
        merb 9y ago
        well sbt uses ivy which is way slower than maven, when it comes to resolving stuff.
    - LaGrange 9y ago
deleted 9y ago
[deleted]
baq 9y ago
this sentence > First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. is the root of what's wrong with this proposal - namely conflating time of dependency resolution with the time of build. cargo (using it because it's been referenced by the post) solves this problem by telling you to check in the cargo.lock that it generates whenever you 'cargo update'. this means that your version control preserves the build dependencies, just as it preserves build recipes (e.g. build.rs or travis or whatever). in this world, there's no 'newest allowed version', because there is only one version. wise men before me said 'as simple as possible, but no simpler' and I'm afraid this proposal is too simple.
Keats 9y ago
Will there be an official proxy or will open-source projects still break when someone moves/deletes a repository?
- icholy 9y ago
  That's what local caches are for.
  - Keats 9y ago
    Sure but me cloning the repository of someone else won't give me that person cache. Without a central cache, it might work on that person computer but not on mine if a repository moved.
cientifico 9y ago
Please don't.
glasser 9y ago
Is the "import compatibility rule" (ie, `/v2` suffixes in package paths) implemented in vgo today?
teacpde 9y ago
If there is neither GOPATH nor vendor directories, where will the downloaded packages be? I digged vgo a bit and found the downloaded packages currently reside in `$GOPATH/src/v/cache`, and seems like vgo won't work without GOPATH for now (https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a896833851a9e54a9/vendor/cmd/go/internal/vgo/init.go#L111 https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a8...).
hota_mazi 9y ago
Amazing that in 2018, a language team needs to be convinced that versioning packages is a good idea. What's next, persuading them that generics are pretty cool too?
- ovao 9y ago
  Considering there are existing and official dependency management tools for Go, the Go team doesn't need to be convinced that versioning is a good idea. This article is a discussion about mainlining a dependency management system as part of the existing `go` tool. They're similarly aware that generics are a desirable thing to have, but haven't determined the ideal way to implement generics into the language.
- dsymonds 9y ago
  Did you actually read the article? It is clear from the background and history sections that the need for versioning was apparent from early on.
shkkmo 9y ago
I think that the choice to use "prefer minimum version by default" going to confuse a lot of new developers used to the industry standard being the opposite. If you are going against the industry default, there should be a strong reasons. I don't think reasons provided really justify this. > First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. Maybe tonight someone will introduce a new version of some dependency, and then tomorrow the same sequence of commands you ran today would produce a different result. This is why we have lock files and they work better for this. While vgo's approach does allow reproducible builds, it doesn't allow you to guarantee reproducibility the way that a lock file that has the specific commit hashes does. With specific commit hashes you can verify the the version you are building with in production is the exact same code as the one your security team audited prior to release (assuming git has fixed its hash collision vulernability). You can get around this by abandoning version constraints in your go.mod file, but then you have to track these version constraints out of band and manually figure out what commit hash to stick in go.mod You could also get around this by storing the hases for the approved versions out of band and creating a build script that verifies these hashes prior to building. Both of these workarounds seem to defeat the point of having a standard package manager in the first point. > Second, to override this default, developers spend their time telling the package manager “no, don't use X,” and then the package manager spends its time searching for a way not to use X. If you are concerned with allowing users to override this default, why not have directive to override this default that can optionally be added to each requirement in the go.mod file. This avoids an unexpected default and doesn't force people who want the industry standard default to use a script to set which dependencies use or don't use the -u flag with 'go get'.
- Zamicol 9y ago
  >With specific commit hashes you can verify the the version you are building with in production is the exact same code as the one your security team audited prior to release (assuming git has fixed its hash collision vulnerability) Can we take a moment and point out that the hash collision vulnerability is still at large with no ETA? This is after years of it being considered insecure I feel like this point continues to be glossed over with versioning systems that depend on git commit hashes. Bruce Schneier warned in February 2005 that SHA needed to be replaced. (https://www.schneier.com/blog/archives/2005/02/cryptanalysis_o.html https://www.schneier.com/blog/archives/2005/02/cryptanalysis...) Git development didn't start until April 2005. So before git was even developed, SHA was identified as needing to be deprecated. So now, 13 years later, this is still an issue.
mCOLlSVIxp6c 9y ago
I predict that under the proposed minimum version selection system some package will decide it's important and flawless enough that users should always depend on the most recent version. The package will recommend that users depend on version v1.0.0 but the first release will be v1.100.0. Subsequent releases will decrement the minor version to make sure newer releases are selected.
- readams 9y ago
  It certainly complicates the picture. I think the only way to make this workable is if you adopt a convention that you specify your minimum dependency if you're a library, but the current version if you're an executable. Of course, this could just be codified in the tool, which would be a big improvement. It also makes testing a lot harder though since your library ends up tested against a different version than what your users are using. Ugh.
- skybrian 9y ago
  Interesting loophole, but it requires the community to go along with it. If it turns out people like the minimum version policy and the package is actually important, I would expect someone to fork the repo and do things the normal way?
- overcyn 9y ago
  User's wouldn't be able to depend on version v1.0.0 if it doesn't exist. From what I understand the tool isn't downloading the list of all versions from GitHub and selecting the minimum. Its searching for the maximum version number (i.e. minimal required version), from your go.mod file and all your dependencies' go.mod file.
  - mCOLlSVIxp6c 9y ago
    I think you're correct. So my nasty little hack wouldn't work. That's probably a good thing.
teacpde 9y ago
On one hand, as a developer, I like a Go builtin package management authored and supported by the Go team; on the other hand, I feel bad for the open source contributors who worked hard on third party package managers. Even dep, the officially supported PDM, seems supposed to fade.
- remus 9y ago
  > Even dep, the officially supported PDM, seems supposed to fade. I beleve this was always the plan. The dep readme certainly makes it sound that way at least: > dep is a prototype dependency management tool for Go. It requires Go 1.8 or newer to compile. dep is safe for production use. > dep is the official experiment, but not yet the official tool. Check out the Roadmap for more on what this means!
platz 9y ago
> A build of a module by itself will always use the specific versions of required dependencies listed in the go.mod file. As part of a larger build, it will only use a newer version if something else in the build requires it. This seems fraught
- icholy 9y ago
  I think you need a noun to go with that adjective.
  - platz 9y ago
    With danger
readams 9y ago
One that that I'm not able to tell from the proposal: is the /v1 at the end of an import path a magical value that the tool handles, or do I literally need to have that directory in my repo and have a version history represented at the top level of every project? Why not do the symbol mangling in the compiler if the goal is to have multiple simultaneous versions built in? This is easy to make backward compatible since it could default to v1.
pcj128 9y ago
Just use Bazel to build your go code. It already solved this problem.
pcj128 9y ago
Just use Bazel to build your go code. It has already solved this problem.
mbertschler 9y ago
Am I totally missing something in this document or does it not talk about where these modules either as zips or source code are stored? It mentions $GOPATH and vendor as not necessary anymore. Where will the actual code be?
j_bond 9y ago
I am curious why go doesn't just vendor dependencies like npm does when confronted with conflicting requirements instead of trying to figure out a version that satisfies all worlds? I always found this to be a nice feature when working with npm. I am also firm believer in version-pinning/lockfiles. Updating versions should be a separate workflow with first-class built-in tools to support it. I think that is the area where most package managers fall flat. They basically rely on the developer to do all the heavy lifting.
- TheDong 9y ago
  nodejs's "require", which pulls different versions depending on which module calls it, is a cool trick. Unfortunately, it also has a downside, and in go that downside would be noticeable. Let's take one easy example: a logging library. Let's say I pull in "logrus v1.0.1" and one of my dependencies pulls in "logrus v1.0.2". In my "main" function, I set logrus's default loglevel to debug (logrus.SetLevel(logrus.DebugLevel)). If go did the thing nodejs does (a different copy of logrus for my dependency than my main), the "logrus.SetLevel" would only affect my package, not any of my dependencies; they'd still log at a default level. This would be true of other things; "func init" code would run once per dependency that had that library, not just once, maps wouldn't be shared, pools, etc. This is a lot more memory usage, but it's also really surprising, especially in the case of logging. I definitely prefer having only one copy of a library in memory and having package-level variables (like loglevel) work as expected. If that ends up failing and I need two different versions, vendor+import-path rewriting allows an escape hatch These days, npm actually tries to minimize and flatten dependency versions as much as possible to avoid the huge memory tax.
mfer 9y ago
As I look this over a couple things really jump out at me... 1. VCS tags are mutable. That's lock files store revision ids. Go is being used to build immutable infrastructure but the proposed package management system uses mutable versions. 2. The proposal is less featureful that dep, npm/yarn in JS, composer in PHP, maven, crates for rust, and others. I wonder how people will react to that.
- Groxx 9y ago
  Mutable tags are my primary concern here, yeah. It seems pretty mitigate-able by using that (wonderful IMO[1]) `v1.2.3-date-sha` syntax though - it's just not human editable[2]. [1]: it fixes so many readability problems with SHA-pinned lock files, easily shows downgrades in `diff` output, and `sort` likely produces the exact result you wanted. [2]: which may not be a problem, since you could in theory just re-run the tool to fix it when you enter "v1.2.3" by hand.
xrstf 9y ago
A lot of this seems to hinge on "go get" being the best way to get Go source code. Most discussions I've read so far come down to "well, this might be cool, but it would break go get, so we cannot ever do it". I for one could live without go get alltogether. In 99% of cases it boils down to a simple git clone anyway, which could (should?) be done by the package manager anyway. Maybe it's time to re-evaluate the existence of go get, now that we're seeing that "just clone master and hope nothing breaks" has obviously not worked for everyone outside of Google? Maybe bundling Go (the compiler and linker) with tools for fetching source code wasn't such a good idea after all? Just my 2 cents...
- thefounder 9y ago
  go get, go install and go build are cool and great tools! Let's not turn them into a maven/pom/npm crap!
- acdha 9y ago
  I find it bizarre how unnecessarily hard that’s been made to avoid a slight amount of extra typing. The github imports work with the most basic options but I’ve seen people waste a lot of time on internal servers, SSH vs. HTTPS, tagging, etc. where it’d have been cleaner, easier and forward-compatible if you just gave it a URL and it shelled out to run “git clone <URL>”.
stock_toaster 9y ago
I feel like the Go devs optimized implementation simplicity early on (likely a sensible choice given Google's "one giant repo" workflow), and that has resulted in some not-insignificant ecosystem complexity later (for others not using "one giant repo" workflows). It will be interesting to see how this plays out in the longer term.
icholy 9y ago
Like all Go things, it looks kinda strange. But after playing around for a bit, it feels really good. The minimal version selection simplicity is genius.
gkya 9y ago
Highly suggest to read the demo at https://research.swtch.com/vgo-tour https://research.swtch.com/vgo-tour, which really clarifies everything. I do not write Go, but I did write some bits in it in the past, and now think that this will be a very useful addition to the Go toolchain. When I looked at Go for the first time I really disliked the simple but sort-of castrated go-build, and didn't really follow the developments since then, but this seems to solve quite a bit of problems. I especially like how you can shadow a certain dependency with a given package at a given path, and how it facilitates vendoring. Hope this becomes included in go soon.
013a 9y ago
This is a well reasoned proposal. I've got a couple qualms. Let's say you care about strict build reproducibility. You want some form of manifest + lock file that defines exactly what versions of dependencies your package expects. Great; more power to you. The only reason you'd want that is if you aren't storing your dependencies with your source code in source control. Otherwise, what's the point? If you were vendoring and committing it, you already have strict reproducibility, and you have the "supported versions" defined in the git repositories that come alongside your dependencies. So, adding this manifest+lock file allows you to get away with not vendoring+committing. Awesome. You've gained reproducibility, but certainly not strict reproducibility. Dependency package maintainers can rewrite git history. They can force push. They can push breaking changes with minor versions. They can delete their repositories. All of these things have already happened in NPM, either maliciously or accidentally; why do we think they wouldn't happen with Go? If you want strict reproducibility but are expecting it with just a manifest+lock file, without dependency vendoring+committing, you're not getting it. Full stop. So, really, by adding a manifest+lock file, you're adding a "third level" of reproducibility (#2 on this list). 1. Low Reproducibility: Pull HEAD on build. 2. Partial Reproducibility: Use tooling to pin to a git commitish, pull this on build. 3a. Full Reproducibility (Easy+Dirty): Vendor+Commit all dependencies. 3b. Full Reproducibility (Hard+Clean): Mirror your dependencies into a new self-controlled git repo. I am struggling to think of a solid use case that would find real value in #2. I have no doubt that people think it would be valuable, but then the moment the left-pad author deletes his repository, you're going to second guess yourself. You didn't want #2; you wanted #3 and settled for #2 because #3 was too hard or dirty and you convinced yourself that adding tooling and version numbers was keeping you safe. Because semver, right? Moreover, this comes at the cost of complexity, which is strictly against Go's core doctrine. Moreover, a #2 solution looks strikingly similar to gopkg.in. Instead of referencing Repository:HEAD we reference Repository:Commitish. The main difference is that gopkg.in is an external service whereas this would be controlled tooling. But by choosing #2 you're already exposing yourself to bad actors, accidents, github going down, all of the above. So you're willing to accept that exposure, but aren't willing to add that little extra exposure of one more service? I agree that the problem today, even with dep, isn't perfect. But I still strongly believe that the answer lies somewhere in the ideas we already have, not by importing ideas from NPM. We need a #3 that is Easy and Clean, not a #2.
codedokode 9y ago
> This proposal ... deprecates GOPATH in favor of a project-based workflow That would be awesome. GOPATH was an awful idea that works only for distribution package managers.

336 points by bketelsen 9y ago

deleted 9y ago

[deleted]

> Minimal Version Selection Means no security fixes at the price of, well, minor developer inconvenience? What is the inconvenience, exactly? > ...tomorrow the same sequence of commands you ran today would produce a different result. I mean, I guess this is technically true. But seems like it shouldn't be an issue in practice as the API you're calling shouldn't have changed, just the implementation. If it has changed, then downgrade the dependency?

bbatha 9y ago
These issues all seem to be solved by lockfiles, which are a part of the dep tool. Why ditch them here? It seems like rsc seems to be hinting at an argument for simplicity. But the long history of package managers, and my anecdotal experience confirms, that you're going to need a lock file no matter what. Good versioning practices are not enough, npm versions 1-4 prove that you need locking, and it must be on by default to make reproducible software. Minimum versioning will likely be better than the newest version for stability, but its not enough.
- stormbeta 9y ago
  Not only that, but locking allows for what is IMO the best way to handle this through CI: Always run your builds with maximal version selection in CI (unlocked), and lock the new versions in if it passes all tests, otherwise leave the previous locked versions and flag it for review.
binarycrusader 9y ago
As a long-time maintainer of various packaging systems (and co-author of one): I find myself wholly in agreement with the idea that maximal version selection is appropriate for operating systems and general software components, but not necessarily desirable for a build system. When you consider the evaluation of a dependency graph in the context of a SAT solver, you realize that the solver would consider both a minimal and maximal version as potentially satisfying the general constraints found in a dependency graph. Whether you then optimize the solution for a minimal or maximal version becomes a matter of policy, not correctness. The security concern can be appropriately addressed by increasing the minimum version required in the appropriate places. With all of that said, I think that Go's potential decision to optimize for minimal version selection is likely to be considered a bug by many because it will not be the "accepted" behavior that most are accustomed to. In particular, I can already imagine a large number of developers adding a fix, expecting others to automatically pick it up in the next build, and being unpleasantly surprised when that doesn't happen. This is an interesting experiment at the least, but I hope they make the optimizing choice for minimal/maximal version configurable (although I doubt they will).
- readams 9y ago
  We can only hope they'll at least have a tool that you can run that will bump up all the versions to specify the latest. But then of course that will make solving dependencies a lot harder since it's much more likely you'll get an unsolvable set of deps. It's hard to see how this could ever help anything. Lock files also have the major advantage that you have a hash of the thing you're downloading so you're not depending on the maintainer not e.g. force pushing something.
  - mook 9y ago
    Because only the minimum version can be specified, you will always get a solvable set of deps. Those deps might not build but they will be solvable.
- munificent 9y ago
  > When you consider the evaluation of a dependency graph in the context of a SAT solver, you realize that the solver would consider both a minimal and maximal version as potentially satisfying the general constraints found in a dependency graph. Whether you then optimize the solution for a minimal or maximal version becomes a matter of policy, not correctness. I believe rsc is hoping they can avoid the need for a SAT solver entirely by going with minimum versions (and, implied, not allowing a newer package to be published with a lower version number).
  - uluyol 9y ago
    Actually, I think the SAT solver is avoided by making the only version constraints of the form <=. Using the min version appears to eschew the need for lock files. Want to upgrade? Bump your min version.
    - binarycrusader 9y ago
      One potential problem with dependency resolution that focuses solely on minimal or maximal-bound resolution is that it usually ignores the untested version combination problems. That is, a given version that satisfies a version bound may not have necessarily been tested with all of the different combinations of versions of components it is used with. It will be interesting to see how minimal version selection interacts with that particular challenge. For a build system it may matter much less than an operating system.
wtetzner 9y ago
> I mean, I guess this is technically true. But seems like it shouldn't be an issue in practice as the API you're calling shouldn't have changed, just the implementation. If it has changed, then downgrade the dependency? And if the new implementation has a new bug, you might be screwed. It worked last week, but not this week. How do I get back to the working version?
- lsaferite 9y ago
  Use 'git bisect' or some similar tool to walk back and find the point where the lock file change triggered the bug?
  - wtetzner 9y ago
    If you have a lock file, there's no problem. I was arguing about what happens when there isn't a lock file. I may have inferred something that wasn't in the original comment, by reading too many of the other comments on this page.
    - makapuf 9y ago
      Sort but given that we're relying on git mostly, what are the advantages of lock files over submodules/subtrees ?
      - wtetzner 9y ago
        Well, that would lock all of the projects into git. What if some libraries are in git, and others are in mercurial and darcs? I think it’s better to have a package manager that’s not reliant on a particular source control system.

Sir_Cmpwn 9y ago

>End of GOPATH Yes yes yes yes finally! This has been the #1 reason I've avoided Go for years and even my recent forays back into the language were only after I figured out some hacks that I could use to avoid it.

jadeklerk 9y ago
Because you didn't want to set an env var?
- merb 9y ago
  GOPATH has a default. Unix: $HOME/go Windows: %USERPATH%/go but on ci servers it could be akward.
- chimeracoder 9y ago
  > Because you didn't want to set an env var? You haven't needed to set a GOPATH since 1.8, which was released over a year ago (we're now at 1.10). Since 1.8, the Go toolchain will use a default GOPATH; the environment variable is only needed as an override.
- lloeki 9y ago
  The problem is not the env var, because whether or not it was set or with a default value, you have to clone the project https://github.com/foo/bar.git https://github.com/foo/bar.git at ~/go/src/github.com/foo/bar and name the imports github.com/foo/bar accordingly and run the go commands from there. I used to (automatically) just export GOPATH="${PWD}/.gopath" but this doesn't work anymore since 1.8 or 1.9. People† at large just want to clone https://github.com/foo/bar.git https://github.com/foo/bar.git wherever they see fit, like directly at ~/Workspace/bar, ~/work/bar, ~/projects/contrib/bar or even /tmp/bar. † "People" includes CI systems that expect the typical "clone and cd straight into" thing to work, resulting in a lot of boilerplate and/or symlink hacks to work around such expectations.
ovao 9y ago
For what reasons did you need to avoid it?

deleted 9y ago

[deleted]

readams 9y ago

Damn it. Just when dep seemed like it was going to finally end the horrible nightmare that is go dependency management, this comes along. There's this great survey of all the progress that had been made and then "but we're doing it in this bizarre terrible way because..."

howeyc 9y ago

I like the removal of GOPATH, even though I don't mind it at all, it can be an annoyance, especially when I don't want to co-mingle work/personal stuff. Now I don't need to worry about setting GOPATH based on what I'm working on. Plus sometimes I'd wonder, is it all in vendor, or is it using GOPATH for something. Plus now libraries are "modules" so libraries can have dependencies for specific versions, before the question is what do you do if multiple dependencies have the same dependencies in their own separate vendor directories. This change removes that, as it's handled by the vgo tool for all modules in that build. go.mod is kind of a cross between lock-file and dependency listing. I think it will work alright. All together, it seems to be a cross between gb and dep, while also attempting to solve library-packages tracking dependencies too.

fiatjaf 9y ago

> we also must not remove the best parts of the current go command: its simplicity, speed, and understandability. Then you propose to introduce the concept of modules and make the world a hell? I believe versioning should be left of to third-party tools. For example, no one is complaining about the lack of versioning built in Node.js.

merb 9y ago

this looks damn complex.

steveklabnik 9y ago

Glad to see this! I'm still digesting the details, but to comment on https://research.swtch.com/cargo-newest.html https://research.swtch.com/cargo-newest.html Cargo does not use the latest version. Saying toml = "0.4.1" is the same as saying toml = "^0.4.1" NOT saying toml = "=0.4.1" which is what rsc would guess. This decision was made because ^ is the most reasonable default; over-use of `=` destroy's Cargo's ability to help you upgrade, given that you basically asked it to not think about it at all. Further, it would significantly bloat binaries, as this would mean many more duplicate versions of packages in your project. Basically, idiomatic use of = is only in very specific situations. We could have required that you always specify some operator, but we also like reasonable defaults. Writing `^` everywhere is just more typing. The transitive dependency issue isn't any different, really: that's due to said packages also not using =.

rsc 9y ago
Hi. As I said in that page, I do know that cargo is working as designed, and that "0.4.1" is the same as "^0.4.1". My point is maybe a little more subtle, that cargo takes the newest allowed under the constraints, so given "^0.4.1", it has a choice between 0.4.1, 0.4.2, 0.4.3, 0.4.4, and 0.4.5, and it takes the last. When you're adding a new direct dependency, taking the latest is almost certainly right. But when you're pulling in a new transitive dependency, I think taking the last is problematic: it seems much safer to me to take the one closest to what the author of that code tested with. I'll elaborate more on this in a post tomorrow. For what it's worth, I think I do understand what cargo is doing, and why, and I completely respect that approach. I just think this other approach might have some different properties worth exploring. As I've been working on this I've been using cargo as the gold standard, really. If the end of this all is that people say Go's package management is as nice as cargo, then I will be very very happy. A long way to go yet for sure.
- steveklabnik 9y ago
  To be clear, I don't think you don't understand it; I'm afraid that the way that you've worded it means that others won't understand it. I also very much don't think you should cargo cult! As munificient said above, this is a Hard Problem, and there are a lot of options. In some senses, I'm glad you're not just copying things, as well, that's how we all learn.

Animats 9y ago

I have misgivings about all this version pinning. At first, it seems to make things easier. Programs don't break because some imported package changed. So it looks like a win. At first. Over time, though, version pinning builds up technical debt. You have software locked to old unmaintained versions of packages. If you later try to bring some package up to date, it can break the fragile lace of dependencies locked in through past version pin decisions. Some version pinning systems let you import multiple versions of the same package into the same build to accommodate dependencies on different version. That bloats executable code and may cause problems with multiple copies of the same data. On the other side, once package users are using version pinning, it's easier for package maintainers to make changes that aren't backwards compatible. This makes it harder for package users to upgrade to the new package and catch up. Three years after you start version pinning, it's going to be a major effort to clean up the mess. The Go crowd tries to avoid churn. Their language and libraries don't change too much. That's a good, practical decision. Go should reject version pinning as incompatible with the goals of Go.

condescendence 9y ago
Technical debt is definitely something to worry about, however I think most companies' first priority is a working product. Technical debt is to be handled during the development cycle, not in source code and most of the issues you bring up (bloated software, version pinning) are solved in build systems and again should not be handled by source code. Anyone working in an enterprise enviorment, especially that has clients on different systems, would instantly crumble without being able to target specfic builds. Legacy and long term support software is everything. >Go should reject version pinning as incompatible with the goals of Go. This would instantly make go a 'no-go' in any enterprise enviorment.
rsc 9y ago
We do reject version pinning. There is no way to pin a particular version. The only constraint you can express is ">= this specific version". If nothing else pushes it forward, though, you'll keep getting that specific version. But it's not pinning, it's just stating a minimum, and the system uses the single oldest version that satisfies all the stated minimums (the max of the mins).

pknopf 9y ago

Generics first please, I can live with using tools like vndr and deep.

sctb 9y ago
We've heard this one before, and uppercase doesn't summon it any more effectively. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- pknopf 9y ago
  I was being cheeky. Thank you for your attention.
rsc 9y ago
Sorry, maybe generics next.
- pknopf 9y ago
  Thanks! Vendoring needs to be solved and unified, I get it. But to me, the biggest thorn is generics. I have ran into too many cases where generics would have made my life a lot easier.

leshow 9y ago

How does Go not have package versioning? What do you guys use to get the appropriate version of a dep?

abhia 9y ago
There's a bunch of external packages. One by the go team: https://github.com/golang/dep https://github.com/golang/dep
- iends 9y ago
  None of the top 10 of contributors to dep are from the Go core team, dep is a community project that eventually clawed their way into official recognition by sheer force of will by some great people.

deleted 9y ago

[deleted]

munificent 9y ago

I'm going to comment mostly on the parts of the proposal that I think are wrong, but don't take this to be an overall negative response. I'm excited to see smart folks working on this, and package management is a really hard problem. There are no silver bullets to code reuse. Context for those who don't know: I along with Natalie Weizenbaum wrote pub[1], the package manager used for Dart. > Instead of concluding from Hyrum's law that semantic versioning is impossible, I conclude that builds should be careful to use exactly the same versions of each dependency that the author did, unless forced to do otherwise. That is, builds should default to being as reproducible as possible. Right on. Another way to state this is: Changing the version of a dependency should be an explicit user action, and not an implicit side effect of installing dependencies. import "github.com/go-yaml/yaml/v2" > Creating v2.0.0, which in semantic versioning denotes a major break, therefore creates a new package with a new import path, as required by import compatibility. Because each major version has a different import path, a given Go executable might contain one of each major version. This is expected and desirable. It keeps programs building and allows parts of a very large program to update from v1 to v2 independently. It took me several readings to realize that you encode the major version requirement both in the import string and in the module requirements. The former lets you have multiple copies of the "same" module in your app at different major versions. The latter lets you express more precise version requirements like "I need at least 2.3, not just 2.anything". I think it's really going to confuse users to have the major version in both places. What does it mean if I my code has: import "github.com/go-yaml/yaml/v2" But my go.mod has: require ( "github.com/go-yaml/yaml" v1.5.2 ) I don't know if the goal of this is to avoid lockfiles, or to allow multiple versions of the same package to co-exist, but I think it's going to end up a confusing solution that doesn't cleanly solve any problem. For what it's worth, Dart does not let you have two versions of the same package in your application, even different major versions. This restriction does cause real pain, but it doesn't appear to be insurmountable. Most of the pain seems to be in the performance issues over-constrained dependencies caused in our old version solver and not in the user's code itself. In almost all cases, I think there is a single version of a given package that would work in practice, and I think it's confusing for users to have an application that has multiple versions of what they think of as the "same" package inside it. This may be less of an issue in Go because it's structurally typed, but in Dart you could get weird errors like "Expected a Foo but got a Foo" because those "Foo"s are actually from different versions of "foo". Requiring a single version avoids that. > I believe this is the wrong default, for two important reasons. First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. Maybe tonight someone will introduce a new version of some dependency, and then tomorrow the same sequence of commands you ran today would produce a different result. No, I think newest (stable version) is the right default. Every package manager in the world works this way and the odds that they all got this wrong are slim at this point. At the point in time that the user is explicitly choosing to mess with their dependencies, picking the current state of the art right then is likely what the user wants. If I'm starting a brand new from scratch Ruby on Rails application today, in 2017, there is no reason it should default to having me use Rails 1.0 from 2005. Every version of the package is new to me because I'm changing my dependencies right now. Might as well give me the version that gets me as up-to-date as possible because once I start building on top of it, it gets increasingly hard to change it. Encouraging me to build my app in terms of an API that may already be quite out of date seems perverse. > This proposal takes a different approach, which I call minimal version selection. It defaults to using the oldest allowed version of every package involved in the build. This decision does not change from today to tomorrow, because no older version will be published. I think this is confusing older versions and lower. You could, I suppose, build a package manager that forbids publishing a version number lower than any previously published version of the package and thus declare this to be true by fiat. But, in practice, I don't think most package managers do this. In particular, it's fairly common for a package to have multiple simultaneously supported major or minor versions. For example, Python supports both the 2.x and 3.x lines. 2.7 was released two years after 3.0. When a security issue is found in a package, it's common to see point releases get released for older major/minor versions. So if foo has 1.1.0 and 1.2.0 out today and a security bug that affects both is found, the maintainers will likely release 1.1.1 and 1.2.1. This means 1.1.1 is released later than 1.2.0. I think preferring minimum versions also has negative practical consequences. Package maintainers have an easier job if most of their users are on similar, recent versions of the package's own dependencies. It's no fun getting bug reports from users who are using your code with ancient versions of its dependencies. As a maintainer, you're spending most of your time ensuring your code still works with the latest so have your users in a different universe makes it harder to be in sync with them. Look at, for example, how much more painful Android development is compared to iOS because Android has such a longer tail of versions still in the wild that app developers need to deal with. If you do minimum version selection, my hunch is that package maintainers will just constantly ship new versions of their packages that bump the minimum dependencies to forcibly drag their users forword. Or they'll simply state that they don't support older versions beyond some point in time even when the package's own manifest states that it technically does. There is a real fundamental tension here. Users — once they have their app working — generally want stability and reproducibility. No surprises when they aren't opting into them. But the maintainers of the packages those users rely on want all of their users in the same bucket on the latest and greatest, not smeared out over a long list of configurations to support. A good package manager will balance those competing aims to foster a healthy ecosystem, not just pick one or the other. [1]: https://pub.dartlang.org/ https://pub.dartlang.org/

tetraodonpuffer 9y ago

How would you specify dependencies on a commit/tag in a particular branch (say, during development I always want to build with the HEAD of the development branch for this module)? I know I could clone what I want somewhere and use a replace directive inside the go.mod file, but I was wondering if there's any way to do this using only vgo.

teacpde 9y ago
To name untagged commits, the pseudo-version v0.0.0- yyyymmddhhmmss-commit identifies a specific commit made on the given date.

marcrosoft 9y ago

Please don't remove vendoring...

LaGrange 9y ago
Yeah, this. You want reproducible builds? Vendor your crap. Check it into your repository. Live free.
- hota_mazi 9y ago
  Or adopt a system with a sane versioning system and immutable artifacts, e.g. Maven Central.
  - merb 9y ago
    which still means that your ci needs to download this stuff. with vendor it is already checked it. consider builds on docker etc, where build caching is hard. go is a breeze. build takes like half a second because everything is already there. mvn will first download the world. I'm a scala developer and sbt will probably take like 10 minutes just downloading stuff, if a cache missed (even with a local proxy).
    - pjmlp 9y ago
      I wonder how it is possible to take 10 minutes, unless it is downloading directly from internet. We always have internal Plexus mirrors, and the Jenkins servers have their global .m2 local repository. Even big JEE applications barely take more than 5 minutes to build. The only build system I really dislike are Android builds with Gradle, trying to beat C++ compilation times.
      - merb 9y ago
        well sbt uses ivy which is way slower than maven, when it comes to resolving stuff.
  - LaGrange 9y ago

deleted 9y ago

[deleted]

baq 9y ago

this sentence > First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. is the root of what's wrong with this proposal - namely conflating time of dependency resolution with the time of build. cargo (using it because it's been referenced by the post) solves this problem by telling you to check in the cargo.lock that it generates whenever you 'cargo update'. this means that your version control preserves the build dependencies, just as it preserves build recipes (e.g. build.rs or travis or whatever). in this world, there's no 'newest allowed version', because there is only one version. wise men before me said 'as simple as possible, but no simpler' and I'm afraid this proposal is too simple.

Keats 9y ago

Will there be an official proxy or will open-source projects still break when someone moves/deletes a repository?

icholy 9y ago
That's what local caches are for.
- Keats 9y ago
  Sure but me cloning the repository of someone else won't give me that person cache. Without a central cache, it might work on that person computer but not on mine if a repository moved.

cientifico 9y ago

Please don't.

glasser 9y ago

Is the "import compatibility rule" (ie, `/v2` suffixes in package paths) implemented in vgo today?

teacpde 9y ago

If there is neither GOPATH nor vendor directories, where will the downloaded packages be? I digged vgo a bit and found the downloaded packages currently reside in `$GOPATH/src/v/cache`, and seems like vgo won't work without GOPATH for now (https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a896833851a9e54a9/vendor/cmd/go/internal/vgo/init.go#L111 https://github.com/golang/vgo/blob/b6ca6ae975e2b066c002388a8...).

hota_mazi 9y ago

Amazing that in 2018, a language team needs to be convinced that versioning packages is a good idea. What's next, persuading them that generics are pretty cool too?

ovao 9y ago
Considering there are existing and official dependency management tools for Go, the Go team doesn't need to be convinced that versioning is a good idea. This article is a discussion about mainlining a dependency management system as part of the existing `go` tool. They're similarly aware that generics are a desirable thing to have, but haven't determined the ideal way to implement generics into the language.
dsymonds 9y ago
Did you actually read the article? It is clear from the background and history sections that the need for versioning was apparent from early on.

shkkmo 9y ago

I think that the choice to use "prefer minimum version by default" going to confuse a lot of new developers used to the industry standard being the opposite. If you are going against the industry default, there should be a strong reasons. I don't think reasons provided really justify this. > First, the meaning of “newest allowed version” can change due to external events, namely new versions being published. Maybe tonight someone will introduce a new version of some dependency, and then tomorrow the same sequence of commands you ran today would produce a different result. This is why we have lock files and they work better for this. While vgo's approach does allow reproducible builds, it doesn't allow you to guarantee reproducibility the way that a lock file that has the specific commit hashes does. With specific commit hashes you can verify the the version you are building with in production is the exact same code as the one your security team audited prior to release (assuming git has fixed its hash collision vulernability). You can get around this by abandoning version constraints in your go.mod file, but then you have to track these version constraints out of band and manually figure out what commit hash to stick in go.mod You could also get around this by storing the hases for the approved versions out of band and creating a build script that verifies these hashes prior to building. Both of these workarounds seem to defeat the point of having a standard package manager in the first point. > Second, to override this default, developers spend their time telling the package manager “no, don't use X,” and then the package manager spends its time searching for a way not to use X. If you are concerned with allowing users to override this default, why not have directive to override this default that can optionally be added to each requirement in the go.mod file. This avoids an unexpected default and doesn't force people who want the industry standard default to use a script to set which dependencies use or don't use the -u flag with 'go get'.

Zamicol 9y ago
>With specific commit hashes you can verify the the version you are building with in production is the exact same code as the one your security team audited prior to release (assuming git has fixed its hash collision vulnerability) Can we take a moment and point out that the hash collision vulnerability is still at large with no ETA? This is after years of it being considered insecure I feel like this point continues to be glossed over with versioning systems that depend on git commit hashes. Bruce Schneier warned in February 2005 that SHA needed to be replaced. (https://www.schneier.com/blog/archives/2005/02/cryptanalysis_o.html https://www.schneier.com/blog/archives/2005/02/cryptanalysis...) Git development didn't start until April 2005. So before git was even developed, SHA was identified as needing to be deprecated. So now, 13 years later, this is still an issue.

mCOLlSVIxp6c 9y ago

I predict that under the proposed minimum version selection system some package will decide it's important and flawless enough that users should always depend on the most recent version. The package will recommend that users depend on version v1.0.0 but the first release will be v1.100.0. Subsequent releases will decrement the minor version to make sure newer releases are selected.

readams 9y ago
It certainly complicates the picture. I think the only way to make this workable is if you adopt a convention that you specify your minimum dependency if you're a library, but the current version if you're an executable. Of course, this could just be codified in the tool, which would be a big improvement. It also makes testing a lot harder though since your library ends up tested against a different version than what your users are using. Ugh.
skybrian 9y ago
Interesting loophole, but it requires the community to go along with it. If it turns out people like the minimum version policy and the package is actually important, I would expect someone to fork the repo and do things the normal way?
overcyn 9y ago
User's wouldn't be able to depend on version v1.0.0 if it doesn't exist. From what I understand the tool isn't downloading the list of all versions from GitHub and selecting the minimum. Its searching for the maximum version number (i.e. minimal required version), from your go.mod file and all your dependencies' go.mod file.
- mCOLlSVIxp6c 9y ago
  I think you're correct. So my nasty little hack wouldn't work. That's probably a good thing.

teacpde 9y ago

On one hand, as a developer, I like a Go builtin package management authored and supported by the Go team; on the other hand, I feel bad for the open source contributors who worked hard on third party package managers. Even dep, the officially supported PDM, seems supposed to fade.

remus 9y ago
> Even dep, the officially supported PDM, seems supposed to fade. I beleve this was always the plan. The dep readme certainly makes it sound that way at least: > dep is a prototype dependency management tool for Go. It requires Go 1.8 or newer to compile. dep is safe for production use. > dep is the official experiment, but not yet the official tool. Check out the Roadmap for more on what this means!

platz 9y ago

> A build of a module by itself will always use the specific versions of required dependencies listed in the go.mod file. As part of a larger build, it will only use a newer version if something else in the build requires it. This seems fraught

icholy 9y ago
I think you need a noun to go with that adjective.
- platz 9y ago
  With danger

readams 9y ago

One that that I'm not able to tell from the proposal: is the /v1 at the end of an import path a magical value that the tool handles, or do I literally need to have that directory in my repo and have a version history represented at the top level of every project? Why not do the symbol mangling in the compiler if the goal is to have multiple simultaneous versions built in? This is easy to make backward compatible since it could default to v1.

pcj128 9y ago

Just use Bazel to build your go code. It already solved this problem.

pcj128 9y ago

Just use Bazel to build your go code. It has already solved this problem.

mbertschler 9y ago

Am I totally missing something in this document or does it not talk about where these modules either as zips or source code are stored? It mentions $GOPATH and vendor as not necessary anymore. Where will the actual code be?

j_bond 9y ago

I am curious why go doesn't just vendor dependencies like npm does when confronted with conflicting requirements instead of trying to figure out a version that satisfies all worlds? I always found this to be a nice feature when working with npm. I am also firm believer in version-pinning/lockfiles. Updating versions should be a separate workflow with first-class built-in tools to support it. I think that is the area where most package managers fall flat. They basically rely on the developer to do all the heavy lifting.

TheDong 9y ago
nodejs's "require", which pulls different versions depending on which module calls it, is a cool trick. Unfortunately, it also has a downside, and in go that downside would be noticeable. Let's take one easy example: a logging library. Let's say I pull in "logrus v1.0.1" and one of my dependencies pulls in "logrus v1.0.2". In my "main" function, I set logrus's default loglevel to debug (logrus.SetLevel(logrus.DebugLevel)). If go did the thing nodejs does (a different copy of logrus for my dependency than my main), the "logrus.SetLevel" would only affect my package, not any of my dependencies; they'd still log at a default level. This would be true of other things; "func init" code would run once per dependency that had that library, not just once, maps wouldn't be shared, pools, etc. This is a lot more memory usage, but it's also really surprising, especially in the case of logging. I definitely prefer having only one copy of a library in memory and having package-level variables (like loglevel) work as expected. If that ends up failing and I need two different versions, vendor+import-path rewriting allows an escape hatch These days, npm actually tries to minimize and flatten dependency versions as much as possible to avoid the huge memory tax.

mfer 9y ago

As I look this over a couple things really jump out at me... 1. VCS tags are mutable. That's lock files store revision ids. Go is being used to build immutable infrastructure but the proposed package management system uses mutable versions. 2. The proposal is less featureful that dep, npm/yarn in JS, composer in PHP, maven, crates for rust, and others. I wonder how people will react to that.

Groxx 9y ago
Mutable tags are my primary concern here, yeah. It seems pretty mitigate-able by using that (wonderful IMO[1]) `v1.2.3-date-sha` syntax though - it's just not human editable[2]. [1]: it fixes so many readability problems with SHA-pinned lock files, easily shows downgrades in `diff` output, and `sort` likely produces the exact result you wanted. [2]: which may not be a problem, since you could in theory just re-run the tool to fix it when you enter "v1.2.3" by hand.

xrstf 9y ago

A lot of this seems to hinge on "go get" being the best way to get Go source code. Most discussions I've read so far come down to "well, this might be cool, but it would break go get, so we cannot ever do it". I for one could live without go get alltogether. In 99% of cases it boils down to a simple git clone anyway, which could (should?) be done by the package manager anyway. Maybe it's time to re-evaluate the existence of go get, now that we're seeing that "just clone master and hope nothing breaks" has obviously not worked for everyone outside of Google? Maybe bundling Go (the compiler and linker) with tools for fetching source code wasn't such a good idea after all? Just my 2 cents...

thefounder 9y ago
go get, go install and go build are cool and great tools! Let's not turn them into a maven/pom/npm crap!
acdha 9y ago
I find it bizarre how unnecessarily hard that’s been made to avoid a slight amount of extra typing. The github imports work with the most basic options but I’ve seen people waste a lot of time on internal servers, SSH vs. HTTPS, tagging, etc. where it’d have been cleaner, easier and forward-compatible if you just gave it a URL and it shelled out to run “git clone <URL>”.

stock_toaster 9y ago

I feel like the Go devs optimized implementation simplicity early on (likely a sensible choice given Google's "one giant repo" workflow), and that has resulted in some not-insignificant ecosystem complexity later (for others not using "one giant repo" workflows). It will be interesting to see how this plays out in the longer term.

icholy 9y ago

Like all Go things, it looks kinda strange. But after playing around for a bit, it feels really good. The minimal version selection simplicity is genius.

gkya 9y ago

Highly suggest to read the demo at https://research.swtch.com/vgo-tour https://research.swtch.com/vgo-tour, which really clarifies everything. I do not write Go, but I did write some bits in it in the past, and now think that this will be a very useful addition to the Go toolchain. When I looked at Go for the first time I really disliked the simple but sort-of castrated go-build, and didn't really follow the developments since then, but this seems to solve quite a bit of problems. I especially like how you can shadow a certain dependency with a given package at a given path, and how it facilitates vendoring. Hope this becomes included in go soon.

013a 9y ago

This is a well reasoned proposal. I've got a couple qualms. Let's say you care about strict build reproducibility. You want some form of manifest + lock file that defines exactly what versions of dependencies your package expects. Great; more power to you. The only reason you'd want that is if you aren't storing your dependencies with your source code in source control. Otherwise, what's the point? If you were vendoring and committing it, you already have strict reproducibility, and you have the "supported versions" defined in the git repositories that come alongside your dependencies. So, adding this manifest+lock file allows you to get away with not vendoring+committing. Awesome. You've gained reproducibility, but certainly not strict reproducibility. Dependency package maintainers can rewrite git history. They can force push. They can push breaking changes with minor versions. They can delete their repositories. All of these things have already happened in NPM, either maliciously or accidentally; why do we think they wouldn't happen with Go? If you want strict reproducibility but are expecting it with just a manifest+lock file, without dependency vendoring+committing, you're not getting it. Full stop. So, really, by adding a manifest+lock file, you're adding a "third level" of reproducibility (#2 on this list). 1. Low Reproducibility: Pull HEAD on build. 2. Partial Reproducibility: Use tooling to pin to a git commitish, pull this on build. 3a. Full Reproducibility (Easy+Dirty): Vendor+Commit all dependencies. 3b. Full Reproducibility (Hard+Clean): Mirror your dependencies into a new self-controlled git repo. I am struggling to think of a solid use case that would find real value in #2. I have no doubt that people think it would be valuable, but then the moment the left-pad author deletes his repository, you're going to second guess yourself. You didn't want #2; you wanted #3 and settled for #2 because #3 was too hard or dirty and you convinced yourself that adding tooling and version numbers was keeping you safe. Because semver, right? Moreover, this comes at the cost of complexity, which is strictly against Go's core doctrine. Moreover, a #2 solution looks strikingly similar to gopkg.in. Instead of referencing Repository:HEAD we reference Repository:Commitish. The main difference is that gopkg.in is an external service whereas this would be controlled tooling. But by choosing #2 you're already exposing yourself to bad actors, accidents, github going down, all of the above. So you're willing to accept that exposure, but aren't willing to add that little extra exposure of one more service? I agree that the problem today, even with dep, isn't perfect. But I still strongly believe that the answer lies somewhere in the ideas we already have, not by importing ideas from NPM. We need a #3 that is Easy and Clean, not a #2.

codedokode 9y ago

> This proposal ... deprecates GOPATH in favor of a project-based workflow That would be awesome. GOPATH was an awful idea that works only for distribution package managers.