10 ms·

The Principles of Versioning in Go

174 points by crawshaw 7y ago

pcwalton 7y ago
The basic issue with this criticism of Cargo is that the fact that updating dependencies can cause breakage never actually goes away. The package management tool either makes this problem easy, or it makes it hard. The approach Cargo chooses is to make "cargo update" more likely to work by allowing library authors to specify incompatibilities. This can go wrong in edge cases. But that's inherent to the problem itself. Minimum version selection only seems to work because it doesn't actually solve the problem of keeping dependencies up to date. The moment you want to do that, you're stuck with "go get -u", which is just a worse version of "SAT solving" because it doesn't know how to detect and avoid incompatibilities. (As an aside, I don't like the term "SAT solving" for the problem of package version selection. By focusing on an implementation detail, it makes the problem seem scarier than it actually is. Register allocation is NP hard too, but nobody calls the register allocator "the SAT solver".)
- deleted 7y ago
  [deleted]
- dilap 7y ago
  Why should I care about upgrading versions, though? From my perspective making some program P where I've decided to pull in a dependency on library L, I'm only interested in getting a new version of L if (1) it has some new feature I want or (2) there is a relevant security vulnerability. I don't see what is gained by trying to ask for newer versions of libraries than those I've specifically requested.
  - mfer 7y ago
    How do you know about security fixes (or even relevant bug fixes) in dependencies? How do you know about them in transitive rather than direct dependencies? If you don't know to update how will you get the fixes?
    - dilap 7y ago
      By periodically looking at release notes for my dependencies, and, (if I'm being diligent or don't trust my dependencies to be doing audits of their own dependencies), transitive dependencies. In general I never want to change any dependencies w/o explicitly thinking about the change and setting aside time for testing and breakage. And at the time I'm adding some new direct dependency L, I will pick the version I want (presumably the latest). As for its dependencies, why should I want them to be the latest? I just want the versions that are most likely to work correctly w/ the version of L I have selected. Right?
      - mfer 7y ago
        I totally respect that. I also know that many (most) aren’t like that. There are a variety of reasons like some not having time. What can they do? The experience for those is often more time consuming than in the past. This decreases their productivity and happiness with the tooling.
  - pcwalton 7y ago
    Bug fixes and performance improvements!
    - Skinney 7y ago
      But in this case. Don't you already have an idea, through profiling or debugging, which package causes bugs or is slow? And if you do, figuring out if theres a newer version that improves your situation should be rather simple. I've stopped blindly updating my dependencies, as people seem to have very different ideas what semver means. minor and patch upgrades in node and ruby programs have broken my applications several times. Even when semver is done correctly, a breaking change in a package usually also means that you have to upgrade to the breaking change to get other goodies like improved performance and bug-fixes. This is something that takes a lot of time. At work we've basically stopped doing breaking changes in our private packages, opting instead in for a scheme similar to what you have to do in Go for everything to work. Versioning is, at the end of the day, a cultural problem. And not having the tools to do breaking changes well can actually turn out well. At least, that's my current thinking. I may very well be, and often am, wrong.
  - cesarb 7y ago
    Or (3) it has fixed bugs (including ones we haven't hit yet) or (4) it has better compatibility with the surrounding ecosystem (which is not static). More than once, the fix for a mysterious bug we encountered in production was just "upgrade a particular dependency to a newer minor release" (changing nothing on our code), since another user of that dependency had already encountered that same obscure bug, and together with the developers had already investigated it and found a fix. Had we been more diligent in keeping our dependencies up to date, even when they have no new features we want or security fixes, we would not have encountered that bug.
    - dilap 7y ago
      That’s very optimistic! In my experience, software doesn’t actually get less buggy with time, just differently-buggy, and sometimes even more buggy (which makes sense, if bugs go like complexity, and software gains complexity with time).
- sansnomme 7y ago
  I think calling it a SAT solver is appropriate here. A lot of package managers were "organically grown" i.e. they were created in response to a problem that arose. Not much thought was given to the underlying problem and in the long term users suffer with e.g. dependencies taking forever to install and strange deadlocks. Recognizing it as a SAT problem from the start means we can easily throw Z3 or similar at it instead of manually adding heuristics everytime a user files a GitHub issue. (See Bundler and other pre-2018/2017 package management tools for example) Also, it's not always a SAT problem. For npm, Java Classloaders it's just tree traversal because conflicting sub dependencies of a dependency can co-exist. That said, the Go's team (and to a lesser extent Python's too) dropped the ball here. None of the challenges of dependency management was unknown or unsolved. Anyone who has used e.g. Maven should be more than familiar. (A 11 section blog post is indeed nice though, good for onboarding new programmers I suppose) If developer experience was considered a first class priority, this could have been solved from day 0. (My personal favorite is NPM since conflicting dependencies can co-exist at the expense of greater disk usage, but memory is cheap these days. No one with an electron app on their desktop has any right to complain about the memory usage of plain text source code.)
  - mfer 7y ago
    > A lot of package managers were "organically grown" Many of the package managers we have now are thought out. Some replaced existing ones while others were crafted by people who had worked on dependency management in the past. Cargo and Composer are two examples of this.
  - pcwalton 7y ago
    See, that's the reason why calling it a SAT solver makes the problem sound scarier than it is. Using Z3--or even MiniSAT--is massive overkill. The naive exponential algorithm is fine, and it leads to more maintainable and understandable code. Rust's build performance has nothing to do with dependency solving. I've never heard of a package manager in which core dependency solving was a speed problem; dep was slow because of I/O, not SAT solving.
    - Groxx 7y ago
- ainar-g 7y ago
  > (As an aside, I don't like the term "SAT solving" for the problem of package version selection. By focusing on an implementation detail, it makes the problem seem scarier than it actually is. Register allocation is NP hard too, but nobody calls the register allocator "the SAT solver".) I don't quite get your criticism here. Russ's Minimal Version Selection algorithm is a package version selection algorithm that does not require SAT solving. Your wording sounds like Russ wrote something like “Package version selection is inherently SAT, so we won't do that ever!”, but he did exactly the opposite by introducing MVS. Maybe you meant that you don't like the term for “Cargo-like package version selection”, but that's a completely different story.
  - littlestymaar 7y ago
    > I don't quite get your criticism here. Russ's Minimal Version Selection algorithm is a package version selection algorithm that does not require SAT solving. That algorithm also doesn't handle updates though, which is quite a big limitation.
- throwaway34241 7y ago
  If I understand right, they support multiple incompatible versions being used at once via major version upgrades. What they aren’t supporting is incompatible dependencies being incorrectly advertised as compatible, then being worked around by other dependencies testing against those versions and explicitly marking them as incompatible, in cases where dependencies have a shared dependency with different requirements. Instead it looks like they are trying to solve this problem outside of the dependency manager, by putting a lot of emphasis on preserving backwards compatibility within major versions being the correct Go practice (and probably also hoping that edge cases get fixed quickly in a collaborative way, since everyone is on the same system). How well this approach will work relatively in practice seems to depend more on the behavior of library authors than any technical factor, without knowing that it seems hard to say that either is obviously superior.
  - pcwalton 7y ago
    I'd describe it instead as that Cargo gives you a way out if your dependencies accidentally break semver, while Go's package manager doesn't. That may be a sensible decision on Go's part, but it doesn't solve the problem of dependency incompatibility. In fact, Go explicitly doesn't solve it.
    - skybrian 7y ago
      Go solves it. The way out if things break is to use `go get` instead of `go get -u`, and then upgrade selectively. Think of `go get` as stable and `go get -u` as unstable. If you're working on unstable, then things may break and you should report bugs and/or contribute patches. Upgrading a library dependency in your module file and committing the change is how you change stable for your library. Downstream users have their own ideas of stable that will be behind yours, until they upgrade.
      - pcwalton 7y ago
        By "solves it" I mean "the system provides a tool that automatically upgrades all your packages to (1) their most recent versions without (2) breaking anything". Go get, without the -u, doesn't provide (1), since it only updates to minimum versions. Go get -u doesn't reliably provide (2), since there is no way to specify incompatibilities.
- skybrian 7y ago
  I think you're right that `go get -u` could be improved on by having some system where the community shares known incompatibilities. It seems like this early-warning system could be built on top of Go modules, though? It might end up being more elegant than what we have, where anyone can report an incompatibility without either module author being involved at first. Another improvement might be having a way to say "give me just the security fixes". At first this could be a new tool that you run instead of `go get -u`, so the Go team doesn't have to be the ones to start this project. People who just want their code to work don't need to care about this. They can run `go get`. There are always going to be people who just want stable code and repeatable builds, and others who are working on migrating the community to the latest code. (And they might be the same person at different times.) It seems like Go's module system might work pretty well at helping them collaborate while staying out of each others' way?
  - Thaxll 7y ago
    "Another improvement might be having a way to say "give me just the security fixes"." It's already possible: https://github.com/golang/go/wiki/Modules https://github.com/golang/go/wiki/Modules https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies https://github.com/golang/go/wiki/Modules#how-to-upgrade-and...
    - skybrian 7y ago
      Uh, I'm not sure why you're linking to that? Maybe I missed something. It doesn't seem to explain anything about library authors marking new versions as security fixes and querying for them, which is what I had in mind.
  - linuxftw 7y ago
    > "give me just the security fixes" The linux world solved this with providing a package repository that maintains stable versions of software. This requires a lot of time and effort. Many software projects simply don't have the labor available to support multiple releases of software. They run tip of master and that's pretty much it.
- ithkuil 7y ago
  > because it doesn't know how to detect and avoid incompatibilities They reshaped the problem: all future revisions of a single entity must be backwards compatible because it's the "right thing to do", and thus, under this assumption, a much simpler solver will do the job. Yes, this moves the problem slightly, since now there are other places that encode versioning information (the import path themselves) and thus some of the incompatibility detection checks performed by package managers are now delegated to the compiler (which will e.g. not find a given import, or complain that foo.Bar is not assignable to foo.Bar because the full import path has a different version tag). Yes, many people do not currently care about the problem of backward compatibility and it will be hard in practice to make everybody raise their standards. But, careful backward compatibility is one of the main ingredients that made Go successful in the first place. A healthy ecosystem should follow the same rules. Incidentally, this whole vgo kerfuffle can help highlight which section of the ecosystem is written without this rule in mind, offering a choice for both the authors of the components (improve their practices) and the consumers (e.g. switch to another library)
  - ithkuil 7y ago
    (which guidelines of polite discourse did I break in order to deserve downvotes?)
mfer 7y ago
> The answer is that we learned from Dep that the general Bundler/Cargo/Dep approach includes some decisions that make software engineering more complex and more challenging. Are those documented anywhere? Are they detailed, discussed, or otherwise dealt with. The people behind those package manages have a significant amount of experience with dependency management including many real world situations. It would be good to have some public discussions on this for everyone's benefit if this is really true. > Principle #1: Compatibility The whole section on compatibility doesn't sit right... > It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. Go programs that work today should continue to work even as future “point” releases of Go 1 arise (Go 1.1, Go 1.2, etc.). This doesn't seem to apply to the tooling. For example, `go get` has a new behavior since the introduction of modules. The behavior changed. So, the compatibility idea only extends so far. Is that hypocritical as an idea because people use the `go` CLI for scripting and the behavior of it changed. > What does compatibility have to do with versioning? It’s important to think about compatibility because the most popular approach to versioning today—semantic versioning—instead encourages incompatibility. That is, semantic versioning has the unfortunate effect of making incompatible changes seem easy. Incompatible changes happen. How many long lived pieces of software have needed to make incompatible changes while supporting people through that process. Do you rewrite? Do you copy the whole codebase to a new sub-directory and add more there? This is what the Go project has started to recommend. If you do that you loose history and some projects move fast (look at Kubernetes client-go). Tools like npm, cargo, and dep provide a means to stay at older versions if that's what you want. The choice is up to the consumer of the package. With Go the choice is now up to the Go team by limiting choice. Is that a good thing? Not if you don't like the choice.
- jerf 7y ago
  "This doesn't seem to apply to the tooling. For example, `go get` has a new behavior since the introduction of modules.... Is that hypocritical as an idea because people use the `go` CLI for scripting and the behavior of it changed." It does not apply to tooling. It is not hypocritical because it was never claimed that it applies to tooling. It has always been a promise about code.
mfer 7y ago
Both Red Monk and TIOBE [1] are noting a decrease in the popularity of Go. Red Monk wondered aloud if this had to do with the way the Go modules / dep situation happened [2]. I wonder if the posts lately are an attempt to win people over to their thinking or to win back popularity. [1] https://www.tiobe.com/tiobe-index/go/ https://www.tiobe.com/tiobe-index/go/ [2] https://redmonk.com/sogrady/2019/07/18/language-rankings-6-19/ https://redmonk.com/sogrady/2019/07/18/language-rankings-6-1...
- dilap 7y ago
  Yeah, I'm a Go fan, and I think modules/minimum-version-control seems like a good idea, but the implementation was handled poorly -- it broke tons of tools, and things still aren't fully working, which is insane. For a language whose appeal rests in large part on "shit just works," it's defnly been a bummer. (Maybe it's actually not something that could've been done better; maybe it's just the nature of the magnitude of the change. In which case I think the mistake was going "1.0" when you knew you had something so important not yet defined.)
  - mfer 7y ago
    There are many cases MVS and updated don’t work for. For example, if you have a dependency on a package inside kubernetes because it doesn’t follow semver. You can’t pin to minor updates and use ‘go get -u’ Control has been taken away from package consumers
- jerf 7y ago
  Tiobe fluctuates wildly, and it isn't really plausible that it plummeted as far as it did on Tiobe in just one month. The chatter on HN will tend to amplify problems beyond what the general public actually feels the correct level is, and even on HN this hasn't exactly been a big topic. (When generic Go topics discuss "what's wrong with Go" this is not generally something that is mentioned.) It seems far more likely that, yet again (this isn't exactly a new thing with Tiobe), the search engines did "something" and the searches changed again. Many, if not most, of the programming language names are susceptible to that, between things that are too short to search for like C and terms of general interest like Python, but Go really has to be the worst. Ten-year-old+ programming languages with at the very least tens of thousands of users (I see "millions" claimed, I find myself a bit skeptical, but maybe... conservatively tens of thousands for sure, though) do not lose 33% of their users in a month. That'd be somewhere in 2-3 orders of magnitude faster than Perl 5 has declined. Implausible.
Groxx 7y ago
A very large issue with go tooling around dependency management is that it's based around concepts like this: >If an old package and a new package have the same import path, the new package must be backwards compatible with the old package. Which are effectively fine conceptually. Every dependency system also supports that, you just make a new project named "thing-v2" instead of bumping a major version. Making it explicit and encouraging it when possible (instead of forcing breaking changes) is good. The problem is that there's absolutely nothing to help you achieve that. Or detect failures to achieve it. Or anything. You can't even reliably rename a single package with existing refactoring tools, good luck copying and refactoring most of an entire repository correctly. If you do it wrong, you might have to do it all over again because it's probably another breaking change. Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful.
- ainar-g 7y ago
  > The problem is that there's absolutely nothing to help you achieve that. (…) There were talks about "go mod release" from the very beginning, but you are currently correct here. > (…) You can't even reliably rename a single package with existing refactoring tools. (…) Strictly false. There is gomvpkg[1]. The absolutely ironic thing is that it's still stuck in the GOPATH mode, despite being a part of the official x/tools repo, but it's not “absolutely nothing”. [1] https://godoc.org/golang.org/x/tools/cmd/gomvpkg https://godoc.org/golang.org/x/tools/cmd/gomvpkg > Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful. I agree. Honestly, the slow progress in moving the community's favourite tools, including gomvpkg, to work in the new ecosystem is a massive failure on the Go authors' side. That just means that they haven't made the move easy and simple enough.
  - Groxx 7y ago
    >Strictly false. There is gomvpkg[1]. The absolutely ironic thing is that it's still stuck in the GOPATH mode, despite being a part of the official x/tools repo, but it's not “absolutely nothing”. IMO renaming packages is a necessary prerequisite, but otherwise orthogonal to handling breaking changes. Personally I don't include it in the "something that could help" group. Either way, it's wildly unreliable in my experience, so that point stands.
tandr 7y ago
(Kind of meta-venting out below, also known as rant.) 1. I don't know why I find Russ's writings are so irksome, even unpleasant to read, sorry. (Please help me on that, maybe I am in "questioning authority" mode and cannot flip out of it.) I cannot shake that feeling of being patronized all the time without being asked. The whole article, no - bunch of his posts, - reads like "we know better". Is it a Google thing, or heavy Rob Pike's influence? Yes, he is bringing some good points, and some very questionable ones. And for the questionable one (Compatibility section for example, or SAT solver question (nobody did it before?), or version naming, or paths for the sub-components...) it feels like he is trying to push explanation about his (and his team?) reasons and motivations behind these decisions, but... it is a monologue. There are more opinions and whole wide world of different software shops with own ideas and practices. 2. Go2 was a promised land, that later translated into "incremental, no code breaking" approach. Nowadays it feels like they are not even talking about it anymore! Looking at general history of programming languages, pretty much every language has had it's "C++" or Modula-2 moments. It is a natural process - an evolution - when unneeded parts disappear, and useful ideas added/extended. Amount of gotchas in Go is not zero, and to me (YMMV) it feels like Go2 would be (was?) a good time to shake off under-thought items, and enforce some new rules. Maybe even add interoperability between .go and .go2 files, maybe even semi-automatic upgrade path to run `go2 fmt ` on old files a la to Kotlin from Java - I don't know. But it feels like an opportunity and momentum for a "peaceful revolution" is lost. 3. Even with non-breaking changes style of change they have managed to break stuff. I, for one, still sit on 1.12 branch - because when I tried to upgrade to 1.13 (twice now), and our pure-Go code just plainly did not build with some obscure messages (I think it was linking or module cache errors). So, why pretend? Code rot is unavoidable - do we like it or not. 99. There are couple more points about chilling effect of "we know better" on a community as whole. (One example would be this whole "error" debacle that visible reduced activity on issue tracker and golang-nuts mailing lists after "hey all, we are going this way because we already went there". Immediately after it felt like people just gave up and left. "Empty town square" feeling.) It would be interesting to see this year results of "the state of Go" survey, but I don't think we will see a worrisome drops there -- the amount of new people coming in would easily override any opinions of people leaving. /end of rant, sorry for a long post
- 7y ago
sagichmal 7y ago
> If an old package and a new package have the same import path, the new package must be backwards compatible with the old package. Simply put, this rule is too strict, and biases modules' UX and semantics too far toward consumers. Not all packages are so mature, and consumed by enough consumers, that they should be forced by the language tooling to maintain backwards compatibility. That requirement creates a great deal of pointless work for the majority, perhaps vast majority, of package authors, who write small and uncertain packages for use by themselves or a small number of their immediate colleagues. For these authors, strict backwards compatibility is not only unnecessary, it's actually undesirable, as it creates needless toil. > The answer is that we learned from Dep that the general Bundler/Cargo/Dep approach includes some decisions that make software engineering more complex and more challenging. For a minority of use cases, where packages are large and mature and imported by many, many consumers -- sure, maybe. But the vast majority of Go code, written by private organizations and used internally, doesn't fit this description.
- mfcl 7y ago
  In that case, can you just stay in v0.x.y?
  - akerl_ 7y ago
    You can either stay in v0.x.y, or you can just always bump the major version. Either option meets Go’s versioning requirements without requiring developers to think about “is this change backwards compatible” when they tag a new release.
- IshKebab 7y ago
  Nothing's physically stopping you breaking backwards compatibility without using a new import path if you want to. You might piss off your users but if they don't care then go for it.
  - grey-area 7y ago
    Doesn’t the go tooling mandate this if you are >= v2.0? The rest of mod looks fine but tying paths to major version changes was a mistake IMO.
shadowgovt 7y ago
Spending my time as of late in npm's particular flavor of DLL hell, I appreciate the Go team taking a stab at this hard and persistent problem. I agree with other commenters' observations that the tooling hasn't caught up with the philosophy, but the philosophy seems on-point (and the key observation that as the dependencies of a project approach infinity, the odds of two branches of a dependency tree wanting to rely on two different versions of the same library approach 100% and there should be a sane way to address that need... npm does a 99% decent job of addressing this, but I've definitely hit issues with npm where objects generated by a newer dependency version have leaked into a codepath where an older version is calling the shots, and it's an absolute hellscape to debug).
- kortex 7y ago
  Npm seems to lack, in the context of the article, the ecosystem and community aspect. It's all like herding cats. Case in point, leftpad. * > as the dependencies of a project approach infinity, the odds of two branches of a dependency tree wanting to rely on two different versions of the same library approach 100% * With experience, I've found that this is precisely the case, with bounds much, much smaller than infinity. The ersatz standard lib can't even standardize, there's underscore and lodash. I think eventually the community will Ostwalt ripen in a sense, small packages will dissolve by attrition and big ones will grow. But the argument is strong for simply decreasing n = number of discrete packages.
mfer 7y ago
Side note, I just noticed that https://godoc.org https://godoc.org does not support Go modules with major versions being incremented (e.g., ending in `/v2`). The new pkg.go.dev does but not the existing godoc.org. This echos some of the other observations of tooling not being easily updated and still not being there, yet.
- BarkMore 7y ago
  pkg.go.dev is the module aware replacement for godoc.org.
  - mfer 7y ago
    The tools, habits, and existing links haven't been updated. godoc.org don't point people to the new site. The "replacement" experience is a little broken.
nyc640 7y ago
I understand why this rubs some people the wrong way, but it really is consistent with Go's vision since the start. I don't think Russ is trying to imply that the issues they are facing when designing the language (dependency management, generics, etc.) are unique or impossible to solve in any way. I think he is simply saying that they are complex problems whose solutions introduce complexity to the language and ecosystem, and when presented with such a trade-off, the Go team has always preferred to lean towards simplicity, even if it means not implementing a feature that some (or even many) developers want/need. While I don't personally agree with many of the decisions they've made, I still respect the choices they've made and I think there is plenty of room for languages with competing philosophies. Because of Go's philosophy of not catering to every demand, it's never going to become the next Java or take over all of Software Engineering, but I think the Go team is fine with that as long as it continues to be useful as a tool for solving problems.
- erikpukinskis 7y ago
  The best code is no code, and the best solution is sometimes just not trying to solve the problem.
- bmurphy1976 7y ago
  They are building what they want. More power to them for making a tool that works for them first and not chasing every flight of fancy. I respect that philosophy because it's given me more of what I actually want and use in a platform than anything since Python. It's not for everybody but for those of us who really like it this is a relief from all the overcomplicated snake oil that's been shoved down our throats for the last 20 years.
- ilovecaching 7y ago
  C is a simple language and yet it constitutes the guts of most computers on this planet and Mars. Simple works. Go is really close to C. In fact, Rob Pike's Go CLI programs on his Github looks pretty much like a standard C program using getopt. Interfaces are just a nicer virtual function table. interface{} is just void *. In that sense, it's already treading ground that is proven to work. Go didn't have to do anything revolutionary because C was already great. I hope the Go team will continue to embody what made C great so the next generation can receive the lessons its taught us.

174 points by crawshaw 7y ago

The basic issue with this criticism of Cargo is that the fact that updating dependencies can cause breakage never actually goes away. The package management tool either makes this problem easy, or it makes it hard. The approach Cargo chooses is to make "cargo update" more likely to work by allowing library authors to specify incompatibilities. This can go wrong in edge cases. But that's inherent to the problem itself. Minimum version selection only seems to work because it doesn't actually solve the problem of keeping dependencies up to date. The moment you want to do that, you're stuck with "go get -u", which is just a worse version of "SAT solving" because it doesn't know how to detect and avoid incompatibilities. (As an aside, I don't like the term "SAT solving" for the problem of package version selection. By focusing on an implementation detail, it makes the problem seem scarier than it actually is. Register allocation is NP hard too, but nobody calls the register allocator "the SAT solver".)

deleted 7y ago
[deleted]
dilap 7y ago
Why should I care about upgrading versions, though? From my perspective making some program P where I've decided to pull in a dependency on library L, I'm only interested in getting a new version of L if (1) it has some new feature I want or (2) there is a relevant security vulnerability. I don't see what is gained by trying to ask for newer versions of libraries than those I've specifically requested.
- mfer 7y ago
  How do you know about security fixes (or even relevant bug fixes) in dependencies? How do you know about them in transitive rather than direct dependencies? If you don't know to update how will you get the fixes?
  - dilap 7y ago
    By periodically looking at release notes for my dependencies, and, (if I'm being diligent or don't trust my dependencies to be doing audits of their own dependencies), transitive dependencies. In general I never want to change any dependencies w/o explicitly thinking about the change and setting aside time for testing and breakage. And at the time I'm adding some new direct dependency L, I will pick the version I want (presumably the latest). As for its dependencies, why should I want them to be the latest? I just want the versions that are most likely to work correctly w/ the version of L I have selected. Right?
    - mfer 7y ago
      I totally respect that. I also know that many (most) aren’t like that. There are a variety of reasons like some not having time. What can they do? The experience for those is often more time consuming than in the past. This decreases their productivity and happiness with the tooling.
- pcwalton 7y ago
  Bug fixes and performance improvements!
  - Skinney 7y ago
    But in this case. Don't you already have an idea, through profiling or debugging, which package causes bugs or is slow? And if you do, figuring out if theres a newer version that improves your situation should be rather simple. I've stopped blindly updating my dependencies, as people seem to have very different ideas what semver means. minor and patch upgrades in node and ruby programs have broken my applications several times. Even when semver is done correctly, a breaking change in a package usually also means that you have to upgrade to the breaking change to get other goodies like improved performance and bug-fixes. This is something that takes a lot of time. At work we've basically stopped doing breaking changes in our private packages, opting instead in for a scheme similar to what you have to do in Go for everything to work. Versioning is, at the end of the day, a cultural problem. And not having the tools to do breaking changes well can actually turn out well. At least, that's my current thinking. I may very well be, and often am, wrong.
- cesarb 7y ago
  Or (3) it has fixed bugs (including ones we haven't hit yet) or (4) it has better compatibility with the surrounding ecosystem (which is not static). More than once, the fix for a mysterious bug we encountered in production was just "upgrade a particular dependency to a newer minor release" (changing nothing on our code), since another user of that dependency had already encountered that same obscure bug, and together with the developers had already investigated it and found a fix. Had we been more diligent in keeping our dependencies up to date, even when they have no new features we want or security fixes, we would not have encountered that bug.
  - dilap 7y ago
    That’s very optimistic! In my experience, software doesn’t actually get less buggy with time, just differently-buggy, and sometimes even more buggy (which makes sense, if bugs go like complexity, and software gains complexity with time).
sansnomme 7y ago
I think calling it a SAT solver is appropriate here. A lot of package managers were "organically grown" i.e. they were created in response to a problem that arose. Not much thought was given to the underlying problem and in the long term users suffer with e.g. dependencies taking forever to install and strange deadlocks. Recognizing it as a SAT problem from the start means we can easily throw Z3 or similar at it instead of manually adding heuristics everytime a user files a GitHub issue. (See Bundler and other pre-2018/2017 package management tools for example) Also, it's not always a SAT problem. For npm, Java Classloaders it's just tree traversal because conflicting sub dependencies of a dependency can co-exist. That said, the Go's team (and to a lesser extent Python's too) dropped the ball here. None of the challenges of dependency management was unknown or unsolved. Anyone who has used e.g. Maven should be more than familiar. (A 11 section blog post is indeed nice though, good for onboarding new programmers I suppose) If developer experience was considered a first class priority, this could have been solved from day 0. (My personal favorite is NPM since conflicting dependencies can co-exist at the expense of greater disk usage, but memory is cheap these days. No one with an electron app on their desktop has any right to complain about the memory usage of plain text source code.)
- mfer 7y ago
  > A lot of package managers were "organically grown" Many of the package managers we have now are thought out. Some replaced existing ones while others were crafted by people who had worked on dependency management in the past. Cargo and Composer are two examples of this.
- pcwalton 7y ago
  See, that's the reason why calling it a SAT solver makes the problem sound scarier than it is. Using Z3--or even MiniSAT--is massive overkill. The naive exponential algorithm is fine, and it leads to more maintainable and understandable code. Rust's build performance has nothing to do with dependency solving. I've never heard of a package manager in which core dependency solving was a speed problem; dep was slow because of I/O, not SAT solving.
  - Groxx 7y ago
ainar-g 7y ago
> (As an aside, I don't like the term "SAT solving" for the problem of package version selection. By focusing on an implementation detail, it makes the problem seem scarier than it actually is. Register allocation is NP hard too, but nobody calls the register allocator "the SAT solver".) I don't quite get your criticism here. Russ's Minimal Version Selection algorithm is a package version selection algorithm that does not require SAT solving. Your wording sounds like Russ wrote something like “Package version selection is inherently SAT, so we won't do that ever!”, but he did exactly the opposite by introducing MVS. Maybe you meant that you don't like the term for “Cargo-like package version selection”, but that's a completely different story.
- littlestymaar 7y ago
  > I don't quite get your criticism here. Russ's Minimal Version Selection algorithm is a package version selection algorithm that does not require SAT solving. That algorithm also doesn't handle updates though, which is quite a big limitation.
throwaway34241 7y ago
If I understand right, they support multiple incompatible versions being used at once via major version upgrades. What they aren’t supporting is incompatible dependencies being incorrectly advertised as compatible, then being worked around by other dependencies testing against those versions and explicitly marking them as incompatible, in cases where dependencies have a shared dependency with different requirements. Instead it looks like they are trying to solve this problem outside of the dependency manager, by putting a lot of emphasis on preserving backwards compatibility within major versions being the correct Go practice (and probably also hoping that edge cases get fixed quickly in a collaborative way, since everyone is on the same system). How well this approach will work relatively in practice seems to depend more on the behavior of library authors than any technical factor, without knowing that it seems hard to say that either is obviously superior.
- pcwalton 7y ago
  I'd describe it instead as that Cargo gives you a way out if your dependencies accidentally break semver, while Go's package manager doesn't. That may be a sensible decision on Go's part, but it doesn't solve the problem of dependency incompatibility. In fact, Go explicitly doesn't solve it.
  - skybrian 7y ago
    Go solves it. The way out if things break is to use `go get` instead of `go get -u`, and then upgrade selectively. Think of `go get` as stable and `go get -u` as unstable. If you're working on unstable, then things may break and you should report bugs and/or contribute patches. Upgrading a library dependency in your module file and committing the change is how you change stable for your library. Downstream users have their own ideas of stable that will be behind yours, until they upgrade.
    - pcwalton 7y ago
      By "solves it" I mean "the system provides a tool that automatically upgrades all your packages to (1) their most recent versions without (2) breaking anything". Go get, without the -u, doesn't provide (1), since it only updates to minimum versions. Go get -u doesn't reliably provide (2), since there is no way to specify incompatibilities.
skybrian 7y ago
I think you're right that `go get -u` could be improved on by having some system where the community shares known incompatibilities. It seems like this early-warning system could be built on top of Go modules, though? It might end up being more elegant than what we have, where anyone can report an incompatibility without either module author being involved at first. Another improvement might be having a way to say "give me just the security fixes". At first this could be a new tool that you run instead of `go get -u`, so the Go team doesn't have to be the ones to start this project. People who just want their code to work don't need to care about this. They can run `go get`. There are always going to be people who just want stable code and repeatable builds, and others who are working on migrating the community to the latest code. (And they might be the same person at different times.) It seems like Go's module system might work pretty well at helping them collaborate while staying out of each others' way?
- Thaxll 7y ago
  "Another improvement might be having a way to say "give me just the security fixes"." It's already possible: https://github.com/golang/go/wiki/Modules https://github.com/golang/go/wiki/Modules https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies https://github.com/golang/go/wiki/Modules#how-to-upgrade-and...
  - skybrian 7y ago
    Uh, I'm not sure why you're linking to that? Maybe I missed something. It doesn't seem to explain anything about library authors marking new versions as security fixes and querying for them, which is what I had in mind.
- linuxftw 7y ago
  > "give me just the security fixes" The linux world solved this with providing a package repository that maintains stable versions of software. This requires a lot of time and effort. Many software projects simply don't have the labor available to support multiple releases of software. They run tip of master and that's pretty much it.
ithkuil 7y ago
> because it doesn't know how to detect and avoid incompatibilities They reshaped the problem: all future revisions of a single entity must be backwards compatible because it's the "right thing to do", and thus, under this assumption, a much simpler solver will do the job. Yes, this moves the problem slightly, since now there are other places that encode versioning information (the import path themselves) and thus some of the incompatibility detection checks performed by package managers are now delegated to the compiler (which will e.g. not find a given import, or complain that foo.Bar is not assignable to foo.Bar because the full import path has a different version tag). Yes, many people do not currently care about the problem of backward compatibility and it will be hard in practice to make everybody raise their standards. But, careful backward compatibility is one of the main ingredients that made Go successful in the first place. A healthy ecosystem should follow the same rules. Incidentally, this whole vgo kerfuffle can help highlight which section of the ecosystem is written without this rule in mind, offering a choice for both the authors of the components (improve their practices) and the consumers (e.g. switch to another library)
- ithkuil 7y ago
  (which guidelines of polite discourse did I break in order to deserve downvotes?)

mfer 7y ago

> The answer is that we learned from Dep that the general Bundler/Cargo/Dep approach includes some decisions that make software engineering more complex and more challenging. Are those documented anywhere? Are they detailed, discussed, or otherwise dealt with. The people behind those package manages have a significant amount of experience with dependency management including many real world situations. It would be good to have some public discussions on this for everyone's benefit if this is really true. > Principle #1: Compatibility The whole section on compatibility doesn't sit right... > It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. Go programs that work today should continue to work even as future “point” releases of Go 1 arise (Go 1.1, Go 1.2, etc.). This doesn't seem to apply to the tooling. For example, `go get` has a new behavior since the introduction of modules. The behavior changed. So, the compatibility idea only extends so far. Is that hypocritical as an idea because people use the `go` CLI for scripting and the behavior of it changed. > What does compatibility have to do with versioning? It’s important to think about compatibility because the most popular approach to versioning today—semantic versioning—instead encourages incompatibility. That is, semantic versioning has the unfortunate effect of making incompatible changes seem easy. Incompatible changes happen. How many long lived pieces of software have needed to make incompatible changes while supporting people through that process. Do you rewrite? Do you copy the whole codebase to a new sub-directory and add more there? This is what the Go project has started to recommend. If you do that you loose history and some projects move fast (look at Kubernetes client-go). Tools like npm, cargo, and dep provide a means to stay at older versions if that's what you want. The choice is up to the consumer of the package. With Go the choice is now up to the Go team by limiting choice. Is that a good thing? Not if you don't like the choice.

jerf 7y ago
"This doesn't seem to apply to the tooling. For example, `go get` has a new behavior since the introduction of modules.... Is that hypocritical as an idea because people use the `go` CLI for scripting and the behavior of it changed." It does not apply to tooling. It is not hypocritical because it was never claimed that it applies to tooling. It has always been a promise about code.

mfer 7y ago

Both Red Monk and TIOBE [1] are noting a decrease in the popularity of Go. Red Monk wondered aloud if this had to do with the way the Go modules / dep situation happened [2]. I wonder if the posts lately are an attempt to win people over to their thinking or to win back popularity. [1] https://www.tiobe.com/tiobe-index/go/ https://www.tiobe.com/tiobe-index/go/ [2] https://redmonk.com/sogrady/2019/07/18/language-rankings-6-19/ https://redmonk.com/sogrady/2019/07/18/language-rankings-6-1...

dilap 7y ago
Yeah, I'm a Go fan, and I think modules/minimum-version-control seems like a good idea, but the implementation was handled poorly -- it broke tons of tools, and things still aren't fully working, which is insane. For a language whose appeal rests in large part on "shit just works," it's defnly been a bummer. (Maybe it's actually not something that could've been done better; maybe it's just the nature of the magnitude of the change. In which case I think the mistake was going "1.0" when you knew you had something so important not yet defined.)
- mfer 7y ago
  There are many cases MVS and updated don’t work for. For example, if you have a dependency on a package inside kubernetes because it doesn’t follow semver. You can’t pin to minor updates and use ‘go get -u’ Control has been taken away from package consumers
jerf 7y ago
Tiobe fluctuates wildly, and it isn't really plausible that it plummeted as far as it did on Tiobe in just one month. The chatter on HN will tend to amplify problems beyond what the general public actually feels the correct level is, and even on HN this hasn't exactly been a big topic. (When generic Go topics discuss "what's wrong with Go" this is not generally something that is mentioned.) It seems far more likely that, yet again (this isn't exactly a new thing with Tiobe), the search engines did "something" and the searches changed again. Many, if not most, of the programming language names are susceptible to that, between things that are too short to search for like C and terms of general interest like Python, but Go really has to be the worst. Ten-year-old+ programming languages with at the very least tens of thousands of users (I see "millions" claimed, I find myself a bit skeptical, but maybe... conservatively tens of thousands for sure, though) do not lose 33% of their users in a month. That'd be somewhere in 2-3 orders of magnitude faster than Perl 5 has declined. Implausible.

Groxx 7y ago

A very large issue with go tooling around dependency management is that it's based around concepts like this: >If an old package and a new package have the same import path, the new package must be backwards compatible with the old package. Which are effectively fine conceptually. Every dependency system also supports that, you just make a new project named "thing-v2" instead of bumping a major version. Making it explicit and encouraging it when possible (instead of forcing breaking changes) is good. The problem is that there's absolutely nothing to help you achieve that. Or detect failures to achieve it. Or anything. You can't even reliably rename a single package with existing refactoring tools, good luck copying and refactoring most of an entire repository correctly. If you do it wrong, you might have to do it all over again because it's probably another breaking change. Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful.

ainar-g 7y ago
> The problem is that there's absolutely nothing to help you achieve that. (…) There were talks about "go mod release" from the very beginning, but you are currently correct here. > (…) You can't even reliably rename a single package with existing refactoring tools. (…) Strictly false. There is gomvpkg[1]. The absolutely ironic thing is that it's still stuck in the GOPATH mode, despite being a part of the official x/tools repo, but it's not “absolutely nothing”. [1] https://godoc.org/golang.org/x/tools/cmd/gomvpkg https://godoc.org/golang.org/x/tools/cmd/gomvpkg > Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful. I agree. Honestly, the slow progress in moving the community's favourite tools, including gomvpkg, to work in the new ecosystem is a massive failure on the Go authors' side. That just means that they haven't made the move easy and simple enough.
- Groxx 7y ago
  >Strictly false. There is gomvpkg[1]. The absolutely ironic thing is that it's still stuck in the GOPATH mode, despite being a part of the official x/tools repo, but it's not “absolutely nothing”. IMO renaming packages is a necessary prerequisite, but otherwise orthogonal to handling breaking changes. Personally I don't include it in the "something that could help" group. Either way, it's wildly unreliable in my experience, so that point stands.

tandr 7y ago

(Kind of meta-venting out below, also known as rant.) 1. I don't know why I find Russ's writings are so irksome, even unpleasant to read, sorry. (Please help me on that, maybe I am in "questioning authority" mode and cannot flip out of it.) I cannot shake that feeling of being patronized all the time without being asked. The whole article, no - bunch of his posts, - reads like "we know better". Is it a Google thing, or heavy Rob Pike's influence? Yes, he is bringing some good points, and some very questionable ones. And for the questionable one (Compatibility section for example, or SAT solver question (nobody did it before?), or version naming, or paths for the sub-components...) it feels like he is trying to push explanation about his (and his team?) reasons and motivations behind these decisions, but... it is a monologue. There are more opinions and whole wide world of different software shops with own ideas and practices. 2. Go2 was a promised land, that later translated into "incremental, no code breaking" approach. Nowadays it feels like they are not even talking about it anymore! Looking at general history of programming languages, pretty much every language has had it's "C++" or Modula-2 moments. It is a natural process - an evolution - when unneeded parts disappear, and useful ideas added/extended. Amount of gotchas in Go is not zero, and to me (YMMV) it feels like Go2 would be (was?) a good time to shake off under-thought items, and enforce some new rules. Maybe even add interoperability between .go and .go2 files, maybe even semi-automatic upgrade path to run `go2 fmt ` on old files a la to Kotlin from Java - I don't know. But it feels like an opportunity and momentum for a "peaceful revolution" is lost. 3. Even with non-breaking changes style of change they have managed to break stuff. I, for one, still sit on 1.12 branch - because when I tried to upgrade to 1.13 (twice now), and our pure-Go code just plainly did not build with some obscure messages (I think it was linking or module cache errors). So, why pretend? Code rot is unavoidable - do we like it or not. 99. There are couple more points about chilling effect of "we know better" on a community as whole. (One example would be this whole "error" debacle that visible reduced activity on issue tracker and golang-nuts mailing lists after "hey all, we are going this way because we already went there". Immediately after it felt like people just gave up and left. "Empty town square" feeling.) It would be interesting to see this year results of "the state of Go" survey, but I don't think we will see a worrisome drops there -- the amount of new people coming in would easily override any opinions of people leaving. /end of rant, sorry for a long post

7y ago

sagichmal 7y ago

> If an old package and a new package have the same import path, the new package must be backwards compatible with the old package. Simply put, this rule is too strict, and biases modules' UX and semantics too far toward consumers. Not all packages are so mature, and consumed by enough consumers, that they should be forced by the language tooling to maintain backwards compatibility. That requirement creates a great deal of pointless work for the majority, perhaps vast majority, of package authors, who write small and uncertain packages for use by themselves or a small number of their immediate colleagues. For these authors, strict backwards compatibility is not only unnecessary, it's actually undesirable, as it creates needless toil. > The answer is that we learned from Dep that the general Bundler/Cargo/Dep approach includes some decisions that make software engineering more complex and more challenging. For a minority of use cases, where packages are large and mature and imported by many, many consumers -- sure, maybe. But the vast majority of Go code, written by private organizations and used internally, doesn't fit this description.

mfcl 7y ago
In that case, can you just stay in v0.x.y?
- akerl_ 7y ago
  You can either stay in v0.x.y, or you can just always bump the major version. Either option meets Go’s versioning requirements without requiring developers to think about “is this change backwards compatible” when they tag a new release.
IshKebab 7y ago
Nothing's physically stopping you breaking backwards compatibility without using a new import path if you want to. You might piss off your users but if they don't care then go for it.
- grey-area 7y ago
  Doesn’t the go tooling mandate this if you are >= v2.0? The rest of mod looks fine but tying paths to major version changes was a mistake IMO.

shadowgovt 7y ago

Spending my time as of late in npm's particular flavor of DLL hell, I appreciate the Go team taking a stab at this hard and persistent problem. I agree with other commenters' observations that the tooling hasn't caught up with the philosophy, but the philosophy seems on-point (and the key observation that as the dependencies of a project approach infinity, the odds of two branches of a dependency tree wanting to rely on two different versions of the same library approach 100% and there should be a sane way to address that need... npm does a 99% decent job of addressing this, but I've definitely hit issues with npm where objects generated by a newer dependency version have leaked into a codepath where an older version is calling the shots, and it's an absolute hellscape to debug).

kortex 7y ago
Npm seems to lack, in the context of the article, the ecosystem and community aspect. It's all like herding cats. Case in point, leftpad. * > as the dependencies of a project approach infinity, the odds of two branches of a dependency tree wanting to rely on two different versions of the same library approach 100% * With experience, I've found that this is precisely the case, with bounds much, much smaller than infinity. The ersatz standard lib can't even standardize, there's underscore and lodash. I think eventually the community will Ostwalt ripen in a sense, small packages will dissolve by attrition and big ones will grow. But the argument is strong for simply decreasing n = number of discrete packages.

mfer 7y ago

Side note, I just noticed that https://godoc.org https://godoc.org does not support Go modules with major versions being incremented (e.g., ending in `/v2`). The new pkg.go.dev does but not the existing godoc.org. This echos some of the other observations of tooling not being easily updated and still not being there, yet.

BarkMore 7y ago
pkg.go.dev is the module aware replacement for godoc.org.
- mfer 7y ago
  The tools, habits, and existing links haven't been updated. godoc.org don't point people to the new site. The "replacement" experience is a little broken.

nyc640 7y ago

I understand why this rubs some people the wrong way, but it really is consistent with Go's vision since the start. I don't think Russ is trying to imply that the issues they are facing when designing the language (dependency management, generics, etc.) are unique or impossible to solve in any way. I think he is simply saying that they are complex problems whose solutions introduce complexity to the language and ecosystem, and when presented with such a trade-off, the Go team has always preferred to lean towards simplicity, even if it means not implementing a feature that some (or even many) developers want/need. While I don't personally agree with many of the decisions they've made, I still respect the choices they've made and I think there is plenty of room for languages with competing philosophies. Because of Go's philosophy of not catering to every demand, it's never going to become the next Java or take over all of Software Engineering, but I think the Go team is fine with that as long as it continues to be useful as a tool for solving problems.

erikpukinskis 7y ago
The best code is no code, and the best solution is sometimes just not trying to solve the problem.
bmurphy1976 7y ago
They are building what they want. More power to them for making a tool that works for them first and not chasing every flight of fancy. I respect that philosophy because it's given me more of what I actually want and use in a platform than anything since Python. It's not for everybody but for those of us who really like it this is a relief from all the overcomplicated snake oil that's been shoved down our throats for the last 20 years.
ilovecaching 7y ago
C is a simple language and yet it constitutes the guts of most computers on this planet and Mars. Simple works. Go is really close to C. In fact, Rob Pike's Go CLI programs on his Github looks pretty much like a standard C program using getopt. Interfaces are just a nicer virtual function table. interface{} is just void *. In that sense, it's already treading ground that is proven to work. Go didn't have to do anything revolutionary because C was already great. I hope the Go team will continue to embody what made C great so the next generation can receive the lessons its taught us.