3 ms·

I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means but I still don't understand. What problem does it solve? Why is it needed? Is the

by niftich 10y ago

I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means but I still don't understand. What problem does it solve? Why is it needed? Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for someone who is used to other packaging systems that may not be entirely elegant but work pretty okay most of the time, like Java's Maven or Node's npm' [1] https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ [2] https://github.com/golang/go/wiki/PackageManagementTools https://github.com/golang/go/wiki/PackageManagementTools [3] https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia6NkmB024MUa4T3SJeh8 https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia...

grey-area 10y ago
It solves the problem of reproducible builds when using third party code. You can record state (v 1.3 of pkg foo) or check in code (vendor). It only really gets complex on big projects with lots of dependencies which themselves have dependencies.
- niftich 10y ago
  After more searching, I found a StackOverflow question asking the same thing [1], and the answer appears to be: "Vendoring is the act of making your own copy of the 3rd party packages your project is using. Those copies are traditionally placed inside each project and then saved in the project repository." Being familiar with both Maven and npm, in those communities, this act of checking dependencies into source control is a controversial and hotly-debated topic, but in those package managers is more often than not (please bear with my point) considered an anti-pattern, because both of them provide solutions [2][3][4], like a project's declarative list of dependencies along with a baseline (but often, overridable) conflicting-version-resolution algorithm. Of course, the more I understand the issue, the more questions I have. Why does Go have multiple, competing package managers? Some, like Glide [5] (without judgements about all the others), encourage the type of declarative version management I quoted. Why does Go even ship with built-in package fetcher 'go get'? Why would anyone ever build a package fetcher for a publicly-released ecosystem that has no concept of versions or commit-IDs or content-adressed-hashes, or some other mechanism of tracking state to something that's as inherently mutable as a software package? I have no desire to point fingers, and I don't want to encourage bikeshedding, and I don't want to turn this sub-thread into an in-band debate, as I'm certain it's been done (over and over again) in the past, I just simply feel that 'go get' is fundamentally broken in its understanding of how people in the wild actually write software. I'm suddenly beginning to understand why there are concerted efforts [6][7][8] to address this issue in a community-led, officially-sanctioned way. [1] http://stackoverflow.com/questions/35109393/ http://stackoverflow.com/questions/35109393/ [2] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html http://maven.apache.org/guides/introduction/introduction-to-... [3] https://github.com/ning/maven-dependency-versions-check-plugin https://github.com/ning/maven-dependency-versions-check-plug... [4] http://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/ http://bytearcher.com/articles/semver-explained-why-theres-a... [5] https://github.com/Masterminds/glide https://github.com/Masterminds/glide [6] https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8 https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsW... [7] https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy... [8] https://github.com/sdboyer/gps/wiki/Rationale https://github.com/sdboyer/gps/wiki/Rationale
  - grey-area 10y ago
    go get does what it says and no more. It's sufficient for many large companies, esp with monorepos. Vendoring also works just fine for many uses and has the virtue of making dependencies explicit. Pkg management is useful if you have a large ecosystem of rapidly changing libraries with many fragile dependencies which overlap. If can encourage overuse of external code though (eg left-pad). It is not a panacea or the one true way.
twotwotwo 10y ago
It's common practice for folks building Go binaries (but not libraries) to check in the code for their dependencies. So whenever you build Docker, you get the same thing unless Docker made a change (either by changing Docker itself or explicitly checking in an upgrade to a dep). There are a few nice things about it; vendoring means you won't have a dependency on other repos or archives being available, and if you're a binary maintainer you can see source changes in your own code and that of your dependencies side-by-side if you're, say, investigating a bug that came up between certain revisions. The not-so-nice things are pretty much what you'd think. There isn't a command to say "give me upgrades, but not ones that the author flagged as breaking changes". If you're writing a _library_ that is depended on by other _libraries_ there is no one conventional way to safely do breaking changes, so there's a lot of effort to avoid those changes plus sometimes hacks like different repo paths for different versions. Part of the history is that, internally, Google's got one version of third-party code in the latest rev of their source control tree, by company policy. You upgrade something for one, you upgrade it for the whole codebase. Sounds daunting, but they have lots of testing, canary servers, and so on to catch problems. There is a great talk about some of this by Rachel Potvin (can't recall if it covers the third_party/ stuff specifically): https://www.youtube.com/watch?v=W71BTkUbdqE https://www.youtube.com/watch?v=W71BTkUbdqE So Go's initial model didn't consider multiple versions of packages because Google had approached upgrades a different way internally. Of course, unlike Google, Go users on the outside will never have any central mechanism to keep all the world in sync when there are API changes. Hence the conversation today about package management.
- twblalock 10y ago
  I think another reason they didn't consider proper versioning is that only a few of the Go core developers have a background in languages with a good dependency management ecosystem. C and C++ have never really had that, and Python never really has either, despite what fans of it seem to think. I think the bottom line is that the Go team simply doesn't know how good the ecosystem is in a lot of newer languages. I see this reflected in other decisions in Go as well, such as the absurd amount of boilerplate needed to manage some of the Goroutines, and the way string are handled.
deleted 10y ago
[deleted]

by niftich 10y ago

grey-area 10y ago

It solves the problem of reproducible builds when using third party code. You can record state (v 1.3 of pkg foo) or check in code (vendor). It only really gets complex on big projects with lots of dependencies which themselves have dependencies.

niftich 10y ago
After more searching, I found a StackOverflow question asking the same thing [1], and the answer appears to be: "Vendoring is the act of making your own copy of the 3rd party packages your project is using. Those copies are traditionally placed inside each project and then saved in the project repository." Being familiar with both Maven and npm, in those communities, this act of checking dependencies into source control is a controversial and hotly-debated topic, but in those package managers is more often than not (please bear with my point) considered an anti-pattern, because both of them provide solutions [2][3][4], like a project's declarative list of dependencies along with a baseline (but often, overridable) conflicting-version-resolution algorithm. Of course, the more I understand the issue, the more questions I have. Why does Go have multiple, competing package managers? Some, like Glide [5] (without judgements about all the others), encourage the type of declarative version management I quoted. Why does Go even ship with built-in package fetcher 'go get'? Why would anyone ever build a package fetcher for a publicly-released ecosystem that has no concept of versions or commit-IDs or content-adressed-hashes, or some other mechanism of tracking state to something that's as inherently mutable as a software package? I have no desire to point fingers, and I don't want to encourage bikeshedding, and I don't want to turn this sub-thread into an in-band debate, as I'm certain it's been done (over and over again) in the past, I just simply feel that 'go get' is fundamentally broken in its understanding of how people in the wild actually write software. I'm suddenly beginning to understand why there are concerted efforts [6][7][8] to address this issue in a community-led, officially-sanctioned way. [1] http://stackoverflow.com/questions/35109393/ http://stackoverflow.com/questions/35109393/ [2] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html http://maven.apache.org/guides/introduction/introduction-to-... [3] https://github.com/ning/maven-dependency-versions-check-plugin https://github.com/ning/maven-dependency-versions-check-plug... [4] http://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/ http://bytearcher.com/articles/semver-explained-why-theres-a... [5] https://github.com/Masterminds/glide https://github.com/Masterminds/glide [6] https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8 https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsW... [7] https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy... [8] https://github.com/sdboyer/gps/wiki/Rationale https://github.com/sdboyer/gps/wiki/Rationale
- grey-area 10y ago
  go get does what it says and no more. It's sufficient for many large companies, esp with monorepos. Vendoring also works just fine for many uses and has the virtue of making dependencies explicit. Pkg management is useful if you have a large ecosystem of rapidly changing libraries with many fragile dependencies which overlap. If can encourage overuse of external code though (eg left-pad). It is not a panacea or the one true way.

twotwotwo 10y ago

It's common practice for folks building Go binaries (but not libraries) to check in the code for their dependencies. So whenever you build Docker, you get the same thing unless Docker made a change (either by changing Docker itself or explicitly checking in an upgrade to a dep). There are a few nice things about it; vendoring means you won't have a dependency on other repos or archives being available, and if you're a binary maintainer you can see source changes in your own code and that of your dependencies side-by-side if you're, say, investigating a bug that came up between certain revisions. The not-so-nice things are pretty much what you'd think. There isn't a command to say "give me upgrades, but not ones that the author flagged as breaking changes". If you're writing a _library_ that is depended on by other _libraries_ there is no one conventional way to safely do breaking changes, so there's a lot of effort to avoid those changes plus sometimes hacks like different repo paths for different versions. Part of the history is that, internally, Google's got one version of third-party code in the latest rev of their source control tree, by company policy. You upgrade something for one, you upgrade it for the whole codebase. Sounds daunting, but they have lots of testing, canary servers, and so on to catch problems. There is a great talk about some of this by Rachel Potvin (can't recall if it covers the third_party/ stuff specifically): https://www.youtube.com/watch?v=W71BTkUbdqE https://www.youtube.com/watch?v=W71BTkUbdqE So Go's initial model didn't consider multiple versions of packages because Google had approached upgrades a different way internally. Of course, unlike Google, Go users on the outside will never have any central mechanism to keep all the world in sync when there are API changes. Hence the conversation today about package management.

twblalock 10y ago
I think another reason they didn't consider proper versioning is that only a few of the Go core developers have a background in languages with a good dependency management ecosystem. C and C++ have never really had that, and Python never really has either, despite what fans of it seem to think. I think the bottom line is that the Go team simply doesn't know how good the ecosystem is in a lot of newer languages. I see this reflected in other decisions in Go as well, such as the absurd amount of boilerplate needed to manage some of the Goroutines, and the way string are handled.

deleted 10y ago

[deleted]