12 ms·

Experimental Dependency Vendoring in Go 1.5

174 points by NateDad 11y ago

wumbernang 11y ago
That's really nice. This is a royal PITA with .net and nuget and I've get to find a half decent solution that allows you to actually have the source rather than a bunch of symbol-less dll dependencies or a ridiculous build stack.
- romanovcode 11y ago
  Since MVC6/VS2015 beta you can actually get the source instead of dlls. However your build times will increase dramatically so it's a bad idea unless very needed.
  - wumbernang 11y ago
    Interesting - thanks for pointing it out. I was completely unaware of that. Going to go and play with it on my win10 test box now :)
NateDad 11y ago
I really like this proposal. It's basically the best of all current dependency tools - you get reproducibility without needing external configuration files, without hugely long vendor import statements, and still supporting raw go-get.
AYBABTME 11y ago
This is pretty much what we've been doing at DO, but we were dealing with GOPATH manipulation to achieve it. By having `GOPATH="third_party/:docode/"`, `go get` pulls packages into `third_party` automatically, and `go install` finds our binaries in `docode/src/services/foosvc/cmd/foosvcd`. Migrating to use `vendor/` instead of `third_party/` will be trivial, and not having to modify GOPATH will be nice. Having a GO_VENDOR=1 env var on top of the `go get -vendor` feature flag would be nice, so that the feature works with minimal disruption.
- deleted 11y ago
  [deleted]
- technoweenie 11y ago
  How do you handle a custom GOPATH like that when the current repository needs to be in there too? We've tried manually symlinking the repo dir into the custom '$GOPATH/src/github.com/user/repo' dir, but it feels really hacky.
  - anezvigin 11y ago
    You can separate multiple paths with a colon.
    - bradfitz 11y ago
      More specifically, with os.PathListSeparator. (colon on Unix, semicolon on Windows and \x00 on Plan 9)
  - lstoll 11y ago
    We just keep all our go code in one repo.
vruiz 11y ago
This is nice. Finally it seems like we are going somewhere. The call for proposals had not yielded anything groundbreaking so far. Tools like gb are nice but as Russ says it's not gogettable. This seems like a good compromise.
lobster_johnson 11y ago
Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder? In other words, you would still want something like godep or gb, because manually copying dependencies into a folder (or using git submodules, ugh), and then tracking their versions through their lifecycle, isn't a workable solution in most situations. This is clearly an improvement over "go get", but I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself. Go is so opinionated about so many things, and yet so little interested in having a strong opinion here. The Unix philosophy is great, to be sure, but the Unix philosophy is about increasing modularity through simplicity, and this problem is about modularity.
- vruiz 11y ago
  > I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself. They don't want to try solving a problem they don't really have. As RSC says: > the community understands and encounters these problems more than we on the Go team at Google do.
  - lobster_johnson 11y ago
    I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google. Even if your company doesn't need or want Bundler/Cargo-style dependency management doesn't mean that you are intellectually unfit to design out a good solution together with the community. Dogfooding is great and all, but it's actually possible to design technical solutions that you won't personally use.
    - sqs 11y ago
      > I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google. There are thousands of programming languages out there, yet Go has quite a bit of traction. They haven't forced anyone to use it, so there must be some reason for that. Maybe Go is successful because it was made with Google's technical workflow in mind.
deleted 11y ago
[deleted]
0xdeadbeefbabe 11y ago
A vendors B and C, but C vendors D and E. According to this proposal do you lay out the files like this: (A (vendor (B C D E)))? Edit: I guess HN tree markup is not working :)
- jooon 11y ago
  Yes. That would work. However, I also believe this will work: (A (vendor (B (C (vendor (D E)))))). https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU8dMKbcJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU... https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7FHipusJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7... https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS8RfjnEJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS...
  - jooon 11y ago
    and if you have both: (A (vendor (B (C (vendor (D E))) D E))) C would pick D E in its C/vendor before D E in A/vendor before D E in $GOPATH/src ps. I have edited these comments about 5 times to get the parentheses right. :)
deleted 11y ago
[deleted]
drewolson 11y ago
Am I understanding correctly that this proposal again side steps / ignores pinning dependencies for libraries? To me, this is the biggest current problem in the ecosystem and, as a library author, this doesn't solve any of my issues. I hope I misread somehow.
- enneff 11y ago
  With this scheme libraries can vendor their dependencies, if necessary.
  - drewolson 11y ago
    Can you reference the part of the proposal that states this? I'm reading the following points: > $GOPATH/src/foo will be imported as foo by non-main package $GOPATH/src/mypkg/p. > $GOPATH/src/mypkg/external/foo will be imported as foo by a non-main package p anywhere, when p is being built as a dependency of command $GOPATH/src/mypkg/c. It seems to me this is saying that libraries will use external packages only when built as a dependency of a main package. Again, this means library authors are out of luck.
    - jooon 11y ago
      There are two proposals. You seem to refer to the Google doc proposal that was linked to in the first message of the mailing list thread several months ago. The link on hackernews points to an answer by Google that was written just a few hours ago that instead "propose" the following: > If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
      - drewolson 11y ago
        Ah, thanks.
  - groks 11y ago
    You're pushing the Go ecosystem in this direction: https://spot.livejournal.com/312320.html
Lethalman 11y ago
How we solved it in Nix: http://lethalman.blogspot.it/2015/02/developing-in-golang-with-nix-package.html http://lethalman.blogspot.it/2015/02/developing-in-golang-wi... We don't need go get or any other tool, and we have about 174 packaged libraries in our repository.
ngrilly 11y ago
This is a great and necessary improvement. Being able to vendor dependencies without having to rewrite import paths is like having the best of both worlds. The proposed mechanism looks similar to what Node.js does with the node_modules directories: https://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders https://nodejs.org/api/modules.html#modules_loading_from_nod...

174 points by NateDad 11y ago

That's really nice. This is a royal PITA with .net and nuget and I've get to find a half decent solution that allows you to actually have the source rather than a bunch of symbol-less dll dependencies or a ridiculous build stack.

romanovcode 11y ago
Since MVC6/VS2015 beta you can actually get the source instead of dlls. However your build times will increase dramatically so it's a bad idea unless very needed.
- wumbernang 11y ago
  Interesting - thanks for pointing it out. I was completely unaware of that. Going to go and play with it on my win10 test box now :)

NateDad 11y ago

I really like this proposal. It's basically the best of all current dependency tools - you get reproducibility without needing external configuration files, without hugely long vendor import statements, and still supporting raw go-get.

AYBABTME 11y ago

This is pretty much what we've been doing at DO, but we were dealing with GOPATH manipulation to achieve it. By having `GOPATH="third_party/:docode/"`, `go get` pulls packages into `third_party` automatically, and `go install` finds our binaries in `docode/src/services/foosvc/cmd/foosvcd`. Migrating to use `vendor/` instead of `third_party/` will be trivial, and not having to modify GOPATH will be nice. Having a GO_VENDOR=1 env var on top of the `go get -vendor` feature flag would be nice, so that the feature works with minimal disruption.

deleted 11y ago
[deleted]
technoweenie 11y ago
How do you handle a custom GOPATH like that when the current repository needs to be in there too? We've tried manually symlinking the repo dir into the custom '$GOPATH/src/github.com/user/repo' dir, but it feels really hacky.
- anezvigin 11y ago
  You can separate multiple paths with a colon.
  - bradfitz 11y ago
    More specifically, with os.PathListSeparator. (colon on Unix, semicolon on Windows and \x00 on Plan 9)
- lstoll 11y ago
  We just keep all our go code in one repo.

vruiz 11y ago

This is nice. Finally it seems like we are going somewhere. The call for proposals had not yielded anything groundbreaking so far. Tools like gb are nice but as Russ says it's not gogettable. This seems like a good compromise.

lobster_johnson 11y ago

Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder? In other words, you would still want something like godep or gb, because manually copying dependencies into a folder (or using git submodules, ugh), and then tracking their versions through their lifecycle, isn't a workable solution in most situations. This is clearly an improvement over "go get", but I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself. Go is so opinionated about so many things, and yet so little interested in having a strong opinion here. The Unix philosophy is great, to be sure, but the Unix philosophy is about increasing modularity through simplicity, and this problem is about modularity.

vruiz 11y ago
> I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself. They don't want to try solving a problem they don't really have. As RSC says: > the community understands and encounters these problems more than we on the Go team at Google do.
- lobster_johnson 11y ago
  I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google. Even if your company doesn't need or want Bundler/Cargo-style dependency management doesn't mean that you are intellectually unfit to design out a good solution together with the community. Dogfooding is great and all, but it's actually possible to design technical solutions that you won't personally use.
  - sqs 11y ago
    > I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google. There are thousands of programming languages out there, yet Go has quite a bit of traction. They haven't forced anyone to use it, so there must be some reason for that. Maybe Go is successful because it was made with Google's technical workflow in mind.

deleted 11y ago

[deleted]

0xdeadbeefbabe 11y ago

A vendors B and C, but C vendors D and E. According to this proposal do you lay out the files like this: (A (vendor (B C D E)))? Edit: I guess HN tree markup is not working :)

jooon 11y ago
Yes. That would work. However, I also believe this will work: (A (vendor (B (C (vendor (D E)))))). https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU8dMKbcJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU... https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7FHipusJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7... https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS8RfjnEJ https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS...
- jooon 11y ago
  and if you have both: (A (vendor (B (C (vendor (D E))) D E))) C would pick D E in its C/vendor before D E in A/vendor before D E in $GOPATH/src ps. I have edited these comments about 5 times to get the parentheses right. :)

deleted 11y ago

[deleted]

drewolson 11y ago

Am I understanding correctly that this proposal again side steps / ignores pinning dependencies for libraries? To me, this is the biggest current problem in the ecosystem and, as a library author, this doesn't solve any of my issues. I hope I misread somehow.

enneff 11y ago
With this scheme libraries can vendor their dependencies, if necessary.
- drewolson 11y ago
  Can you reference the part of the proposal that states this? I'm reading the following points: > $GOPATH/src/foo will be imported as foo by non-main package $GOPATH/src/mypkg/p. > $GOPATH/src/mypkg/external/foo will be imported as foo by a non-main package p anywhere, when p is being built as a dependency of command $GOPATH/src/mypkg/c. It seems to me this is saying that libraries will use external packages only when built as a dependency of a main package. Again, this means library authors are out of luck.
  - jooon 11y ago
    There are two proposals. You seem to refer to the Google doc proposal that was linked to in the first message of the mailing list thread several months ago. The link on hackernews points to an answer by Google that was written just a few hours ago that instead "propose" the following: > If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
    - drewolson 11y ago
      Ah, thanks.
- groks 11y ago
  You're pushing the Go ecosystem in this direction: https://spot.livejournal.com/312320.html

Lethalman 11y ago

How we solved it in Nix: http://lethalman.blogspot.it/2015/02/developing-in-golang-with-nix-package.html http://lethalman.blogspot.it/2015/02/developing-in-golang-wi... We don't need go get or any other tool, and we have about 174 packaged libraries in our repository.

ngrilly 11y ago

This is a great and necessary improvement. Being able to vendor dependencies without having to rewrite import paths is like having the best of both worlds. The proposed mechanism looks similar to what Node.js does with the node_modules directories: https://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders https://nodejs.org/api/modules.html#modules_loading_from_nod...