10 ms·

Reproducible Builds in Go

233 points by Spiritus 11y ago

vanmount 11y ago
I'm very excited about GB. GVM made working on multiple projects already a lot easier, but dependencies are still a huge pain... I don't know if GB is the best answer to the problem, but at least it's a big step forward and I'll definitely try it out.
GutenYe 11y ago
yet another try......
- deleted 11y ago
  [deleted]
- _ak 11y ago
  I attended the talk. Wait for the video, I personally found the Dave's arguments and reasoning convincing.
deleted 11y ago
[deleted]
cakoose 11y ago
On slide 9 it says that enhancing the syntax for "import" would be a backwards-incompatible syntax change. Sure, old compilers wouldn't accept newer code, but newer compilers would accept old code without breaking the behavior. Is Go's definition of "backwards compatible" different from the usual one?
- laumars 11y ago
  The issue is more with older code not compiling on newer compilers if the syntax is changed. Go works on the basis of no breaking changes between major versions. I guess theoretically it could be changed in such a way that both the older and newer syntax would be valid on newer compilers, but that adds additional complexity which is also an anti-Go idiom. So sadly we're left with two alternatives: 1) code rewriting / generation, or 2) 3rd party compilers.
  - cakoose 11y ago
    But it looks like Go has done similar things in the past. For example, the slicing operation gained an additional optional argument (https://golang.org/doc/go1.2#three_index https://golang.org/doc/go1.2#three_index). It's not exactly the same thing, but it seems roughly in the same league of difficulty. Based on what I'm reading, versioning is a serious pain point. Sure, you always have to make sure the payoff is worth the additional complexity (universally, not just in Go) but the author seems to categorically disqualify this option by incorrectly saying that it would break backwards compatibility.
    - laumars 11y ago
      > But it looks like Go has done similar things in the past. For example, the slicing operation gained an additional optional argument (https://golang.org/doc/go1.2#three_index https://golang.org/doc/go1.2#three_index). It's not exactly the same thing, but it seems roughly in the same league of difficulty. It's not really same in my opinion. Slices already accepted multiple parameters - or even none. So adding another optional parameter doesn't affect the syntax. Where as import names are all stored in a single quoted string, so you'd have to rethink the entire structure of the import syntax (which leaves you with two import different methods of expressing imports - which is messy) or use some nasty kludge of including versioning within the import string. What's more versioning is only half the story here. There's other issues with Go's imports that this author aims to resolve - such as the dependency on a static repo. Because import paths are based directly on the repo path (eg github.com) it makes it a pain to prototype code in a private repository before pushing changes public. And then there's issues with dependency on any specific resource (you can't use mirrored repos with go get), plus a whole slew of related issues regarding hard-coded repo paths. So versioning is only half the story. > Based on what I'm reading, versioning is a serious pain point. Sure, you always have to make sure the payoff is worth the additional complexity (universally, not just in Go) but the author seems to categorically disqualify this option by incorrectly saying that it would break backwards compatibility. Well I've already argued that any clean solution would break backwards compatibility and any kludge would be messy and rather short sighted (plus potentially may not address the other issues with go get and import()). So I think it's a little misjudged for you to say he's "incorrect" in his opinion in the way that you have.
- politician 11y ago
  Slightly OT, but it's kind of strange to suggest enhancements to the import statement when the spec explicitly states that an import statement contains an opaque string. It's the go build tool that add semantics to the values of the import statement, not the language.
stormbrew 11y ago
If you're going to vendor everything anyways, I don't really see why you need a new tool. But IANAG, so maybe I missed something?
- davecheney 11y ago
  It's vendoring without rewriting, http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#25 http://go-talks.appspot.com/github.com/davecheney/presentati...
humanfromearth 11y ago
TL;DR: Don't use `go get` if you want reproducible builds. It's early for me to tell, but looks like gb is something like a mix between a gvm (virtual env) and godep (vendorizing). I'm skeptical about things like "don't use the standard, use my thing", but I will try it anyway.
- sagichmal 11y ago
  The author has reasonable credibility/authority in the community. Definitely worth a look.
AnkhMorporkian 11y ago
I love the presentation, but the GIL problem and Go's package problems aren't really even comparable. The GIL is an unfortunate consequence of the CPython implementation that can't really be solved without some serious hacks or computationally expensive workarounds. The Go "problem" is almost entirely an implementation problem; something evidenced by a simple fixed implementation. Honestly, the entire presentation was nearly lost on me by the weird comparison at the beginning. I recovered, but seriously, very strange comparison for anyone who understands what the GIL represents. Edit: And really, was github the best target to go to for availability when it came to developer downtime on the edited XKCD? Github's downtime is pretty near 0. I think I remember roughly an hour of downtime in 2014 in the super early morning, and nothing before that until 2012 maybe.
- deleted 11y ago
  [deleted]
- davecheney 11y ago
  Everyone knows the GIL as Python's big problem; dependency management is Go's big problem -- it's an analogy, that's it, nothing more.
- davecheney 11y ago
  > And really, was github the best target to go to for availability when it came to developer downtime on the edited XKCD? Github's downtime is pretty near 0. I think I remember roughly an hour of downtime in 2014 in the super early morning, and nothing before that until 2012 maybe. There was something about 2 days of DDOS earlier this year, but again the specifics aren't important -- the important part is, if you are in charge of delivering a product written in Go, you're going to look like an ass if your build fails because some random part of the internet is having a bad day.
- shadowmint 11y ago
  To be fair, the comparison isn't completely off. The GIL was an implementation detail that was initially considered not a particularly big deal, but turned out to be a Really Big Deal later because no one thought about the problem from the beginning. The lack of a package ecosystem for go and 'go get is good enough' is exactly the same; it was for quite some time simply considered to not really be a big problem. ...but it turns out, when you're doing complicated things, not having repeatable builds really is a Big Problem. ...but yes, they're in different leagues. Solving this one won't be anywhere near as troublesome.
caiusdurling 11y ago
Reminds me of developing rails apps way back in the day before RVM gemsets or bundler came along. You'd vendor all the gems (& specific version for your app) into `vendor/gems` ala http://nubyonrails.com/articles/2005/12/22/freeze-other-gems-to-rails-lib-directory http://nubyonrails.com/articles/2005/12/22/freeze-other-gems...
- vidarh 11y ago
  I still vendor all the gems, only I use Bundler to do it. It's pretty much crazy not to, in my opinion. Stuff gets removed, and networks are unreliable, so locking/resolving dependencies to specific versions is insufficient.
  - caiusdurling 11y ago
    Sorry yes, I meant it reminded me of vendoring gems _by hand_. I think everyone is vendoring gems via Bundler these days, because it would be insanity not to!
endymi0n 11y ago
Sad to say so, but this really makes a lot of sense - and it's dead simple, something I really appreciate as a Gopher. Suddenly you're able to have full control over a project, use private repos effortlessly (even with SSH protocol) and most importantly for me, not check in all the dependencies and their changes like with Godep, which just keeps up messing up the history although having nothing to do with the project. Just Git submodules and you're done.
- sagichmal 11y ago
  > Just Git submodules and you're done. It would be a disaster if popular Go projects started using submodules, and consequently required users and contributors to play the submodule init/update/sync dance. Please, please -- prefer subtrees.
  - davecheney 11y ago
    gb doesn't mandate any particular DVCS or DVCS strategy -- I understand it's a deeply personal issue for most. People are smart, if they like the tool, they'll figure out how to integrate it into whatever DVCS they use.
  - ansible 11y ago
    I prefer subtrees to submodules. Not that subtrees are a cakewalk, in fact I wrote up a tutorial using them. [1] The main issue with submodules (for me anyway) is that it seems to be all too easy to revert a change in a submodule if you aren't careful. It is not very people-friendly because you're just looking at hashes, and it isn't clear which one is newer and which is older. [1] https://github.com/jamesgraves/example-go-app https://github.com/jamesgraves/example-go-app
  - DannoHung 11y ago
    Submodules should be the right solution. Subtrees are weird as hell even if they require less interaction. I wonder if anyone in the git team has considered doing any updates on the porcelain for them. edit: BTW, even though syncs may be separate (and let's be honest, they should be really infrequent). You can do the update/init thing in one command: git submodule update --init --recursive
deleted 11y ago
[deleted]
ghodss 11y ago
One major limitation of this approach is that any project that wishes to vendor or lock their dependencies can no longer be used as a dependency for another project. From the gb GitHub: > A project is the consumer of your own source code, and possibly dependencies that your code consumes; nothing consumes the code from a project. This seems to imply that any code outside of a project (i.e. the code inside vendor/src) has no recourse for indicating the versions of their dependencies. This is nice in that it simplifies the problem, but to completely remove the ability for any and all libraries to indicate the versions of their dependencies seems unnecessarily restrictive. If I build a library for others to use, and I have a dependency, I want to be be able to lock to a specific version, or at least give my preference for a version. Of course, this creates its own issues - what do you do when two libraries depend on two different versions of the same library? (Also known as the diamond dependency problem.) This is where the Go culture helps, where as long as you pick a later version, things are likely to work. But I'd rather have the tooling let me detect the two versions that the two libraries want, show that there is a mismatch, and give me the ability to override and pick one (probably the later one). Instead, the gb approach eliminates the ability for the libraries to even have the ability to indicate what version they would prefer, which makes it even more difficult to get a bunch of libraries that share dependencies to work correctly together. godep (https://github.com/tools/godep https://github.com/tools/godep) seems to have the best compromise: vendor dependencies without path rewriting (though with GOPATH rewriting), but also keep track of their versions in a Godep.json file. You can gracefully pick between conflicting versions upstream if need be.
- deleted 11y ago
  [deleted]
- eikenberry 11y ago
  +1 for godep w/o import path rewriting. It isn't addressed in the presentation at all, only using it with path rewriting is mention and all the problems associated with it are with the path rewriting. IMO it is the best solution right now with just 1 issue, that it is not included with go. This is a pain with CI systems (e.g. Jenkins) where you have plugins to provide go itself, but have to figure out a way to get godep around to run your build. Right now I'm punting and just doing a go-get godep, then using it to build my project. I'm not happy with that though.
davecheney 11y ago
Hi HN, There was a video recorded at this meetup, but I'm not in control of that. In the mean time, the getting started document [1] is the contents of the "demo time" slide. 1. https://github.com/constabulary/gb/blob/master/getting-started.md https://github.com/constabulary/gb/blob/master/getting-start...
- joewalnes 11y ago
  Dave thanks for doing this. Not just for building the tool but for a very well thought out and convincing presentation. I'm completely with you. My own solution has similar ideals (per project versioned deps) though a different solution. https://github.com/joewalnes/go-getter/blob/master/README.md https://github.com/joewalnes/go-getter/blob/master/README.md. Honestly I'm just glad others are finally talking about the elephant in the room.
- politician 11y ago
  Thanks for building this; I completely agree that the `go` build tool is the problem, but settled on Goop as a stop-gap solution.
shocks 11y ago
It'd be nice if we could somehow use semver. I'd like to get patches automatically... edit; if you downvoted, maybe comment and explain why?
aikah 11y ago
Yet another problem the go team refuses to see. Either the go team says "not our problem" then people are stuck with doing things multiple and incompatibles ways , or the go team says "we already solve the problem, vendoring" and the go team doesnt understand the problem at first place. That's the recurrent pattern with the Go team. Everything cannot be dealt with in userland, or you'll end up with fragmentation. And No , vendoring doesn't solve the problem.
- davecheney 11y ago
  Honest question: Why does vendoring not solve the problem ?
  - kungfooguru 11y ago
    I bet it works great for Google or anyone else who keeps everything in one big repo and isn't developing open source applications.
    - deleted 11y ago
      [deleted]
    - Titanous 11y ago
      Vendoring works great for us, we keep everything in one big repo and it's open source. The problem is that the current tooling for managing this is horrible.
      - nulltype 11y ago
        What is horrible about managing it? The biggest problem I encountered with vendoring is getting rid of git submodules.
        Titanous 11y ago
        We currently use godep, and it breaks very easily due to tags and differing architectures and OSes (among other things). In addition to being unreliable, it's pretty slow. I'm not aware of any other tools that do import rewriting+vendoring and work better than godep.
Aissen 11y ago
Got my hopes up for deterministic builds, but then I read on slide 5: Out of scope: compiler doesn't produce byte for byte comparable binaries
- davecheney 11y ago
  There are others working on that problem. I probably shouldn't have mentioned it; if I hadn't, you wouldn't have thought about it, and then nobody would have had their hopes dashed.
  - Aissen 11y ago
    English isn't my first language, so when I read "reproducible" I thought "deterministic"; I also didn't think you'd be working on the "vendoring" problem since I thought it was a solved issue (which it isn't for all the reasons in the post). Anyways, thanks for your work.
    - davecheney 11y ago
      I think the compilers that produce elf output are almost deterministic, at least when producing a fully static binary (no CGO). I don't think that holds for mach-o or PE binaries, but I'm really speaking off the cuff here, as I said, others care about this and are working on it separately.
    - ComputerGuru 11y ago
      However, English is my first language, and I thought exactly the same.
  - silon3 11y ago
    Does JIT (or late/deploy time optimize/link) help here? Would it be easier to make deterministic builds when optimizer is not involved?
    - davecheney 11y ago
      Nope, this is about the source that goes into the compiler.
humanfromearth 11y ago
@davecheney Thanks for your work. If I publish my project on github and use gb with it, then people who want to contribute have to install gb as well. Do you think that in the future there is a chance to do the building of a gb project without the gb itself?
- deleted 11y ago
  [deleted]
- davecheney 11y ago
  Maybe, but it's not something I'm focusing on.
eternalban 11y ago
Go would hugely benefit from a hierarchical /.go in $GOPATH. This would allow for a robust versioning scheme, and allow the language to scale in the future.
- davecheney 11y ago
  Can you explain your idea in more detail ?
  - eternalban 11y ago
    Think /.git, Dave. /.go vendors # map simple import name to explicit (ugly) path ... # other stuff for additional e.g. generate features Hierarchical: Think OO-ish shadowing of .go directives and metainfo. Is this clear?
    - davecheney 11y ago
      > Is this clear? not really
      - eternalban 11y ago
        # ---------------------------------- # general layout of the land # ---------------------------------- $GOPATH/src/.go $GOPATH/src/.go/config $GOPATH/src/.go/vendors # for all projects # ---------------------------------- # $GOPATH/.go/vendors sketch # ---------------------------------- lib/mq = github.com/lib/pq@master mgo = labix.org/v2/mgo ... etc # ---------------------------------- # specific project 1 # uses the global version of package labix.org/mgo # uses a project specific package hotness # ---------------------------------- $GOPATH/src/mygofoo $GOPATH/src/mygofoo/foo.go $GOPATH/src/mygofoo/.go $GOPATH/src/mygofoo/.go/vendors # ---------------------------------- # foo.go fragment # ---------------------------------- package mygofoo import ( "mgo" // version spec'd in global .go "lib/pq" // version spec'd in project specific .go "hotness" // version spec'd in project specific .go ... ) ... # ---------------------------------- # $GOPATH/src/mygofoo/.go/vendors sketch # ---------------------------------- lib/mq = github.com/lib/pq@experimental // lets pretend hotness = bitbucket.org/wiz/hotness@master ... etc # go tool > cd $GOPATH/src/mygofoo > go build -vendors using: mgo -> labix.org/v2/mgo lib/pq -> github.com/lib/pq@experimental hotness -> bitbucket.org/wiz/hotness@master > # go tool can obviously allow for command line mods of the .go
iddqd 11y ago
I think I'll stick to using submodules and a Makefile with GOPATH set to the vendor directory. Hasn't failed me yet.
drewolson 11y ago
How is this different from godep[1] other than being the root of its own GOPATH and having a different naming convention for the location of vendored dependencies? [1] - https://github.com/tools/godep https://github.com/tools/godep
ahmetmsft 11y ago
Can somebody please share a mirror? App Engine instance seems like it has exceeded the quota. (lol)
YZF 11y ago
The link isn't working for me. What we do is keep a copy of all packages we use and then assemble everything into a workspace as part of the automated build. Nothing is pulled from GitHub at build time. All those copies are versioned so you can have different projects use different versions. The dependencies are specified via some pre-existing build infrastructure but essentially there is one file in the project listing all the dependencies and where to find the packages. Developers can still have their own personal workspace and build with different versions but the "real" build is always reproducible. I think many people using Go or wanting to use Go in commercial/production environments are finding a little bit of impedance mismatch between the Go view of the world (the workspace and packages) and what their existing tooling does. It's not too terrible to deal with but it would be nice if we had a little more flexibility. go get could support some nicer/automated version of the above where you have the ability to download/mirror packages to a shared drive or to another source control system and pull from there to your workspace based on specific dependency versions. This is fairly minor friction though.
tshannon 11y ago
Might be overkill, but for my large projects, I simply host my own gogs [1] instance and vendor in all my dependencies to that instance. I get reproducible builds, and I can bring in any updates I want manually when I'm ready for me. 1.http://gogs.io/ http://gogs.io/
pas256 11y ago
The number of parallels between Go and Ruby keeps me entertained. Isn't this just rvm and bundler all over again?
- duaneb 11y ago
  Well, thankfully there's nothing like the nightmare 1.8.6 -> 1.9 switch, and it has loads of tools working with static code that ruby could never have.

233 points by Spiritus 11y ago

I'm very excited about GB. GVM made working on multiple projects already a lot easier, but dependencies are still a huge pain... I don't know if GB is the best answer to the problem, but at least it's a big step forward and I'll definitely try it out.

GutenYe 11y ago

yet another try......

deleted 11y ago
[deleted]
_ak 11y ago
I attended the talk. Wait for the video, I personally found the Dave's arguments and reasoning convincing.

deleted 11y ago

[deleted]

cakoose 11y ago

On slide 9 it says that enhancing the syntax for "import" would be a backwards-incompatible syntax change. Sure, old compilers wouldn't accept newer code, but newer compilers would accept old code without breaking the behavior. Is Go's definition of "backwards compatible" different from the usual one?

laumars 11y ago
The issue is more with older code not compiling on newer compilers if the syntax is changed. Go works on the basis of no breaking changes between major versions. I guess theoretically it could be changed in such a way that both the older and newer syntax would be valid on newer compilers, but that adds additional complexity which is also an anti-Go idiom. So sadly we're left with two alternatives: 1) code rewriting / generation, or 2) 3rd party compilers.
- cakoose 11y ago
  But it looks like Go has done similar things in the past. For example, the slicing operation gained an additional optional argument (https://golang.org/doc/go1.2#three_index https://golang.org/doc/go1.2#three_index). It's not exactly the same thing, but it seems roughly in the same league of difficulty. Based on what I'm reading, versioning is a serious pain point. Sure, you always have to make sure the payoff is worth the additional complexity (universally, not just in Go) but the author seems to categorically disqualify this option by incorrectly saying that it would break backwards compatibility.
  - laumars 11y ago
    > But it looks like Go has done similar things in the past. For example, the slicing operation gained an additional optional argument (https://golang.org/doc/go1.2#three_index https://golang.org/doc/go1.2#three_index). It's not exactly the same thing, but it seems roughly in the same league of difficulty. It's not really same in my opinion. Slices already accepted multiple parameters - or even none. So adding another optional parameter doesn't affect the syntax. Where as import names are all stored in a single quoted string, so you'd have to rethink the entire structure of the import syntax (which leaves you with two import different methods of expressing imports - which is messy) or use some nasty kludge of including versioning within the import string. What's more versioning is only half the story here. There's other issues with Go's imports that this author aims to resolve - such as the dependency on a static repo. Because import paths are based directly on the repo path (eg github.com) it makes it a pain to prototype code in a private repository before pushing changes public. And then there's issues with dependency on any specific resource (you can't use mirrored repos with go get), plus a whole slew of related issues regarding hard-coded repo paths. So versioning is only half the story. > Based on what I'm reading, versioning is a serious pain point. Sure, you always have to make sure the payoff is worth the additional complexity (universally, not just in Go) but the author seems to categorically disqualify this option by incorrectly saying that it would break backwards compatibility. Well I've already argued that any clean solution would break backwards compatibility and any kludge would be messy and rather short sighted (plus potentially may not address the other issues with go get and import()). So I think it's a little misjudged for you to say he's "incorrect" in his opinion in the way that you have.
politician 11y ago
Slightly OT, but it's kind of strange to suggest enhancements to the import statement when the spec explicitly states that an import statement contains an opaque string. It's the go build tool that add semantics to the values of the import statement, not the language.

stormbrew 11y ago

If you're going to vendor everything anyways, I don't really see why you need a new tool. But IANAG, so maybe I missed something?

davecheney 11y ago
It's vendoring without rewriting, http://go-talks.appspot.com/github.com/davecheney/presentations/reproducible-builds.slide#25 http://go-talks.appspot.com/github.com/davecheney/presentati...

humanfromearth 11y ago

TL;DR: Don't use `go get` if you want reproducible builds. It's early for me to tell, but looks like gb is something like a mix between a gvm (virtual env) and godep (vendorizing). I'm skeptical about things like "don't use the standard, use my thing", but I will try it anyway.

sagichmal 11y ago
The author has reasonable credibility/authority in the community. Definitely worth a look.

AnkhMorporkian 11y ago

I love the presentation, but the GIL problem and Go's package problems aren't really even comparable. The GIL is an unfortunate consequence of the CPython implementation that can't really be solved without some serious hacks or computationally expensive workarounds. The Go "problem" is almost entirely an implementation problem; something evidenced by a simple fixed implementation. Honestly, the entire presentation was nearly lost on me by the weird comparison at the beginning. I recovered, but seriously, very strange comparison for anyone who understands what the GIL represents. Edit: And really, was github the best target to go to for availability when it came to developer downtime on the edited XKCD? Github's downtime is pretty near 0. I think I remember roughly an hour of downtime in 2014 in the super early morning, and nothing before that until 2012 maybe.

deleted 11y ago
[deleted]
davecheney 11y ago
Everyone knows the GIL as Python's big problem; dependency management is Go's big problem -- it's an analogy, that's it, nothing more.
davecheney 11y ago
> And really, was github the best target to go to for availability when it came to developer downtime on the edited XKCD? Github's downtime is pretty near 0. I think I remember roughly an hour of downtime in 2014 in the super early morning, and nothing before that until 2012 maybe. There was something about 2 days of DDOS earlier this year, but again the specifics aren't important -- the important part is, if you are in charge of delivering a product written in Go, you're going to look like an ass if your build fails because some random part of the internet is having a bad day.
shadowmint 11y ago
To be fair, the comparison isn't completely off. The GIL was an implementation detail that was initially considered not a particularly big deal, but turned out to be a Really Big Deal later because no one thought about the problem from the beginning. The lack of a package ecosystem for go and 'go get is good enough' is exactly the same; it was for quite some time simply considered to not really be a big problem. ...but it turns out, when you're doing complicated things, not having repeatable builds really is a Big Problem. ...but yes, they're in different leagues. Solving this one won't be anywhere near as troublesome.

caiusdurling 11y ago

Reminds me of developing rails apps way back in the day before RVM gemsets or bundler came along. You'd vendor all the gems (& specific version for your app) into `vendor/gems` ala http://nubyonrails.com/articles/2005/12/22/freeze-other-gems-to-rails-lib-directory http://nubyonrails.com/articles/2005/12/22/freeze-other-gems...

vidarh 11y ago
I still vendor all the gems, only I use Bundler to do it. It's pretty much crazy not to, in my opinion. Stuff gets removed, and networks are unreliable, so locking/resolving dependencies to specific versions is insufficient.
- caiusdurling 11y ago
  Sorry yes, I meant it reminded me of vendoring gems _by hand_. I think everyone is vendoring gems via Bundler these days, because it would be insanity not to!

endymi0n 11y ago

Sad to say so, but this really makes a lot of sense - and it's dead simple, something I really appreciate as a Gopher. Suddenly you're able to have full control over a project, use private repos effortlessly (even with SSH protocol) and most importantly for me, not check in all the dependencies and their changes like with Godep, which just keeps up messing up the history although having nothing to do with the project. Just Git submodules and you're done.

sagichmal 11y ago
> Just Git submodules and you're done. It would be a disaster if popular Go projects started using submodules, and consequently required users and contributors to play the submodule init/update/sync dance. Please, please -- prefer subtrees.
- davecheney 11y ago
  gb doesn't mandate any particular DVCS or DVCS strategy -- I understand it's a deeply personal issue for most. People are smart, if they like the tool, they'll figure out how to integrate it into whatever DVCS they use.
- ansible 11y ago
  I prefer subtrees to submodules. Not that subtrees are a cakewalk, in fact I wrote up a tutorial using them. [1] The main issue with submodules (for me anyway) is that it seems to be all too easy to revert a change in a submodule if you aren't careful. It is not very people-friendly because you're just looking at hashes, and it isn't clear which one is newer and which is older. [1] https://github.com/jamesgraves/example-go-app https://github.com/jamesgraves/example-go-app
- DannoHung 11y ago
  Submodules should be the right solution. Subtrees are weird as hell even if they require less interaction. I wonder if anyone in the git team has considered doing any updates on the porcelain for them. edit: BTW, even though syncs may be separate (and let's be honest, they should be really infrequent). You can do the update/init thing in one command: git submodule update --init --recursive

deleted 11y ago

[deleted]

ghodss 11y ago

One major limitation of this approach is that any project that wishes to vendor or lock their dependencies can no longer be used as a dependency for another project. From the gb GitHub: > A project is the consumer of your own source code, and possibly dependencies that your code consumes; nothing consumes the code from a project. This seems to imply that any code outside of a project (i.e. the code inside vendor/src) has no recourse for indicating the versions of their dependencies. This is nice in that it simplifies the problem, but to completely remove the ability for any and all libraries to indicate the versions of their dependencies seems unnecessarily restrictive. If I build a library for others to use, and I have a dependency, I want to be be able to lock to a specific version, or at least give my preference for a version. Of course, this creates its own issues - what do you do when two libraries depend on two different versions of the same library? (Also known as the diamond dependency problem.) This is where the Go culture helps, where as long as you pick a later version, things are likely to work. But I'd rather have the tooling let me detect the two versions that the two libraries want, show that there is a mismatch, and give me the ability to override and pick one (probably the later one). Instead, the gb approach eliminates the ability for the libraries to even have the ability to indicate what version they would prefer, which makes it even more difficult to get a bunch of libraries that share dependencies to work correctly together. godep (https://github.com/tools/godep https://github.com/tools/godep) seems to have the best compromise: vendor dependencies without path rewriting (though with GOPATH rewriting), but also keep track of their versions in a Godep.json file. You can gracefully pick between conflicting versions upstream if need be.

deleted 11y ago
[deleted]
eikenberry 11y ago
+1 for godep w/o import path rewriting. It isn't addressed in the presentation at all, only using it with path rewriting is mention and all the problems associated with it are with the path rewriting. IMO it is the best solution right now with just 1 issue, that it is not included with go. This is a pain with CI systems (e.g. Jenkins) where you have plugins to provide go itself, but have to figure out a way to get godep around to run your build. Right now I'm punting and just doing a go-get godep, then using it to build my project. I'm not happy with that though.

davecheney 11y ago

Hi HN, There was a video recorded at this meetup, but I'm not in control of that. In the mean time, the getting started document [1] is the contents of the "demo time" slide. 1. https://github.com/constabulary/gb/blob/master/getting-started.md https://github.com/constabulary/gb/blob/master/getting-start...

joewalnes 11y ago
Dave thanks for doing this. Not just for building the tool but for a very well thought out and convincing presentation. I'm completely with you. My own solution has similar ideals (per project versioned deps) though a different solution. https://github.com/joewalnes/go-getter/blob/master/README.md https://github.com/joewalnes/go-getter/blob/master/README.md. Honestly I'm just glad others are finally talking about the elephant in the room.
politician 11y ago
Thanks for building this; I completely agree that the `go` build tool is the problem, but settled on Goop as a stop-gap solution.

shocks 11y ago

It'd be nice if we could somehow use semver. I'd like to get patches automatically... edit; if you downvoted, maybe comment and explain why?

aikah 11y ago

Yet another problem the go team refuses to see. Either the go team says "not our problem" then people are stuck with doing things multiple and incompatibles ways , or the go team says "we already solve the problem, vendoring" and the go team doesnt understand the problem at first place. That's the recurrent pattern with the Go team. Everything cannot be dealt with in userland, or you'll end up with fragmentation. And No , vendoring doesn't solve the problem.

davecheney 11y ago
Honest question: Why does vendoring not solve the problem ?
- kungfooguru 11y ago
  I bet it works great for Google or anyone else who keeps everything in one big repo and isn't developing open source applications.
  - deleted 11y ago
    [deleted]
  - Titanous 11y ago
    Vendoring works great for us, we keep everything in one big repo and it's open source. The problem is that the current tooling for managing this is horrible.
    - nulltype 11y ago
      What is horrible about managing it? The biggest problem I encountered with vendoring is getting rid of git submodules.
      - Titanous 11y ago
        We currently use godep, and it breaks very easily due to tags and differing architectures and OSes (among other things). In addition to being unreliable, it's pretty slow. I'm not aware of any other tools that do import rewriting+vendoring and work better than godep.

Aissen 11y ago

Got my hopes up for deterministic builds, but then I read on slide 5: Out of scope: compiler doesn't produce byte for byte comparable binaries

davecheney 11y ago
There are others working on that problem. I probably shouldn't have mentioned it; if I hadn't, you wouldn't have thought about it, and then nobody would have had their hopes dashed.
- Aissen 11y ago
  English isn't my first language, so when I read "reproducible" I thought "deterministic"; I also didn't think you'd be working on the "vendoring" problem since I thought it was a solved issue (which it isn't for all the reasons in the post). Anyways, thanks for your work.
  - davecheney 11y ago
    I think the compilers that produce elf output are almost deterministic, at least when producing a fully static binary (no CGO). I don't think that holds for mach-o or PE binaries, but I'm really speaking off the cuff here, as I said, others care about this and are working on it separately.
  - ComputerGuru 11y ago
    However, English is my first language, and I thought exactly the same.
- silon3 11y ago
  Does JIT (or late/deploy time optimize/link) help here? Would it be easier to make deterministic builds when optimizer is not involved?
  - davecheney 11y ago
    Nope, this is about the source that goes into the compiler.

humanfromearth 11y ago

@davecheney Thanks for your work. If I publish my project on github and use gb with it, then people who want to contribute have to install gb as well. Do you think that in the future there is a chance to do the building of a gb project without the gb itself?

deleted 11y ago
[deleted]
davecheney 11y ago
Maybe, but it's not something I'm focusing on.

eternalban 11y ago

Go would hugely benefit from a hierarchical /.go in $GOPATH. This would allow for a robust versioning scheme, and allow the language to scale in the future.

davecheney 11y ago
Can you explain your idea in more detail ?
- eternalban 11y ago
  Think /.git, Dave. /.go vendors # map simple import name to explicit (ugly) path ... # other stuff for additional e.g. generate features Hierarchical: Think OO-ish shadowing of .go directives and metainfo. Is this clear?
  - davecheney 11y ago
    > Is this clear? not really
    - eternalban 11y ago
      # ---------------------------------- # general layout of the land # ---------------------------------- $GOPATH/src/.go $GOPATH/src/.go/config $GOPATH/src/.go/vendors # for all projects # ---------------------------------- # $GOPATH/.go/vendors sketch # ---------------------------------- lib/mq = github.com/lib/pq@master mgo = labix.org/v2/mgo ... etc # ---------------------------------- # specific project 1 # uses the global version of package labix.org/mgo # uses a project specific package hotness # ---------------------------------- $GOPATH/src/mygofoo $GOPATH/src/mygofoo/foo.go $GOPATH/src/mygofoo/.go $GOPATH/src/mygofoo/.go/vendors # ---------------------------------- # foo.go fragment # ---------------------------------- package mygofoo import ( "mgo" // version spec'd in global .go "lib/pq" // version spec'd in project specific .go "hotness" // version spec'd in project specific .go ... ) ... # ---------------------------------- # $GOPATH/src/mygofoo/.go/vendors sketch # ---------------------------------- lib/mq = github.com/lib/pq@experimental // lets pretend hotness = bitbucket.org/wiz/hotness@master ... etc # go tool > cd $GOPATH/src/mygofoo > go build -vendors using: mgo -> labix.org/v2/mgo lib/pq -> github.com/lib/pq@experimental hotness -> bitbucket.org/wiz/hotness@master > # go tool can obviously allow for command line mods of the .go

iddqd 11y ago

I think I'll stick to using submodules and a Makefile with GOPATH set to the vendor directory. Hasn't failed me yet.

drewolson 11y ago

How is this different from godep[1] other than being the root of its own GOPATH and having a different naming convention for the location of vendored dependencies? [1] - https://github.com/tools/godep https://github.com/tools/godep

ahmetmsft 11y ago

Can somebody please share a mirror? App Engine instance seems like it has exceeded the quota. (lol)

YZF 11y ago

The link isn't working for me. What we do is keep a copy of all packages we use and then assemble everything into a workspace as part of the automated build. Nothing is pulled from GitHub at build time. All those copies are versioned so you can have different projects use different versions. The dependencies are specified via some pre-existing build infrastructure but essentially there is one file in the project listing all the dependencies and where to find the packages. Developers can still have their own personal workspace and build with different versions but the "real" build is always reproducible. I think many people using Go or wanting to use Go in commercial/production environments are finding a little bit of impedance mismatch between the Go view of the world (the workspace and packages) and what their existing tooling does. It's not too terrible to deal with but it would be nice if we had a little more flexibility. go get could support some nicer/automated version of the above where you have the ability to download/mirror packages to a shared drive or to another source control system and pull from there to your workspace based on specific dependency versions. This is fairly minor friction though.

tshannon 11y ago

Might be overkill, but for my large projects, I simply host my own gogs [1] instance and vendor in all my dependencies to that instance. I get reproducible builds, and I can bring in any updates I want manually when I'm ready for me. 1.http://gogs.io/ http://gogs.io/

pas256 11y ago

The number of parallels between Go and Ruby keeps me entertained. Isn't this just rvm and bundler all over again?

duaneb 11y ago
Well, thankfully there's nothing like the nightmare 1.8.6 -> 1.9 switch, and it has loads of tools working with static code that ruby could never have.