5 ms·

Hellogopher – Just “git clone” and “make” any Go project

135 points by FiloSottile 10y ago

verandaguy 10y ago
As someone who's more fond of this approach than Go's native one, it's funny how we've come full circle (even if this never gains widespread adoption). Thank goodness for the inclusion of sensible vendoring, by the way! Go's default system broke a lot of older code because of the native vendoring system (if you can call it that).
justincormack 10y ago
I have long used docker to build all my Go so I can put it where I wish, and where makes sense for projects that are not all Go. Looking forward to a permanent fix upstream eventually.
FiloSottile 10y ago
Hey all, author here. Happy to take any feedback, in particular “I tried and it didn’t work out of the box” stories. I will NOT tell you to RTFM. I recommend reading https://github.com/cloudflare/hellogopher#why https://github.com/cloudflare/hellogopher#why for more insight. There’s also a gif demo https://twitter.com/FiloSottile/status/822745605806112768 https://twitter.com/FiloSottile/status/822745605806112768 EDIT: uploaded the keynote I gave at Golab where I present the problem and introduce this project https://vimeo.com/200469720 https://vimeo.com/200469720
- akavel 10y ago
  Am I right to understand that it's "build from anywhere" as long as "anywhere" is not Windows? (Please don't tell me about MinGW/Cygwin if the whole idea is to "make life simpler".)
  - worg 10y ago
    Anywhere in the FS of your OS From the readme: >They expect to just clone a repository anywhere, and be able to build it
  - FiloSottile 10y ago
    Sadly, correct. There is no way a standalone tool couldn't reproduce this behavior on Windows, and the eventual integrated solution definitely will, but it's just beyond the scope of hellogopher and outside my expertise.
    - e12e 10y ago
      I'm also a little confused. This is a fix for the standard go installer on platforms not setting a GOPATH, so that "go get" doesn't work without manually configuring GOPATH? Or does it actually do anything else (except "break" windows builds - or rather introduce a system complementary to "got get" that isn't quite as cross platform as go itself)?
      - FiloSottile 10y ago
        This is not about platforms (or about go get), this is about users who don't want to or know how to correctly use GOPATH. It doesn't break anything, since you can still use go get, go build, or anything that worked before. It doesn't help Windows users, but it doesn't hurt them.
        kkirsche 10y ago
        But as of 1.8 there is a default gopath negating that
        TheDong 10y ago
        A default gopath doesn't help a user who does "cd dev; git clone https://github.com/coolprojects/project; https://github.com/coolprojects/project; cd project; go build" (or make) The default gopath does not help there. Sure, if a user knows how to use go's insane global-project-directory-bullshit then they don't need this, but since it's a go-specific bullshit, they might not. Every sane language lets you have a directory where you want for each unrelated project. Only go dictates some crazy tree and defaults to creating it for you in $HOME
- reledi 10y ago
  Thanks for making this, hope it catches on. As a non-gopher, whenever I came across a project where I needed to build a Go binary I'd get blocked by the GOPATH and since then I've just avoided such projects.
- lunixbochs 10y ago
  I did something similar for one of my projects (github.com/lunixbochs/usercorn) if you want another perspective. My primary goal was `make` would work out of the box in most environments (on OS X / Linux), even if you did not have Go installed at all. This includes downloading the Go binaries for your platform, building three C libraries, fixing gopath, and automatically determining packages for `go get`. https://github.com/lunixbochs/usercorn/blob/master/Makefile https://github.com/lunixbochs/usercorn/blob/master/Makefile
weitzj 10y ago
This looks awesome. I haven't tried it yet, but the hopefully it supports writing test coverage files for multiple packages, i.e. I already split the code up into separate packages and put each package in a corresponding subfolder. Using go test ./... I want to collect coverage reports for all my packages while ignoring `vendor` So my workaround is something like the following, where I generate a coverage report for each package to later concatenate them into a final coverage report ``` for i in $(go list ./... | grep -v 'vendor') do coverName="$(basename $i).coverraw" go test --ldflags '-extldflags "-static"' -v -race -coverprofile="${coverName}" "${i}" | tee -a "${test_outfile}" done```
- FiloSottile 10y ago
  Just run "make cover" :) It does exactly what you want, and it handles overlapping coverage blocks (if two packages have tests that run the same portion of code), which just concatenating will mess up. (It uses gocovmerge behind the scenes, but it will be hidden from you.)
  - weitzj 10y ago
    Awesome! It would be nice to include something like go2xunit so you can hook the results up to the Jenkins coverage reports.
    - FiloSottile 10y ago
      We do that on the Cloudflare CI, but I deemed it too specific for hellogopher. (I can be convinced otherwise.) Instead, you run hellogopher with CI=1, and it drops logs and artifacts in fixed locations, that you can then post-process as you like.
      - weitzj 10y ago
        Ah. OK. Yeah. That works as well for me. Thank you
    - weitzj 10y ago
Sphax 10y ago
I don't understand how HN ranks posts: this was at the very top with just 9 points, no comments, how does that work exactly ?
- weitzj 10y ago
  If the post gets upvotes in a short amount of time it ranks higher
- overcast 10y ago
  score = (votes - 1) / pow((item_hour_age+2), gravity) Which means, newer posts have a higher weight. As time goes by, the score lowers. The constant gravity can be adjusted to lower or raise how fast this occurs.
blablabla123 10y ago
I think Go works very well without Makefiles. (IMHO even better) You can just enter `go build .` You can install the stuff using your favorite package manager, e.g. dpkg - then it's even possible to uninstall your stuff.
- misnome 10y ago
  Then why is this necessary - why isn't it just: all: go build
  - Sphax 10y ago
    All this is necessary because people don't expect or don't want to put their code/projects under $GOPATH/src. My take on it is don't fight the GOPATH, because a lot of Go tooling expects your code to be in a valid GOPATH.
    - Anderkent 10y ago
      This is one of the main reasons I hate working with go projects. I have about 50 different git repos checked out on my machine right now; written in different languages. Over they years I've worked out a structure for these repos that helps me keep track of what's where and switch between them easily. But then go comes along and is like 'no, you MUST check this out in this particular location'. And also that location has to be 6 directory layers deep, for some reason. It's just pointlessly broken when compared to every other language, where a project is self-contained within a directory.
      - blablabla123 10y ago
        Ok but when you hate them, why do you check them out manually? You can just mkdir $GOPATH ; go get { pkg1, pkg2, ..., pkg50 } I mean, nobody complains that npm -g puts stuff into some weird node directory that you should never ever touch manually. Or did you ever compile a random Java project that doesn't use Maven? I think regarding convenience Go is top notch. Regarding "weirdness" it's in the middle between Unix C and Java.
Ruphin 10y ago
Projects like these and their popularity are indicative of how unapproachable the Go development environment is to new users. One of the first things I did when introducing Go at work is write a bunch of script that solved this exact issue for our team, and I suspect that a solution like this has been written a million times over already. I love the Go language but the development environment tooling is just awful (aside from some great things like gofmt). This requirement of having files in specific folders to be able to build, the (lack of a sane) vendoring system, both problems keep popping up and spawning new solutions over and over. There is no excuse for a modern language to force people to learn how to use the tools before they can even start using the language. Tools should work for the user, not the other way around. Other languages get this right, so why can't Go? Kudos to the author for spending some time to document and publish their solution to this specific issue. If these projects get enough traction, like govendor, perhaps the language authors will at some point incorporate the most popular solutions into the language platform. At least it'll save everyone else from rewriting the same tool.
- bborud 10y ago
  I started using Go last year and I was very pleased with how approachable and quick to learn it was. Including the tools. The role of $GOPATH was a bit new to me, but apart from that it has mostly been very smooth sailing. I'd be interested in reading a bit more about the problems you experienced. Can you recommend any articles or blog postings that describe what kinds of problems people tend to have?
  - vbernat 10y ago
    People may just not want to have their source code into a special hierarchy where their own source code is mangled with the source code of the dependencies. I also use a Makefile to setup an ad hoc GOPATH to workaround this.
    - majewsky 10y ago
      Me too. I work under the assumption that a packager can just grab the tarball of the latest stable tag from the repo's releases page on Github, unpack it somewhere and `make && make install`. I converged upon the same approach of setting up an ad-hoc gopath.
deleted 10y ago
[deleted]
codedokode 10y ago
I also have spent some time to make bash scripts that would allow me to copy my Go project into a temporary folder to linux machine and compile it.
drdaeman 10y ago
Sorry, maybe this is a stupid question... but does this require me to store all the dependencies under ./vendor in the repo? I would really prefer it ./vendor would be .gitignored (like .virtualenv or node_modules) and `make deps` would parse the source and fetch the deps, sort of like `go get` does. Bundling all the dependencies in your own repo just feels wrong to me. Especially if it's not git submodules (and I've heard submodules are frowned upon by those who have tried them - I've haven't had such necessity, so don't hold any opinion). At least very similar approaches had led to some really bad experiences I had in early 2000's with PHP (it was typical to bundle vendored deps there because of relative lack of package management).
- FiloSottile 10y ago
  That depends on the tool you want to use to manage your vendor folder, so no, hellogopher does not enforce an opinion on that. For example, you could modify the Makefile to run "gvt restore" before building. However, I warmly recommend checking vendor/ in: not doing so requires the user to have gvt (or whatever tool), breaks go get, and adds unnecessary external dependencies (think left-pad). More details here https://github.com/FiloSottile/gvt#alternative-not-checking-in-vendored-source https://github.com/FiloSottile/gvt#alternative-not-checking-...
  - drdaeman 10y ago
    I see. Guess, bare `go get` just can't get a dependency into `./vendor/` so either a third-party or checking in is required. Is there any suggestions/best practices how to maintain `vendor/` under the source control? I mean, handling updates, branch merges and all the disaster that accidental local patches may bring - this sort of stuff?
    - TheDong 10y ago
      The real problem with go get here is that it can't pick a version, only latest. Checking in vendor is much safer because your dependencies can't break you. there are no good upstream suggestions on how to deal with vendor. My preference is to use to use http://glide.sh/ http://glide.sh/ and store 'glide.yaml' and 'glide.lock' in the repo, at which point the vendor folder can be recreated to the correct versions without needing to parse code or checkin vendor. Glide is also really good at handling updates, pulling from alternate sources, tracking alternate branch names, etc etc.
nnx 10y ago
The annoyance/confusion/hurdle of GOPATH for beginners in the language will finally be resolved with the upcoming Go 1.8 release. https://beta.golang.org/doc/go1.8#gopath https://beta.golang.org/doc/go1.8#gopath
- FiloSottile 10y ago
  The default GOPATH is great, but it only helps running "go get", it doesn't solve developing in arbitrary locations on the filesystem. I address it at the end of my introductory talk: https://vimeo.com/200469720 https://vimeo.com/200469720
djhworld 10y ago
One thing that I really like about this is it makes building go things via Homebrew that much easier, I've just been converting my tools and things I install via a personal tap to use this - great effort!

135 points by FiloSottile 10y ago

As someone who's more fond of this approach than Go's native one, it's funny how we've come full circle (even if this never gains widespread adoption). Thank goodness for the inclusion of sensible vendoring, by the way! Go's default system broke a lot of older code because of the native vendoring system (if you can call it that).

justincormack 10y ago

I have long used docker to build all my Go so I can put it where I wish, and where makes sense for projects that are not all Go. Looking forward to a permanent fix upstream eventually.

FiloSottile 10y ago

Hey all, author here. Happy to take any feedback, in particular “I tried and it didn’t work out of the box” stories. I will NOT tell you to RTFM. I recommend reading https://github.com/cloudflare/hellogopher#why https://github.com/cloudflare/hellogopher#why for more insight. There’s also a gif demo https://twitter.com/FiloSottile/status/822745605806112768 https://twitter.com/FiloSottile/status/822745605806112768 EDIT: uploaded the keynote I gave at Golab where I present the problem and introduce this project https://vimeo.com/200469720 https://vimeo.com/200469720

akavel 10y ago
Am I right to understand that it's "build from anywhere" as long as "anywhere" is not Windows? (Please don't tell me about MinGW/Cygwin if the whole idea is to "make life simpler".)
- worg 10y ago
  Anywhere in the FS of your OS From the readme: >They expect to just clone a repository anywhere, and be able to build it
- FiloSottile 10y ago
  Sadly, correct. There is no way a standalone tool couldn't reproduce this behavior on Windows, and the eventual integrated solution definitely will, but it's just beyond the scope of hellogopher and outside my expertise.
  - e12e 10y ago
    I'm also a little confused. This is a fix for the standard go installer on platforms not setting a GOPATH, so that "go get" doesn't work without manually configuring GOPATH? Or does it actually do anything else (except "break" windows builds - or rather introduce a system complementary to "got get" that isn't quite as cross platform as go itself)?
    - FiloSottile 10y ago
      This is not about platforms (or about go get), this is about users who don't want to or know how to correctly use GOPATH. It doesn't break anything, since you can still use go get, go build, or anything that worked before. It doesn't help Windows users, but it doesn't hurt them.
      - kkirsche 10y ago
        But as of 1.8 there is a default gopath negating that
        TheDong 10y ago
        A default gopath doesn't help a user who does "cd dev; git clone https://github.com/coolprojects/project; https://github.com/coolprojects/project; cd project; go build" (or make) The default gopath does not help there. Sure, if a user knows how to use go's insane global-project-directory-bullshit then they don't need this, but since it's a go-specific bullshit, they might not. Every sane language lets you have a directory where you want for each unrelated project. Only go dictates some crazy tree and defaults to creating it for you in $HOME
reledi 10y ago
Thanks for making this, hope it catches on. As a non-gopher, whenever I came across a project where I needed to build a Go binary I'd get blocked by the GOPATH and since then I've just avoided such projects.
lunixbochs 10y ago
I did something similar for one of my projects (github.com/lunixbochs/usercorn) if you want another perspective. My primary goal was `make` would work out of the box in most environments (on OS X / Linux), even if you did not have Go installed at all. This includes downloading the Go binaries for your platform, building three C libraries, fixing gopath, and automatically determining packages for `go get`. https://github.com/lunixbochs/usercorn/blob/master/Makefile https://github.com/lunixbochs/usercorn/blob/master/Makefile

weitzj 10y ago

This looks awesome. I haven't tried it yet, but the hopefully it supports writing test coverage files for multiple packages, i.e. I already split the code up into separate packages and put each package in a corresponding subfolder. Using go test ./... I want to collect coverage reports for all my packages while ignoring `vendor` So my workaround is something like the following, where I generate a coverage report for each package to later concatenate them into a final coverage report ``` for i in $(go list ./... | grep -v 'vendor') do coverName="$(basename $i).coverraw" go test --ldflags '-extldflags "-static"' -v -race -coverprofile="${coverName}" "${i}" | tee -a "${test_outfile}" done```

FiloSottile 10y ago
Just run "make cover" :) It does exactly what you want, and it handles overlapping coverage blocks (if two packages have tests that run the same portion of code), which just concatenating will mess up. (It uses gocovmerge behind the scenes, but it will be hidden from you.)
- weitzj 10y ago
  Awesome! It would be nice to include something like go2xunit so you can hook the results up to the Jenkins coverage reports.
  - FiloSottile 10y ago
    We do that on the Cloudflare CI, but I deemed it too specific for hellogopher. (I can be convinced otherwise.) Instead, you run hellogopher with CI=1, and it drops logs and artifacts in fixed locations, that you can then post-process as you like.
    - weitzj 10y ago
      Ah. OK. Yeah. That works as well for me. Thank you
  - weitzj 10y ago

Sphax 10y ago

I don't understand how HN ranks posts: this was at the very top with just 9 points, no comments, how does that work exactly ?

weitzj 10y ago
If the post gets upvotes in a short amount of time it ranks higher
overcast 10y ago
score = (votes - 1) / pow((item_hour_age+2), gravity) Which means, newer posts have a higher weight. As time goes by, the score lowers. The constant gravity can be adjusted to lower or raise how fast this occurs.

blablabla123 10y ago

I think Go works very well without Makefiles. (IMHO even better) You can just enter `go build .` You can install the stuff using your favorite package manager, e.g. dpkg - then it's even possible to uninstall your stuff.

misnome 10y ago
Then why is this necessary - why isn't it just: all: go build
- Sphax 10y ago
  All this is necessary because people don't expect or don't want to put their code/projects under $GOPATH/src. My take on it is don't fight the GOPATH, because a lot of Go tooling expects your code to be in a valid GOPATH.
  - Anderkent 10y ago
    This is one of the main reasons I hate working with go projects. I have about 50 different git repos checked out on my machine right now; written in different languages. Over they years I've worked out a structure for these repos that helps me keep track of what's where and switch between them easily. But then go comes along and is like 'no, you MUST check this out in this particular location'. And also that location has to be 6 directory layers deep, for some reason. It's just pointlessly broken when compared to every other language, where a project is self-contained within a directory.
    - blablabla123 10y ago
      Ok but when you hate them, why do you check them out manually? You can just mkdir $GOPATH ; go get { pkg1, pkg2, ..., pkg50 } I mean, nobody complains that npm -g puts stuff into some weird node directory that you should never ever touch manually. Or did you ever compile a random Java project that doesn't use Maven? I think regarding convenience Go is top notch. Regarding "weirdness" it's in the middle between Unix C and Java.

Ruphin 10y ago

Projects like these and their popularity are indicative of how unapproachable the Go development environment is to new users. One of the first things I did when introducing Go at work is write a bunch of script that solved this exact issue for our team, and I suspect that a solution like this has been written a million times over already. I love the Go language but the development environment tooling is just awful (aside from some great things like gofmt). This requirement of having files in specific folders to be able to build, the (lack of a sane) vendoring system, both problems keep popping up and spawning new solutions over and over. There is no excuse for a modern language to force people to learn how to use the tools before they can even start using the language. Tools should work for the user, not the other way around. Other languages get this right, so why can't Go? Kudos to the author for spending some time to document and publish their solution to this specific issue. If these projects get enough traction, like govendor, perhaps the language authors will at some point incorporate the most popular solutions into the language platform. At least it'll save everyone else from rewriting the same tool.

bborud 10y ago
I started using Go last year and I was very pleased with how approachable and quick to learn it was. Including the tools. The role of $GOPATH was a bit new to me, but apart from that it has mostly been very smooth sailing. I'd be interested in reading a bit more about the problems you experienced. Can you recommend any articles or blog postings that describe what kinds of problems people tend to have?
- vbernat 10y ago
  People may just not want to have their source code into a special hierarchy where their own source code is mangled with the source code of the dependencies. I also use a Makefile to setup an ad hoc GOPATH to workaround this.
  - majewsky 10y ago
    Me too. I work under the assumption that a packager can just grab the tarball of the latest stable tag from the repo's releases page on Github, unpack it somewhere and `make && make install`. I converged upon the same approach of setting up an ad-hoc gopath.

deleted 10y ago

[deleted]

codedokode 10y ago

I also have spent some time to make bash scripts that would allow me to copy my Go project into a temporary folder to linux machine and compile it.

drdaeman 10y ago

Sorry, maybe this is a stupid question... but does this require me to store all the dependencies under ./vendor in the repo? I would really prefer it ./vendor would be .gitignored (like .virtualenv or node_modules) and `make deps` would parse the source and fetch the deps, sort of like `go get` does. Bundling all the dependencies in your own repo just feels wrong to me. Especially if it's not git submodules (and I've heard submodules are frowned upon by those who have tried them - I've haven't had such necessity, so don't hold any opinion). At least very similar approaches had led to some really bad experiences I had in early 2000's with PHP (it was typical to bundle vendored deps there because of relative lack of package management).

FiloSottile 10y ago
That depends on the tool you want to use to manage your vendor folder, so no, hellogopher does not enforce an opinion on that. For example, you could modify the Makefile to run "gvt restore" before building. However, I warmly recommend checking vendor/ in: not doing so requires the user to have gvt (or whatever tool), breaks go get, and adds unnecessary external dependencies (think left-pad). More details here https://github.com/FiloSottile/gvt#alternative-not-checking-in-vendored-source https://github.com/FiloSottile/gvt#alternative-not-checking-...
- drdaeman 10y ago
  I see. Guess, bare `go get` just can't get a dependency into `./vendor/` so either a third-party or checking in is required. Is there any suggestions/best practices how to maintain `vendor/` under the source control? I mean, handling updates, branch merges and all the disaster that accidental local patches may bring - this sort of stuff?
  - TheDong 10y ago
    The real problem with go get here is that it can't pick a version, only latest. Checking in vendor is much safer because your dependencies can't break you. there are no good upstream suggestions on how to deal with vendor. My preference is to use to use http://glide.sh/ http://glide.sh/ and store 'glide.yaml' and 'glide.lock' in the repo, at which point the vendor folder can be recreated to the correct versions without needing to parse code or checkin vendor. Glide is also really good at handling updates, pulling from alternate sources, tracking alternate branch names, etc etc.

nnx 10y ago

The annoyance/confusion/hurdle of GOPATH for beginners in the language will finally be resolved with the upcoming Go 1.8 release. https://beta.golang.org/doc/go1.8#gopath https://beta.golang.org/doc/go1.8#gopath

FiloSottile 10y ago
The default GOPATH is great, but it only helps running "go get", it doesn't solve developing in arbitrary locations on the filesystem. I address it at the end of my introductory talk: https://vimeo.com/200469720 https://vimeo.com/200469720

djhworld 10y ago

One thing that I really like about this is it makes building go things via Homebrew that much easier, I've just been converting my tools and things I install via a personal tap to use this - great effort!