8 ms·

Setup for Your Next Golang Project

88 points by MartinHeinz 7y ago

stevehiehn 7y ago
Hmm, interesting I had assumed /cmd/ was used for cli-ish applications that use a 'command pattern'. I didn't think people also did this for http apis.
- divan 7y ago
  It really depends on the type of the project. You generally use `cmd` package if you have more than 2 binaries in your project. Or, in rare case, where you ship a library and a binary (so you still can import a library by root package path). Otherwise, you don't need cmd/ dir at all.
- tptacek 7y ago
  That is my understanding as well, and having a directory tree that goes like "projectname/cmd/projectlibrary" seems super weird and non-idiomatic to me.
  - herval 7y ago
    pkg/cmd or just /cmd are rather popular in a lot of projects (https://github.com/grafana/grafana/tree/master/pkg/cmd https://github.com/grafana/grafana/tree/master/pkg/cmd / https://github.com/prometheus/prometheus/tree/master/cmd https://github.com/prometheus/prometheus/tree/master/cmd / https://github.com/istio/istio/tree/c3e1824c5e3f3e51ceb4a80b46aee9876be07131/istioctl/cmd https://github.com/istio/istio/tree/c3e1824c5e3f3e51ceb4a80b... / https://github.com/containous/traefik/tree/master/cmd https://github.com/containous/traefik/tree/master/cmd / etc)
    - tptacek 7y ago
      cmd/ directories are normal. Hosting all the business logic for the whole package under cmd/ seems less normal. You kind of need cmd/, because you can only have one main() in a package, and a package with a main() can't easily be imported into another package. That's why the directory exists, not (as I understand it) because there is some merit to burying code 3 directory levels deep.
      - herval 7y ago
        my response was to your comment that "having a directory tree that goes like "projectname/cmd/projectlibrary" seems super weird and non-idiomatic to me"
        PhilippGille 7y ago
        I'm on mobile and didn't check all your examples, but grafana has pkg/cmd and traefik has cmd next to pkg. That's not having the business logic below cmd. Whereas cmd/pkg is.
- pixelrevision 7y ago
  Most projects I see put source at the root so it’s straightforward to import and if there is a commandline utility you can build it ends up in ./cmd/main.go or something. One thing I dislike about go is that the way imports work you often see source code mixed with stuff like docker and make files.
  - sagichmal 7y ago
    Never cmd/main.go, always cmd/binaryname/main.go.
mrinfinite 7y ago
page loads slow, so i cancelled trying to open this link.
- jturpin 7y ago
  Not only that but it seems to be loading the same HTML content 3 times.
- deleted 7y ago
  [deleted]
grammarxcore 7y ago
I think it's important to note that this is a very opinionated way to set up Golang projects. It relies on a lot of tooling not explained in the post plus uses some processes that may or may not be standard (eg the version is one of many ways to do it). In future posts you might want to break down why you do things the way you do and explain why you chose those solutions over others. For those interested in skipping initial page (which seems to be insanely slow), just check out the boilerplate repo: https://github.com/MartinHeinz/go-project-blueprint https://github.com/MartinHeinz/go-project-blueprint
- MartinHeinz 7y ago
  You are right, this is quite opinionated solution, as everybody has their preferred ways of coding. The fact that not everything is explained in this post is because there are 4 parts, which are all linked at the top of README in the repo you linked. The pages loads quite slow indeed, that's because of the traffic coming in right now... The 1GB RAM VM is having hard time serving all the requests...
  - ben0x539 7y ago
    Do you think it might be worth it to prod a moderator to change the URL of this submission to https://towardsdatascience.com/ultimate-setup-for-your-next-golang-project-1cc989ad2a96 https://towardsdatascience.com/ultimate-setup-for-your-next-...?
- kerng 7y ago
  Go with all its intrinsic conventions is very opinionated, so its natural way of arguing about Golang.
throwaway894345 7y ago
> People often say "It works on my machine (and not in cloud)...", to avoid this we have simple solution - always run in docker container. And when I say always I really mean it - build in container, run in container, test in container. Actually I didn't mention it in previous section, but the make test really is "just" docker run. I've had nothing but bad luck with Docker for Mac. It's been super slow. If any container is running at all, the com.docker.hyperkit process eats 160% CPU even if `docker stats` shows the process hovering near 0% CPU. This has been the case for our entire team, and we're in the process of moving away from it.
- jhoechtl 7y ago
  Why aren't you using Linux then?
  - sdegutis 7y ago
    Not GP but Linux on laptops often has its own problems, including unreasonable battery usage that's non-trivially hard to fix.
  - throwaway894345 7y ago
    I like Linux as much as the next guy, but the Linux desktop ecosystem (hardware and software) doesn't get the investment it needs to be on-par with Mac. It's cheaper and easier to jettison Docker than MacOS.
  - ben0x539 7y ago
    Not the person you're responding to, but I'd like to add a data point. I try to use Linux where I can (I use it at home and for personal projects). Alas, I work in an environment where MacOS users are prioritized, so sometimes I cannot use tools or access services controlled by other people who still have "linux support" somewhere in their backlog. Depending on how critical these apps are for my work, putting up with eg. a sub-par Docker experience on MacOS may be the lesser evil.
- nickjj 7y ago
  > I've had nothing but bad luck with Docker for Mac. It's been super slow. I'm not sure if this affects every Mac user. I have a Docker course that has over 15,000+ people who signed up, of which a little less than half use MacOS and of that I've only had a handful of people mention really poor volume performance. I don't have enough data points to know exactly what the cause is. It's definitely a legit problem and it does affect a decent amount of people because I've seen the posts in other places too, but I don't know how wide spread the problem really is. It might be some type of hardware / OS version / tech stack combo (some languages and frameworks require dealing with many more files than others). I happen to use Docker for Windows and it's really really fast too for reference and I never had anyone mention slow volume performance in large apps. Native Linux is as good as it gets as well. I hope one day the folks at Docker figure out why the performance is so bad for some Mac users.
bdibs 7y ago
For comparison, here's a popular standard: https://github.com/golang-standards/project-layout https://github.com/golang-standards/project-layout
- deleted 7y ago
  [deleted]
- sagichmal 7y ago
  It's neither necessarily popular nor standard. Despite the GitHub org name they have no particular accreditation from anyone.
  - andyroid 7y ago
    11.8k stars on GitHub doesn’t qualify as popular?
xealgo 7y ago
I use the cmd directory for my main files, setups, etc... but use project root level packages for the bulk of the business logic. I kinda go the impression here that he uses cmd for all source.
divan 7y ago
Setup for your next Golang project should be an empty "main.go". Let the project grow naturally.
- lowmagnet 7y ago
  Doing the most minimal thing is often the right answer in Go. I recently went down a rabbit hole before finding out that something supported timeout natively. I was glad to delete the code.
  - heyoni 7y ago
    That’s always the best feeling, unless you figure it out a year later, and then you just beat yourself up for it.
- maximente 7y ago
  right? goodness, the post itself even strongly implies this individual simply cannot let simple things be: > The list goes on and it never gets to the point that I'm actually satisfied with the setup...
- atomi 7y ago
  But past that, there is a good GopherCon talk about structuring Go code: https://www.youtube.com/watch?v=oL6JBUk6tj0 https://www.youtube.com/watch?v=oL6JBUk6tj0 Code practically writes itself (in my opinion) if you structure it well.
- _wldu 7y ago
  Yep, and a standard Makefile too.
  - monocasa 7y ago
    What's wrong with starting out using go build, go run, and go test?
    - pstuart 7y ago
michaelperel 7y ago
(Shameless plug): For those looking for a good dev container experience with VSCode that works with go modules: https://github.com/michaelperel/docker-lock#development https://github.com/michaelperel/docker-lock#development edit -- on mobile, scroll down to "development"
dschulz 7y ago
Thanks for sharing. I've been perusing your blog posts and GitHub repositories for a week or so. Very useful stuff :)
thiht 7y ago
> People often say "It works on my machine (and not in cloud)..." That's definitely not a problem I have with Go. Building everything with Docker is wayyy overkill. Also, why is everything under the cmd/ package? It doesn't make sense. As for go modules, using `mod vendor` might not be the best suggestion as it requires using `go build -mod=vendor`. If you use go modules, just use go get and drop the vendor folder.
jen20 7y ago
I would hardly say this structure looks "perfect". For a start: - Use of Docker is a serious problem. I have no intention of ever running Docker on my machine to build some random project for which I have the toolchain already installed. Wrapping Docker with a Makefile is even more irritating. - Use of "package pkg" to keep the version number in is bizarre - I've never seen that before. Some use `package version` _inside_ a `pkg` directory which contains other public packages also, and use of `package version` at the top level is also fairly common. - Multiple modules in one git repository presents many challenges for people who want to reference them at different commits in other projects. - `pkg app` under a `cmd` directory is sometimes fine, but if there is only one app and no library, keeping it in the root should be preferred so that `go get` works properly. Furthermore, other packages being nested under it is not often great, since they are highly likely to be reused by other `cmd`s in the same repository. Personally for Go I've never needed a "starter" project, since not much needs configuring out the box as the toolchain is quite reasonable. If I were evaluating a project structured like this as a potential dependency, I'd knock off major marks for the structure and investigate alternatives. I guess if things came down to the wire, the structure wouldn't be a deal breaker, but I'd certainly aim to avoid it.

88 points by MartinHeinz 7y ago

Hmm, interesting I had assumed /cmd/ was used for cli-ish applications that use a 'command pattern'. I didn't think people also did this for http apis.

divan 7y ago
It really depends on the type of the project. You generally use `cmd` package if you have more than 2 binaries in your project. Or, in rare case, where you ship a library and a binary (so you still can import a library by root package path). Otherwise, you don't need cmd/ dir at all.
tptacek 7y ago
That is my understanding as well, and having a directory tree that goes like "projectname/cmd/projectlibrary" seems super weird and non-idiomatic to me.
- herval 7y ago
  pkg/cmd or just /cmd are rather popular in a lot of projects (https://github.com/grafana/grafana/tree/master/pkg/cmd https://github.com/grafana/grafana/tree/master/pkg/cmd / https://github.com/prometheus/prometheus/tree/master/cmd https://github.com/prometheus/prometheus/tree/master/cmd / https://github.com/istio/istio/tree/c3e1824c5e3f3e51ceb4a80b46aee9876be07131/istioctl/cmd https://github.com/istio/istio/tree/c3e1824c5e3f3e51ceb4a80b... / https://github.com/containous/traefik/tree/master/cmd https://github.com/containous/traefik/tree/master/cmd / etc)
  - tptacek 7y ago
    cmd/ directories are normal. Hosting all the business logic for the whole package under cmd/ seems less normal. You kind of need cmd/, because you can only have one main() in a package, and a package with a main() can't easily be imported into another package. That's why the directory exists, not (as I understand it) because there is some merit to burying code 3 directory levels deep.
    - herval 7y ago
      my response was to your comment that "having a directory tree that goes like "projectname/cmd/projectlibrary" seems super weird and non-idiomatic to me"
      - PhilippGille 7y ago
        I'm on mobile and didn't check all your examples, but grafana has pkg/cmd and traefik has cmd next to pkg. That's not having the business logic below cmd. Whereas cmd/pkg is.
pixelrevision 7y ago
Most projects I see put source at the root so it’s straightforward to import and if there is a commandline utility you can build it ends up in ./cmd/main.go or something. One thing I dislike about go is that the way imports work you often see source code mixed with stuff like docker and make files.
- sagichmal 7y ago
  Never cmd/main.go, always cmd/binaryname/main.go.

mrinfinite 7y ago

page loads slow, so i cancelled trying to open this link.

jturpin 7y ago
Not only that but it seems to be loading the same HTML content 3 times.
deleted 7y ago
[deleted]

grammarxcore 7y ago

I think it's important to note that this is a very opinionated way to set up Golang projects. It relies on a lot of tooling not explained in the post plus uses some processes that may or may not be standard (eg the version is one of many ways to do it). In future posts you might want to break down why you do things the way you do and explain why you chose those solutions over others. For those interested in skipping initial page (which seems to be insanely slow), just check out the boilerplate repo: https://github.com/MartinHeinz/go-project-blueprint https://github.com/MartinHeinz/go-project-blueprint

MartinHeinz 7y ago
You are right, this is quite opinionated solution, as everybody has their preferred ways of coding. The fact that not everything is explained in this post is because there are 4 parts, which are all linked at the top of README in the repo you linked. The pages loads quite slow indeed, that's because of the traffic coming in right now... The 1GB RAM VM is having hard time serving all the requests...
- ben0x539 7y ago
  Do you think it might be worth it to prod a moderator to change the URL of this submission to https://towardsdatascience.com/ultimate-setup-for-your-next-golang-project-1cc989ad2a96 https://towardsdatascience.com/ultimate-setup-for-your-next-...?
kerng 7y ago
Go with all its intrinsic conventions is very opinionated, so its natural way of arguing about Golang.

throwaway894345 7y ago

> People often say "It works on my machine (and not in cloud)...", to avoid this we have simple solution - always run in docker container. And when I say always I really mean it - build in container, run in container, test in container. Actually I didn't mention it in previous section, but the make test really is "just" docker run. I've had nothing but bad luck with Docker for Mac. It's been super slow. If any container is running at all, the com.docker.hyperkit process eats 160% CPU even if `docker stats` shows the process hovering near 0% CPU. This has been the case for our entire team, and we're in the process of moving away from it.

jhoechtl 7y ago
Why aren't you using Linux then?
- sdegutis 7y ago
  Not GP but Linux on laptops often has its own problems, including unreasonable battery usage that's non-trivially hard to fix.
- throwaway894345 7y ago
  I like Linux as much as the next guy, but the Linux desktop ecosystem (hardware and software) doesn't get the investment it needs to be on-par with Mac. It's cheaper and easier to jettison Docker than MacOS.
- ben0x539 7y ago
  Not the person you're responding to, but I'd like to add a data point. I try to use Linux where I can (I use it at home and for personal projects). Alas, I work in an environment where MacOS users are prioritized, so sometimes I cannot use tools or access services controlled by other people who still have "linux support" somewhere in their backlog. Depending on how critical these apps are for my work, putting up with eg. a sub-par Docker experience on MacOS may be the lesser evil.
nickjj 7y ago
> I've had nothing but bad luck with Docker for Mac. It's been super slow. I'm not sure if this affects every Mac user. I have a Docker course that has over 15,000+ people who signed up, of which a little less than half use MacOS and of that I've only had a handful of people mention really poor volume performance. I don't have enough data points to know exactly what the cause is. It's definitely a legit problem and it does affect a decent amount of people because I've seen the posts in other places too, but I don't know how wide spread the problem really is. It might be some type of hardware / OS version / tech stack combo (some languages and frameworks require dealing with many more files than others). I happen to use Docker for Windows and it's really really fast too for reference and I never had anyone mention slow volume performance in large apps. Native Linux is as good as it gets as well. I hope one day the folks at Docker figure out why the performance is so bad for some Mac users.

bdibs 7y ago

For comparison, here's a popular standard: https://github.com/golang-standards/project-layout https://github.com/golang-standards/project-layout

deleted 7y ago
[deleted]
sagichmal 7y ago
It's neither necessarily popular nor standard. Despite the GitHub org name they have no particular accreditation from anyone.
- andyroid 7y ago
  11.8k stars on GitHub doesn’t qualify as popular?

xealgo 7y ago

I use the cmd directory for my main files, setups, etc... but use project root level packages for the bulk of the business logic. I kinda go the impression here that he uses cmd for all source.

divan 7y ago

Setup for your next Golang project should be an empty "main.go". Let the project grow naturally.

lowmagnet 7y ago
Doing the most minimal thing is often the right answer in Go. I recently went down a rabbit hole before finding out that something supported timeout natively. I was glad to delete the code.
- heyoni 7y ago
  That’s always the best feeling, unless you figure it out a year later, and then you just beat yourself up for it.
maximente 7y ago
right? goodness, the post itself even strongly implies this individual simply cannot let simple things be: > The list goes on and it never gets to the point that I'm actually satisfied with the setup...
atomi 7y ago
But past that, there is a good GopherCon talk about structuring Go code: https://www.youtube.com/watch?v=oL6JBUk6tj0 https://www.youtube.com/watch?v=oL6JBUk6tj0 Code practically writes itself (in my opinion) if you structure it well.
_wldu 7y ago
Yep, and a standard Makefile too.
- monocasa 7y ago
  What's wrong with starting out using go build, go run, and go test?
  - pstuart 7y ago

michaelperel 7y ago

(Shameless plug): For those looking for a good dev container experience with VSCode that works with go modules: https://github.com/michaelperel/docker-lock#development https://github.com/michaelperel/docker-lock#development edit -- on mobile, scroll down to "development"

dschulz 7y ago

Thanks for sharing. I've been perusing your blog posts and GitHub repositories for a week or so. Very useful stuff :)

thiht 7y ago

> People often say "It works on my machine (and not in cloud)..." That's definitely not a problem I have with Go. Building everything with Docker is wayyy overkill. Also, why is everything under the cmd/ package? It doesn't make sense. As for go modules, using `mod vendor` might not be the best suggestion as it requires using `go build -mod=vendor`. If you use go modules, just use go get and drop the vendor folder.

jen20 7y ago

I would hardly say this structure looks "perfect". For a start: - Use of Docker is a serious problem. I have no intention of ever running Docker on my machine to build some random project for which I have the toolchain already installed. Wrapping Docker with a Makefile is even more irritating. - Use of "package pkg" to keep the version number in is bizarre - I've never seen that before. Some use `package version` _inside_ a `pkg` directory which contains other public packages also, and use of `package version` at the top level is also fairly common. - Multiple modules in one git repository presents many challenges for people who want to reference them at different commits in other projects. - `pkg app` under a `cmd` directory is sometimes fine, but if there is only one app and no library, keeping it in the root should be preferred so that `go get` works properly. Furthermore, other packages being nested under it is not often great, since they are highly likely to be reused by other `cmd`s in the same repository. Personally for Go I've never needed a "starter" project, since not much needs configuring out the box as the toolchain is quite reasonable. If I were evaluating a project structured like this as a potential dependency, I'd knock off major marks for the structure and investigate alternatives. I guess if things came down to the wire, the structure wouldn't be a deal breaker, but I'd certainly aim to avoid it.