12 ms·

Using Makefile(s) for Go

137 points by prakashdanish 7y ago

finaliteration 7y ago
I’ve been using Makefiles for Go development basically since I started with the language. It’s really effective for me and makes compilation and, in my case, deployment to AWS Lambdas via CloudFormation commands (also invoked by Make) really simple. It’s also easy to bring someone up to speed with how building and deploying works.
- rob74 7y ago
  It might make it easier to bring someone up to speed for your project, but he won't learn a damn thing about building other Go projects - I guess that's one of the arguments that the opponents of make, er, make...
  - marcus_holmes 7y ago
    there's no standard for building Go projects (except the use of "go build"). Make is common enough that learning how to use it is a worthwhile use of time for a new Go developer. I also use a Makefile for my projects, for all the above reasons, but mostly so that I can hand it to a new team member and say "clone the repo, install make if you haven't already, then choose either make docker_init or make localdev_init" and know that they'll have a working installation about 15 minutes later. Not supporting Windows is a furphy - we're developing a Linux-based web server. There's no point trying to develop this on Windows. Yes, you can do it with Docker, but you'll always be wondering if that error was from your code, or from some misalignment of stuff in your tech stack.
echlebek 7y ago
I've seen a lot of developers, especially developers with C backgrounds, reach for Makefiles when approaching Go development, and I think it's a harmful practice. So, I'd like to offer a respectful but firm rebuttal to this article. :) I dislike using make(1) with Go for two reasons. The first is that make was developed for building C projects, and therefore is oriented around that task. Building C projects is a lot different than building Go projects, and it involves stitching together a lot of pieces, with plenty of intermediate results. make(1) has first class support for intermediate results, which are expressed as targets. If you look at the article, the author has to use a workaround just to avoid this core feature of make(1). The second reason I dislike using make(1) for Go projects is that it harms portability. A Go project should only require the Go compiler to build successfully. Go projects that need make(1) to build will not work out of the box for Windows users, even though Go is fully supported on Windows. For me, this puts Makefiles into the "nonstarter" category, even though I do all of my own development work on Linux. There is just no reason to complicate things for people who don't have make(1) installed. For code generation and other ancillary tasks, Go includes the 'go generate' facility. This feature was created specifically to free developers from depending on external build tools. (https://blog.golang.org/generate https://blog.golang.org/generate) For producing several binaries for one project, use several different main packages in directories that are named what you want your binary to be. Edit: corrected some terminology.
- downerending 7y ago
  IMHE, a language that fights against 'make' is generally poorly designed in that regard. Typically its functionality gets replaced/reinvented by a bespoke and buggy behemoth, which becomes yet one more thing to learn.
- IshKebab 7y ago
  I agree. They claim `make` is simple, but it really isn't. PHONY targets are one example. Unfortunately I've looked for an alternative and didn't really find anything very good. I eventually settled on a Python 3 script. Python 3 is reasonably nice to use with type annotations. It doesn't require compilation and its speed is fine if you're using it to drive other build systems, rather than as a build system itself. Way more people understand Python than Make, and it is a full programming language so you don't get stuck when you want to do something complicated. It doesn't have a build in DAG task system but I'm sure there are a million libraries for that. I haven't had need of that yet, but a quick search turned out https://pydoit.org/ https://pydoit.org/ which looks ok.
  - avar 7y ago
    If you think Python 3 is a better tool than a Makefile you're entirely missing out on what makes "make" a useful tool. The entire point is that it's not a general-purpose programming language, so that you're forced to separate the DAG aspect of your build process from any complex logic. Sure, some things you can't do in a Makefile, and you generally shouldn't try. The Makefile should call some Python / Perl / whatever helper script to accomplish that particular task, separated by a process boundary. By doing it like that you can seamlessly run your build process in parallel if you haven't screwed up in declaring your dependencies. E.g. some script that needs Python to fetch something from a database for the build will run concurrently with the build of the documentation. It also encourages you to structure things in such a way as to have incremental and resumable compilation, e.g. Ctrl+C-ing the middle of a build in a project with a well-maintained Makefile won't require you to run the whole thing from the beginning, but that's typical of someone's home grown "I can do it better" monolithic "./build" shell or python script.
roadbeats 7y ago
It looks very close to https://github.com/azer/go-makefile-example https://github.com/azer/go-makefile-example
cesarb 7y ago
I noticed you didn't mention ".DELETE_ON_ERROR". AFAIK, it's recommended to always use it (according to the GNU make manual: "[...] 'make' will do this if '.DELETE_ON_ERROR' appears as a target. This is almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.")
rraval 7y ago
This... isn't even using the `make` part of Makefiles at all. If you look at the final example, every [1] rule is marked as `.PHONY`. `make` bundles 2 capabilities: a dependency graph and an out-of-date check to rebuild files. This demonstration uses neither. The author would be better served with a shell script and a `case` block. The advantages: - Functions! The `check-environment` rule is really a function call in disguise. - No 2 phase execution. The author talks about using variables like `APP`, but those are make variables with very different semantics than shell variables (which are also available inside the recipes). [1] Yes, there's a `check-environment` "rule" that isn't marked, but it likely should be since it isn't building a file target named `check-environment`.
- panpanna 7y ago
  I disagree. Make is more than a build system, it's also an automation tool. It gives you a fairly flexible format for managing different tasks with shared variables and autocompletion and more. You can do it with a bunch of shell scripts too but I prefer having everything in a single file.
  - boomlinde 7y ago
    > Make is more than a build system, it's also an automation tool. I agree with that, but I don't believe that the GP said otherwise. Make is an automation tool, but what it aims to automate is exactly dependency tracking. Other features, like actually performing the tasks, are off-loaded to the shell and other tools. > It gives you a fairly flexible format for managing different tasks with shared variables and autocompletion and more. You could as well be describing a shell here, which already have these features. > You can do it with a bunch of shell scripts too but I prefer having everything in a single file. What stops you from using a single shell script? Likewise, you have a Makefile delegate tasks to other Makefiles (as the author has done for the build-tokenizer rule). Anyway, you should of course use them in any manner that suits you, but for a guide on "using Makefiles for Go" I think it's an oversight to ignore the main selling point of make by just using it as you would a switched shell script with no dependency tracking. It just introduces another syntax and new caveats to the problem, adding little value.
mikegirouard 7y ago
It's really frustrating seeing so many Makefiles that don't _make_ anything. Make syntax is really odd. I see so many folks go out of their way to deal w/quirks of make when they really just need a shell script. You can see this anti-pattern very quickly when you see `.PHONY` targets for everything. I think make is useful for some aspects of go. GOPATH is becoming less relevant now, but still helpful when you want to have build-time dependencies in $PATH $(GOPATH)/bin/some-dependency: go get -u ... I still use make when building artifacts, especially in CI. But as a default, I almost always try to talk folks out of using make for this sort of stuff.
- GordonS 7y ago
  Is there a way to easily have something like make targets in a shell script, without a ton of boilerplate? > Make syntax is really odd I don't find it particularly strange, except my biggest peeve - the insistence on tabs!
  - frou_dh 7y ago
    Here's a named task runner shellscript (as compared to a Makefile full of .PHONY being used as a named task runner). #!/bin/sh set -e case "$1" in build) ;; run) ;; clean) ;; *) echo "unknown: $1"; exit 2 ;; esac Someone else ITT hinted it could be done like the following. But the last line is dubious because it will happily run anything on PATH. #!/bin/sh set -e test $# -gt 0 build() { : } run() { : } clean() { : } "$@"
    - GordonS 7y ago
      > (as compared to a Makefile full of .PHONY being used as a named task runner) Personally, I've never used PHONY, or had a need to. That said, your bash examples are pretty simple - I especially like the 2nd example, as it would trivially allow having targets that ran other targets, e.g. an "all" target from a makefile: ``` all: build push build: @docker build --tag ${IMG} --tag ${UNSTABLE} . rebuild: @docker build --no-cache --tag ${IMG} --tag ${UNSTABLE} . push: @docker push ${NAME} ```
e2le 7y ago
For projects that use tags to turn on/off compilation options, Makefiles and shell scripts make sense.
IHLayman 7y ago
I use Makefiles for Go projects all the time, but not in the way the article describes. First off, in a pre `go mod` world, if you had dependencies to check before running the build, then a Makefile was the easiest way to manage that. But even in a post `go mod` world, there are good reasons to use one that the article totally overlooks: * Makefiles introduce a topological sort to build steps. This is the reason you use it instead of build shell scripts: it allows build steps to run in parallel, it guarantees order by dependency which is the best way to read build steps, and it makes file freshness an easy element to check for a build step, which is still needed for Go projects with multiple subpackages. * Go projects usually have more than go files that are required in making an executable. If you run a web server and you are bundling static pages into your executable, Makefiles are the best way to handle that. * If you are building for multiple architectures, or want to encode the git tag/branch into the executable, it is better to have that Makefile bake in the necessary options on the build step and keep it uniform across the build. * If you write a Go file and bake that into a Docker image, I find it best to drop the image and container hashes into files so that I can get to them easily for docker exec/attach/rm/rmi commmands. But there is one bigger reason Makefiles work for our entire team. We standardize on using Makefiles as the entrypoint for our builds. We have a polyglot environment at work so sometimes it gets confusing to figure out how to build a project. By standardizing on running 'make' we are all on the same page. Have a Javascript project to webpack? Run make and have make call yarn. Have a python wheel to construct? Run make and have make call python setup.py. You have a Java project that requires a sequence of maven commands to build? Run make and have the makefile call maven. Is that inefficient? You bet it is. Does it make it easier to sort out what to do to build a project for the first time? Yes it does. Does it make it 100% easier for our CI/CD framework to work with multiple languages and scan for the necessary compilers and dependencies? Heck yeah. [edited for lousy formatting]
- panpanna 7y ago
  That was an excellent comment! I use make for almost all my projects (regardless of language) and I have a system where "make init" sets up the environment (install packages, set up containers, and so on) and "make run" runs it and "make test" tests it. Now I can come back to projects from 5-10 years ago and get them running with minimal effort since all the magic is in the makefile and not my in forgetful brain.
jjuel 7y ago
Not related to the contents of the article, but I love the font on that site. I also love the simplicity of the site as well. Nothing takes away from the ability to read. It is so clear and concise.
alexhutcheson 7y ago
Alternatively, you could use Bazel[1], and automatically generate most of your build rules with Gazelle[2]. This would allow you to extend your build system beyond what's available via "go build", while avoiding the well-known pitfalls of Makefiles (config complexity, reproducibility, etc.) [1] https://github.com/bazelbuild/rules_go https://github.com/bazelbuild/rules_go [2] https://github.com/bazelbuild/bazel-gazelle https://github.com/bazelbuild/bazel-gazelle
- meddlepal 7y ago
  I tried Bazel for awhile on a bunch of projects and just found it to be more headache than it's worth. I like the promise, but the tool is... meh
3fe9a03ccd14ca5 7y ago
I don’t mind Makefiles, and usually use them for basic command configuration. However, when the logic gets even a little complicated and I almost always reach for bash. Everybody knows a little bash, and it’s available in almost all systems.
hedora 7y ago
I suggest reading “recursive make considered harmful”. It is a wonderful introduction to make, and explains how to avoid a few pitfalls most make users (including this article) run into. In particular, the targets in the subdirectory makefiles can and should be auto-generated using make itself. There’s no need for the makefiles in the subdirectories (there is also no need to use an external tool to generate them, which is the other mistake people often make).
peterwwillis 7y ago
rm -rf ${APP} is a code smell. If ${APP} is not a directory, -r should not be in this command. At best it is possibly confusing, and at worst if somehow ${APP} accidentally becomes a directory it will just remove it and you will have no idea that it was a directory, whereas just rm -f ${APP} will fail because it can't unlink a directory. Build success is an important factor in a CI/CD pipeline, therefore builds should fail immediately under unexpected behavior. Also, on .PHONY on a single line: But for Makefiles which grow really big, this is not suggested as it could lead to ambiguity and unreadability, hence the preferred way is to explicitly set phony target right before the rule definition. If your Makefile grows really big, it's going to become a nightmare to maintain. Either split up your codebase + builds into sub-directories, or figure out some other way to structure your builds so that it's not super complicated to reason about or maintain them.
- boomlinde 7y ago
  I find that complexity of a Makefile isn't necessarily a function of its size. Ideally, one should be able to reason about each target individually, specifying its dependencies without consideration for how they are generated, whether they already exist etc. In such an ideal situation, it doesn't matter how large the Makefile is. Maintenance problems IMO happen when you can't trust tasks to fully specify their dependencies or that task commands only generate the target output. Over all I agree with your argument, though.
boomlinde 7y ago
This Makefile could as well have been a shell script. It doesn't track changes to dependencies even when it's obvious how to do so. For example, the build rule has an obvious dependency (main.go) and an obvious target ($(APP)). Instead of tracking these which IMO is the primary advantage of using Make, it deliberately destroys the existing build. docked-build always necessarily rebuilds the binary as well Presumably, Go has some kind of build cache making such dependency tracking relatively useless anyway, maybe Docker has too, but if you aren't tracking dependencies and rebuilding only when necessary why use Make instead of a big switch in a shell script? Personally I'd only use Make for Go if I introduce some task that takes significant time and isn't already handled by the go toolchain. Another couple of notes: there are two docker-push rules. The first seems like it was meant to be docker-build. The other is that the docker build rule will tag the build with the HEAD hash, regardless of whether it's building from a clean checkout or a dirty repo.
ascotan 7y ago
Going to throw out my 2 cents here: 1. I don't like multiple makefiles. Icky with lots of duplication and high maintenance. Bad article. 2. When possible I use target expansion to generate targets in the main makefile: APPS:= app1 app2 app3 $(APPS:%=build.%) 3. I prefer to use makefile functions rather than reaching for "bash" where possible: https://www.gnu.org/software/make/manual/html_node/Functions.html#Functions https://www.gnu.org/software/make/manual/html_node/Functions... 4. If something is really complicated - extract to bash Make has been written in 10K languages and the original is still the best
rynop 7y ago
I think the example discussed in this blog is selling Make short. A more complete example with: protoc(protobuf), Docker, DynamoDB local, go modules (and vendoring), and testing can be found at https://github.com/rynop/abp-sam-twirp/blob/master/Makefile https://github.com/rynop/abp-sam-twirp/blob/master/Makefile
apeace 7y ago
Great article, but I'm not sure it's a good idea to segment your Docker images by environment. Part of Docker's appeal is that you can be sure your staging & production containers are bit-for-bit the same. I use a workflow like this: * For all commits on all branches, run tests. If tests don't pass, don't push containers to registry. * For all commits on all branches, build and push a container `{branchname}-{commitsha}` (assuming tests pass). * Code review, etc. * Merge pull request to `master` branch (tests will run, and only push a container if they pass). * Deploy `master-{commitsha}` to staging. * Do your final testing on staging. * Deploy the same `master-{commitsha}` to production. Now you're deploying to production from the master branch, which passed tests, and the container is the same one as you tested on staging. Plus, you can always deploy your non-master `{branchname}-{commitsha}` images to a separate environment, or to staging, if you need to do a bit of experimenting.
knowsuchagency 7y ago
There really is no perfect build tool, but in my experience, nothing touches invoke http://www.pyinvoke.org http://www.pyinvoke.org for building and automation. Any project will eventually have build and deployment scripts with non-trivial amounts of logic in them. The question then becomes whether you want all that complex logic in shell scripts, makefiles, or Python. For me, it's a no-brainer. I'll take the latter every time.
cwojno 7y ago
You don't use makefiles in go! You just take your code, copy go.mod and go.sum into a Docker image, then RUN mod download, then re-copy the rest of the code and run bui... Shit... Docker is a makefile...
narven 7y ago
Nice article. I use makefiles a lot, mostly for all projects that I use, both for frontend and backend, mainly to have the same commands independently of framework/platform that im using. For me its helpful to just run `make` both to build and run a go project and a react project. Another thing you can add is: .DEFAULT_GOAL := start start: fmt swag vet build run Helps define you default command soyou just need to run `make` and will run all inside of `start` Since most of us use `.env` files for enviroment files, you can use something like: # this allows to import into this file all current system envs include .env export And it will inject all of .env file into the current running `process` Also have some other shortcuts (variables): GOCMD=go GOBUILD=$(GOCMD) build GOCLEAN=$(GOCMD) clean GOTEST=$(GOCMD) test GOFMT=gofmt -w GOGET=$(GOCMD) mod download GOVER=$(COCMD) vet GOFILES=$(shell find . -name "*.go" -type f) BINARY_NAME=my-cool-project BINARY_UNIX=$(BINARY_NAME)_prod

137 points by prakashdanish 7y ago

I’ve been using Makefiles for Go development basically since I started with the language. It’s really effective for me and makes compilation and, in my case, deployment to AWS Lambdas via CloudFormation commands (also invoked by Make) really simple. It’s also easy to bring someone up to speed with how building and deploying works.

rob74 7y ago
It might make it easier to bring someone up to speed for your project, but he won't learn a damn thing about building other Go projects - I guess that's one of the arguments that the opponents of make, er, make...
- marcus_holmes 7y ago
  there's no standard for building Go projects (except the use of "go build"). Make is common enough that learning how to use it is a worthwhile use of time for a new Go developer. I also use a Makefile for my projects, for all the above reasons, but mostly so that I can hand it to a new team member and say "clone the repo, install make if you haven't already, then choose either make docker_init or make localdev_init" and know that they'll have a working installation about 15 minutes later. Not supporting Windows is a furphy - we're developing a Linux-based web server. There's no point trying to develop this on Windows. Yes, you can do it with Docker, but you'll always be wondering if that error was from your code, or from some misalignment of stuff in your tech stack.

echlebek 7y ago

I've seen a lot of developers, especially developers with C backgrounds, reach for Makefiles when approaching Go development, and I think it's a harmful practice. So, I'd like to offer a respectful but firm rebuttal to this article. :) I dislike using make(1) with Go for two reasons. The first is that make was developed for building C projects, and therefore is oriented around that task. Building C projects is a lot different than building Go projects, and it involves stitching together a lot of pieces, with plenty of intermediate results. make(1) has first class support for intermediate results, which are expressed as targets. If you look at the article, the author has to use a workaround just to avoid this core feature of make(1). The second reason I dislike using make(1) for Go projects is that it harms portability. A Go project should only require the Go compiler to build successfully. Go projects that need make(1) to build will not work out of the box for Windows users, even though Go is fully supported on Windows. For me, this puts Makefiles into the "nonstarter" category, even though I do all of my own development work on Linux. There is just no reason to complicate things for people who don't have make(1) installed. For code generation and other ancillary tasks, Go includes the 'go generate' facility. This feature was created specifically to free developers from depending on external build tools. (https://blog.golang.org/generate https://blog.golang.org/generate) For producing several binaries for one project, use several different main packages in directories that are named what you want your binary to be. Edit: corrected some terminology.

downerending 7y ago
IMHE, a language that fights against 'make' is generally poorly designed in that regard. Typically its functionality gets replaced/reinvented by a bespoke and buggy behemoth, which becomes yet one more thing to learn.
IshKebab 7y ago
I agree. They claim `make` is simple, but it really isn't. PHONY targets are one example. Unfortunately I've looked for an alternative and didn't really find anything very good. I eventually settled on a Python 3 script. Python 3 is reasonably nice to use with type annotations. It doesn't require compilation and its speed is fine if you're using it to drive other build systems, rather than as a build system itself. Way more people understand Python than Make, and it is a full programming language so you don't get stuck when you want to do something complicated. It doesn't have a build in DAG task system but I'm sure there are a million libraries for that. I haven't had need of that yet, but a quick search turned out https://pydoit.org/ https://pydoit.org/ which looks ok.
- avar 7y ago
  If you think Python 3 is a better tool than a Makefile you're entirely missing out on what makes "make" a useful tool. The entire point is that it's not a general-purpose programming language, so that you're forced to separate the DAG aspect of your build process from any complex logic. Sure, some things you can't do in a Makefile, and you generally shouldn't try. The Makefile should call some Python / Perl / whatever helper script to accomplish that particular task, separated by a process boundary. By doing it like that you can seamlessly run your build process in parallel if you haven't screwed up in declaring your dependencies. E.g. some script that needs Python to fetch something from a database for the build will run concurrently with the build of the documentation. It also encourages you to structure things in such a way as to have incremental and resumable compilation, e.g. Ctrl+C-ing the middle of a build in a project with a well-maintained Makefile won't require you to run the whole thing from the beginning, but that's typical of someone's home grown "I can do it better" monolithic "./build" shell or python script.

roadbeats 7y ago

It looks very close to https://github.com/azer/go-makefile-example https://github.com/azer/go-makefile-example

cesarb 7y ago

I noticed you didn't mention ".DELETE_ON_ERROR". AFAIK, it's recommended to always use it (according to the GNU make manual: "[...] 'make' will do this if '.DELETE_ON_ERROR' appears as a target. This is almost always what you want 'make' to do, but it is not historical practice; so for compatibility, you must explicitly request it.")

rraval 7y ago

This... isn't even using the `make` part of Makefiles at all. If you look at the final example, every [1] rule is marked as `.PHONY`. `make` bundles 2 capabilities: a dependency graph and an out-of-date check to rebuild files. This demonstration uses neither. The author would be better served with a shell script and a `case` block. The advantages: - Functions! The `check-environment` rule is really a function call in disguise. - No 2 phase execution. The author talks about using variables like `APP`, but those are make variables with very different semantics than shell variables (which are also available inside the recipes). [1] Yes, there's a `check-environment` "rule" that isn't marked, but it likely should be since it isn't building a file target named `check-environment`.

panpanna 7y ago
I disagree. Make is more than a build system, it's also an automation tool. It gives you a fairly flexible format for managing different tasks with shared variables and autocompletion and more. You can do it with a bunch of shell scripts too but I prefer having everything in a single file.
- boomlinde 7y ago
  > Make is more than a build system, it's also an automation tool. I agree with that, but I don't believe that the GP said otherwise. Make is an automation tool, but what it aims to automate is exactly dependency tracking. Other features, like actually performing the tasks, are off-loaded to the shell and other tools. > It gives you a fairly flexible format for managing different tasks with shared variables and autocompletion and more. You could as well be describing a shell here, which already have these features. > You can do it with a bunch of shell scripts too but I prefer having everything in a single file. What stops you from using a single shell script? Likewise, you have a Makefile delegate tasks to other Makefiles (as the author has done for the build-tokenizer rule). Anyway, you should of course use them in any manner that suits you, but for a guide on "using Makefiles for Go" I think it's an oversight to ignore the main selling point of make by just using it as you would a switched shell script with no dependency tracking. It just introduces another syntax and new caveats to the problem, adding little value.

mikegirouard 7y ago

It's really frustrating seeing so many Makefiles that don't _make_ anything. Make syntax is really odd. I see so many folks go out of their way to deal w/quirks of make when they really just need a shell script. You can see this anti-pattern very quickly when you see `.PHONY` targets for everything. I think make is useful for some aspects of go. GOPATH is becoming less relevant now, but still helpful when you want to have build-time dependencies in $PATH $(GOPATH)/bin/some-dependency: go get -u ... I still use make when building artifacts, especially in CI. But as a default, I almost always try to talk folks out of using make for this sort of stuff.

GordonS 7y ago
Is there a way to easily have something like make targets in a shell script, without a ton of boilerplate? > Make syntax is really odd I don't find it particularly strange, except my biggest peeve - the insistence on tabs!
- frou_dh 7y ago
  Here's a named task runner shellscript (as compared to a Makefile full of .PHONY being used as a named task runner). #!/bin/sh set -e case "$1" in build) ;; run) ;; clean) ;; *) echo "unknown: $1"; exit 2 ;; esac Someone else ITT hinted it could be done like the following. But the last line is dubious because it will happily run anything on PATH. #!/bin/sh set -e test $# -gt 0 build() { : } run() { : } clean() { : } "$@"
  - GordonS 7y ago
    > (as compared to a Makefile full of .PHONY being used as a named task runner) Personally, I've never used PHONY, or had a need to. That said, your bash examples are pretty simple - I especially like the 2nd example, as it would trivially allow having targets that ran other targets, e.g. an "all" target from a makefile: ``` all: build push build: @docker build --tag ${IMG} --tag ${UNSTABLE} . rebuild: @docker build --no-cache --tag ${IMG} --tag ${UNSTABLE} . push: @docker push ${NAME} ```

e2le 7y ago

For projects that use tags to turn on/off compilation options, Makefiles and shell scripts make sense.

IHLayman 7y ago

I use Makefiles for Go projects all the time, but not in the way the article describes. First off, in a pre `go mod` world, if you had dependencies to check before running the build, then a Makefile was the easiest way to manage that. But even in a post `go mod` world, there are good reasons to use one that the article totally overlooks: * Makefiles introduce a topological sort to build steps. This is the reason you use it instead of build shell scripts: it allows build steps to run in parallel, it guarantees order by dependency which is the best way to read build steps, and it makes file freshness an easy element to check for a build step, which is still needed for Go projects with multiple subpackages. * Go projects usually have more than go files that are required in making an executable. If you run a web server and you are bundling static pages into your executable, Makefiles are the best way to handle that. * If you are building for multiple architectures, or want to encode the git tag/branch into the executable, it is better to have that Makefile bake in the necessary options on the build step and keep it uniform across the build. * If you write a Go file and bake that into a Docker image, I find it best to drop the image and container hashes into files so that I can get to them easily for docker exec/attach/rm/rmi commmands. But there is one bigger reason Makefiles work for our entire team. We standardize on using Makefiles as the entrypoint for our builds. We have a polyglot environment at work so sometimes it gets confusing to figure out how to build a project. By standardizing on running 'make' we are all on the same page. Have a Javascript project to webpack? Run make and have make call yarn. Have a python wheel to construct? Run make and have make call python setup.py. You have a Java project that requires a sequence of maven commands to build? Run make and have the makefile call maven. Is that inefficient? You bet it is. Does it make it easier to sort out what to do to build a project for the first time? Yes it does. Does it make it 100% easier for our CI/CD framework to work with multiple languages and scan for the necessary compilers and dependencies? Heck yeah. [edited for lousy formatting]

panpanna 7y ago
That was an excellent comment! I use make for almost all my projects (regardless of language) and I have a system where "make init" sets up the environment (install packages, set up containers, and so on) and "make run" runs it and "make test" tests it. Now I can come back to projects from 5-10 years ago and get them running with minimal effort since all the magic is in the makefile and not my in forgetful brain.

jjuel 7y ago

Not related to the contents of the article, but I love the font on that site. I also love the simplicity of the site as well. Nothing takes away from the ability to read. It is so clear and concise.

alexhutcheson 7y ago

Alternatively, you could use Bazel[1], and automatically generate most of your build rules with Gazelle[2]. This would allow you to extend your build system beyond what's available via "go build", while avoiding the well-known pitfalls of Makefiles (config complexity, reproducibility, etc.) [1] https://github.com/bazelbuild/rules_go https://github.com/bazelbuild/rules_go [2] https://github.com/bazelbuild/bazel-gazelle https://github.com/bazelbuild/bazel-gazelle

meddlepal 7y ago
I tried Bazel for awhile on a bunch of projects and just found it to be more headache than it's worth. I like the promise, but the tool is... meh

3fe9a03ccd14ca5 7y ago

I don’t mind Makefiles, and usually use them for basic command configuration. However, when the logic gets even a little complicated and I almost always reach for bash. Everybody knows a little bash, and it’s available in almost all systems.

hedora 7y ago

I suggest reading “recursive make considered harmful”. It is a wonderful introduction to make, and explains how to avoid a few pitfalls most make users (including this article) run into. In particular, the targets in the subdirectory makefiles can and should be auto-generated using make itself. There’s no need for the makefiles in the subdirectories (there is also no need to use an external tool to generate them, which is the other mistake people often make).

peterwwillis 7y ago

rm -rf ${APP} is a code smell. If ${APP} is not a directory, -r should not be in this command. At best it is possibly confusing, and at worst if somehow ${APP} accidentally becomes a directory it will just remove it and you will have no idea that it was a directory, whereas just rm -f ${APP} will fail because it can't unlink a directory. Build success is an important factor in a CI/CD pipeline, therefore builds should fail immediately under unexpected behavior. Also, on .PHONY on a single line: But for Makefiles which grow really big, this is not suggested as it could lead to ambiguity and unreadability, hence the preferred way is to explicitly set phony target right before the rule definition. If your Makefile grows really big, it's going to become a nightmare to maintain. Either split up your codebase + builds into sub-directories, or figure out some other way to structure your builds so that it's not super complicated to reason about or maintain them.

boomlinde 7y ago
I find that complexity of a Makefile isn't necessarily a function of its size. Ideally, one should be able to reason about each target individually, specifying its dependencies without consideration for how they are generated, whether they already exist etc. In such an ideal situation, it doesn't matter how large the Makefile is. Maintenance problems IMO happen when you can't trust tasks to fully specify their dependencies or that task commands only generate the target output. Over all I agree with your argument, though.

boomlinde 7y ago

This Makefile could as well have been a shell script. It doesn't track changes to dependencies even when it's obvious how to do so. For example, the build rule has an obvious dependency (main.go) and an obvious target ($(APP)). Instead of tracking these which IMO is the primary advantage of using Make, it deliberately destroys the existing build. docked-build always necessarily rebuilds the binary as well Presumably, Go has some kind of build cache making such dependency tracking relatively useless anyway, maybe Docker has too, but if you aren't tracking dependencies and rebuilding only when necessary why use Make instead of a big switch in a shell script? Personally I'd only use Make for Go if I introduce some task that takes significant time and isn't already handled by the go toolchain. Another couple of notes: there are two docker-push rules. The first seems like it was meant to be docker-build. The other is that the docker build rule will tag the build with the HEAD hash, regardless of whether it's building from a clean checkout or a dirty repo.

ascotan 7y ago

Going to throw out my 2 cents here: 1. I don't like multiple makefiles. Icky with lots of duplication and high maintenance. Bad article. 2. When possible I use target expansion to generate targets in the main makefile: APPS:= app1 app2 app3 $(APPS:%=build.%) 3. I prefer to use makefile functions rather than reaching for "bash" where possible: https://www.gnu.org/software/make/manual/html_node/Functions.html#Functions https://www.gnu.org/software/make/manual/html_node/Functions... 4. If something is really complicated - extract to bash Make has been written in 10K languages and the original is still the best

rynop 7y ago

I think the example discussed in this blog is selling Make short. A more complete example with: protoc(protobuf), Docker, DynamoDB local, go modules (and vendoring), and testing can be found at https://github.com/rynop/abp-sam-twirp/blob/master/Makefile https://github.com/rynop/abp-sam-twirp/blob/master/Makefile

apeace 7y ago

Great article, but I'm not sure it's a good idea to segment your Docker images by environment. Part of Docker's appeal is that you can be sure your staging & production containers are bit-for-bit the same. I use a workflow like this: * For all commits on all branches, run tests. If tests don't pass, don't push containers to registry. * For all commits on all branches, build and push a container `{branchname}-{commitsha}` (assuming tests pass). * Code review, etc. * Merge pull request to `master` branch (tests will run, and only push a container if they pass). * Deploy `master-{commitsha}` to staging. * Do your final testing on staging. * Deploy the same `master-{commitsha}` to production. Now you're deploying to production from the master branch, which passed tests, and the container is the same one as you tested on staging. Plus, you can always deploy your non-master `{branchname}-{commitsha}` images to a separate environment, or to staging, if you need to do a bit of experimenting.

knowsuchagency 7y ago

There really is no perfect build tool, but in my experience, nothing touches invoke http://www.pyinvoke.org http://www.pyinvoke.org for building and automation. Any project will eventually have build and deployment scripts with non-trivial amounts of logic in them. The question then becomes whether you want all that complex logic in shell scripts, makefiles, or Python. For me, it's a no-brainer. I'll take the latter every time.

cwojno 7y ago

You don't use makefiles in go! You just take your code, copy go.mod and go.sum into a Docker image, then RUN mod download, then re-copy the rest of the code and run bui... Shit... Docker is a makefile...

narven 7y ago

Nice article. I use makefiles a lot, mostly for all projects that I use, both for frontend and backend, mainly to have the same commands independently of framework/platform that im using. For me its helpful to just run `make` both to build and run a go project and a react project. Another thing you can add is: .DEFAULT_GOAL := start start: fmt swag vet build run Helps define you default command soyou just need to run `make` and will run all inside of `start` Since most of us use `.env` files for enviroment files, you can use something like: # this allows to import into this file all current system envs include .env export And it will inject all of .env file into the current running `process` Also have some other shortcuts (variables): GOCMD=go GOBUILD=$(GOCMD) build GOCLEAN=$(GOCMD) clean GOTEST=$(GOCMD) test GOFMT=gofmt -w GOGET=$(GOCMD) mod download GOVER=$(COCMD) vet GOFILES=$(shell find . -name "*.go" -type f) BINARY_NAME=my-cool-project BINARY_UNIX=$(BINARY_NAME)_prod