16 ms·
I've seen a lot of developers, especially developers with C backgrounds, reach for Makefiles when approaching Go development, and I think it's a harmful practic
by echlebek 7y ago
I've seen a lot of developers, especially developers with C backgrounds, reach for Makefiles when approaching Go development, and I think it's a harmful practice. So, I'd like to offer a respectful but firm rebuttal to this article. :)
I dislike using make(1) with Go for two reasons.
The first is that make was developed for building C projects, and therefore is oriented around that task. Building C projects is a lot different than building Go projects, and it involves stitching together a lot of pieces, with plenty of intermediate results.
make(1) has first class support for intermediate results, which are expressed as targets.
If you look at the article, the author has to use a workaround just to avoid this core feature of make(1).
The second reason I dislike using make(1) for Go projects is that it harms portability.
A Go project should only require the Go compiler to build successfully. Go projects that need make(1) to build will not work out of the box for Windows users, even though Go is fully supported on Windows. For me, this puts Makefiles into the "nonstarter" category, even though I do all of my own development work on Linux. There is just no reason to complicate things for people who don't have make(1) installed.
For code generation and other ancillary tasks, Go includes the 'go generate' facility. This feature was created specifically to free developers from depending on external build tools. (https://blog.golang.org/generate https://blog.golang.org/generate)
For producing several binaries for one project, use several different main packages in directories that are named what you want your binary to be.
Edit: corrected some terminology.
- downerending 7y agoIMHE, a language that fights against 'make' is generally poorly designed in that regard. Typically its functionality gets replaced/reinvented by a bespoke and buggy behemoth, which becomes yet one more thing to learn.
- IshKebab 7y agoI agree. They claim `make` is simple, but it really isn't. PHONY targets are one example. Unfortunately I've looked for an alternative and didn't really find anything very good. I eventually settled on a Python 3 script. Python 3 is reasonably nice to use with type annotations. It doesn't require compilation and its speed is fine if you're using it to drive other build systems, rather than as a build system itself. Way more people understand Python than Make, and it is a full programming language so you don't get stuck when you want to do something complicated. It doesn't have a build in DAG task system but I'm sure there are a million libraries for that. I haven't had need of that yet, but a quick search turned out https://pydoit.org/ https://pydoit.org/ which looks ok.
- avar 7y agoIf you think Python 3 is a better tool than a Makefile you're entirely missing out on what makes "make" a useful tool. The entire point is that it's not a general-purpose programming language, so that you're forced to separate the DAG aspect of your build process from any complex logic. Sure, some things you can't do in a Makefile, and you generally shouldn't try. The Makefile should call some Python / Perl / whatever helper script to accomplish that particular task, separated by a process boundary. By doing it like that you can seamlessly run your build process in parallel if you haven't screwed up in declaring your dependencies. E.g. some script that needs Python to fetch something from a database for the build will run concurrently with the build of the documentation. It also encourages you to structure things in such a way as to have incremental and resumable compilation, e.g. Ctrl+C-ing the middle of a build in a project with a well-maintained Makefile won't require you to run the whole thing from the beginning, but that's typical of someone's home grown "I can do it better" monolithic "./build" shell or python script.
- IshKebab 7y agoRead the last paragraph of my comment. I am entirely aware of what Make does.
- yori 7y agoPHONY targets are GNU extensions to make. They are not part of the make specification itself. POSIX make is really quite simple and it allows phony targets by depending on a target that has no dependencies or commands in it.
- zimpenfish 7y agoIf you've tried `redo`, what issues did you find that put it into the "not good" bucket? [1] https://github.com/apenwarr/redo https://github.com/apenwarr/redo
- Foxboron 7y ago>A Go project should only require the Go compiler to build successfully. But they don't. The go compiler doesn't support yarn, npm, protobuf, open-api generators, doc generators like md2man, go-bindata-assetfs, gox and everything you need to complete the code generation done in modern Go applications. So how do you orchestrate this? People use Makefiles, bash scripts, go scripts and everything in between and combined. It gives you a plethora of bewildering and confusing build options which can't be solved with `go build`, and neither with `make` in a straight forward fashion. Add `go get -u ... && go mod vendor` with some `npm install` in the Makefile, along with some overriding and/or ignoring `$GOFLAGS`, `$LDFLAGS` and `$CGO_LDFLAGS` and you got yourself an ecosystem hostile for packaging and compilation. A go project can't use `go build` by itself - but it can't really use the Makefile either as people overengineer the process. But let me stress this. Always use plain Makefiles over any other methods. It's there and has been used for decades for a reason.
- akerl_ 7y agoI think there’s a distinction to be drawn here between a couple use cases for Makefiles (specifically for building software): * Makefiles can act as shortcuts for common existing functionality of the build toolchain * Makefiles can add new functionality that is not part of the build toolchain * Makefiles can add new functionality that replicates existing functionality in the build toolchain An example of the first case is one of the first examples in the article: using `make build` to run `go build`. The second includes things like the later example for `make docker-push`. The third includes things like makefiles that generate intermediate files or other things that `go generate` could do. Only the 3rd thing can really meaningfully harm productivity, but in my experience it’s the least common usage of `make`. A Makefiles that wraps `go generate && go build` into `make build` seems fully outside the scope of the portability concern, since a user without Make could just run the same commands themselves. Likewise, a Makefiles that adds `make release` which uploads the build artifact to GitHub Releases or similar isn’t replacing something the go toolchain could do, so it’s also not affecting portability. The user without Make couldn’t have used docker-push anyways, since the go compiler doesn’t support pushing release assets.
- hiccuphippo 7y agoI use make for the first case a lot. If there was a tool for running bash functions from a predefined file just as easy and ubiquitous as make, I would switch in a heartbeat.
- wnoise 7y agoPut "$@" at the bottom of the file. Then type ./filename function. You can extend that to a fancier function dispatcher, argument checker, etc, if you want.
- TheDong 7y agoWell technically simple to do, that misses the point. That is neither ubiquitous, nor (as a result), trivial for a newcomer to understand. If I clone a repo and see a makefile, I know what to do. If I clone a repo and see './hack.sh' or './runme' or './do' or whatever you chose to call it, I have no clue whether I should invoke it or not. There are few alternatives to make that have the same level of mindshare, and thus the same ease of use for a given newcomer to grok what to do. Most alternatives are language specific (e.g. "rake" in rails did a good job of building programmer expectations that you use 'rake ...' to do various common tasks).
- _ph_ 7y agoI avoid using makefiles, unless I need them :). Yes, plain Go projects which are supposed to produce an executable probably won't need a makefile and I haven't used any for those. But if your project should produce a shared library, it is nice to wrap the build command in a makefile, as it is easier to type "make". Also, when integrating into larger projects or when other tasks in addition to building the Go project are required, unifying them with makefiles can be helpful.
- majewsky 7y agoYou make it sound like `go install` and `go test` are the only things you're ever going to run in a Go repository. This is blatantly untrue. For example, these are the invocations for the test suite for one of my Go programs: https://github.com/sapcc/limes/blob/364317fa9a25065bcf9384c844927377aad10239/Makefile#L30-L48 https://github.com/sapcc/limes/blob/364317fa9a25065bcf9384c8... Why should I have to enter all of this manually every single time? (And before you argue that gofmt, golint and go vet run in the editor if you've set it up properly: That's true, and that's how I have my editor set up. That part of the test is to catch the external contributors that don't.) > The second reason I dislike using make(1) for Go projects is that it harms portability. A Go project should only require the Go compiler to build successfully. For many of my projects, a Makefile is the main reason why repos work with `go get` at all. I use `make` which prepares all the generated files and non-Go artifacts (typically bundled into `bindata.go`), so that I can commit these in the repo. Then when a user comes along, they can `go get` the application because all the bespoke compilation steps have already been done by me via my Makefile. An example of this: https://github.com/majewsky/alltag/blob/df161b55fa4c7eba0abec82d2cf0df34e49b0ad4/Makefile#L10-L22 https://github.com/majewsky/alltag/blob/df161b55fa4c7eba0abe...
- aequitas 7y agoI agree if you end up using Make like its used in C projects: compiling intermediate objects, linking them, etc. In that case the Go compiler should suffice. I almost exclusively use Make for projects in any language nowadays as workflow automation, this includes Go, Python, Terraform, Docker builds. In this way Make is a indispensable tool for me. As for me it's portable where it matters (macOS, Linux, WSL), it's ubiquitous, it has a stable API and it's behaviour is well know to me. Sometimes I have to work around some shortcomings of Make, eg: things that don't produce a file as result, where you `touch` a fake artifact file. But this is a minor annoyance for me compared to what Make brings in term of how simple and declarative I can automate my workflows.
- mikojan 7y agoWhy do people on this website use "make(1)" instead of "make" in writing? And I know it's in the man pages but what is this number even for?
- SifJar 7y agoThe number is the "section" of the manual: https://www.kernel.org/doc/man-pages/ https://www.kernel.org/doc/man-pages/ My guess is make(1) is to distinguish from make(1p) - http://man7.org/linux/man-pages/man1/make.1p.html http://man7.org/linux/man-pages/man1/make.1p.html
- masklinn 7y ago> My guess is make(1) is to distinguish from make(1p) - http://man7.org/linux/man-pages/man1/make.1p.html http://man7.org/linux/man-pages/man1/make.1p.html Or just reflexively included just in case, because of commonly encountering potentially ambiguous situations that you just disambiguate by default, even for non-ambiguous situations.
- rustyfe 7y agoThe (1) in make(1) corresponds to which section of the manual[1] make is in. This is useful in some cases to distinguish between things that might be in multiple sections like printf(1) the user command and printf(3) the C library function. When everyone knows what's being discussed, I think it's mostly people trying to give a shibboleth that they've read the fine manual. [1] - https://en.wikipedia.org/wiki/Man_page#Manual_sections https://en.wikipedia.org/wiki/Man_page#Manual_sections
- nocman 7y agoAs the previous replies stated, the "1" is the section of the manual the page you are requesting occurs in. If you have trouble remembering what the sections are, I recommend running "man man" in the terminal. The section numbers are explained near the beginning.
- jjgreen 7y ago
- Sean1708 7y agoIt's not just Go, loads of projects use Makefiles as a collection of Bash scripts. I've never really been certain what it actually buys you...
- kerng 7y agoAs soon as you have a semi-complex project makefiles or some other custom scripts are required. Go tools alone won't do. There is this attitude that sees Go as the center of the universe, it is not. If people say idiomatic, it makes me laugh.
- robbyt 7y agoMake is available everywhere that matters, and is a simple declarative way to encompass build actions. What are the alternatives? Bash? Not declarative, and requires lots more code. Some go rewrite of Make? Not universal, possibly not maintained in the future. Rake? Ugh, Ruby. I strongly believe that make is the least worst way to build go projects, but please change my mind by suggesting some alternatives, not by complaining about the shortcomings of make.
- jen20 7y agoThis is patently untrue. Make is not available by default on Windows, which - whether you like it or not - matters as a platform for a large number of developers. _Fortunately_ some CI images (certainly GitHub Actions and Azure DevOps) install GNU Make on their windows images by default, but it absolutely cannot be assumed for average developers.
- cweagans 7y ago"Available" and "available by default" are two different things, and GP didn't claim the latter.
- deleted 7y ago[deleted]
- robbyt 7y agoFor the sake of this argument, windows support doesn't really matter. If you're forced to use Windows for a job- that sucks. You're probably used to doing lots of silly things just to get a reasonable development environment. It's like saying Make is a bad system because it doesn't support left-to-right languages like Hebrew or Arabic.
- jen20 7y agoWhile I have in the past shared this view (and vocalised it at Microsoft events...), I’m not so sure that dismissing the majority operating system is such a good idea if you want a project to gain traction.
- IHLayman 7y ago"For code generation and other ancillary tasks, Go includes the 'go generate' facility. This feature was created specifically to free developers from depending on external build tools. (https://blog.golang.org/generate)" https://blog.golang.org/generate)" Please please please don't use go generate! While I respect your position, go generate is the worst and I hope they eventually deprecate it in future go versions. We tried go generate in some of our code and it went very badly: * go generate is placed in a comment line. Comments should never be executable, they should be used for explanation. If I am trying to trace execution, I shouldn't be forced to scan through comments looking for side effects. * from the go generate man page: "Within a package, generate processes the source files in a package in file name order, one at a time." You can't order your generate commands in a way you want, you have to order them using file order, or keep all of your go generate lines in the same file, which defeats the purpose of go generate. * go generate will run every time, regardless of freshness of file. So if you need to run protoc or some other protocol buffer compiler, you have to regenerate it every single time regardless of whether or not it is needed which makes the build run way slower. * What are the dependencies of this project? If I use go generate, I have to run some clumsy grep command to (hopefully) find all of the go generate comments in the package. Sorry, go generate is to golang what COMEFROM is to INTERCAL. Please avoid if you can help it. If that means a shell script or heaven forbid a makefile, so be it.
- leetrout 7y agoGo build constraints (build tags) also go in the comments and I think it works well enough. The arguments against are similar to those against struct tags. Without a properly defined pragma paradigm in Go comments at the top of a file work Well Enough™ IMO. I've also never seen anyone call out to protoc from a go generate flag. Sure, you can do that but it hasn't been common in the Go shops I've worked at. > You can't order your generate commands in a way you want, you have to order them using file order, or keep all of your go generate lines in the same file, which defeats the purpose of go generate. I've never had a problem with confining generated code to a single file. Do you have more details on why this is a problem? Given that all the files in a package are "flattened" I don't see why this causes a problem...
- rad_gruchalski 7y agoAnd when you distribute the project, how do you „document” all possible build / packaging / release / test options. Shell script? Readme? I look at things like Jaeger and all I see is a Makefile with all possible operations for that project neatly placed in a single portable, actionable format. If I have no make, sure, I’ll copy paste the command and run manually. But why would I? edit: spelling
- GhettoMaestro 7y agoWho cares, really? I use Makefiles for everything from eliminating 8284738 random bash scripts to orchestrating global infrastructure deployments with Terraform in a docker container. None of my above fits your “correct” view of make. But it works fine and has for years.
- jrockway 7y agoIf you download some random tar file with go code in it, I agree that you should expect to be able to build it with only go installed. "go get" depends on this, and largely works well! But the day to day act of developing a system that uses go involves more than just building a go binary. Your go program might depend on things like generated protocol buffers. You need some way to regenerate those when you edit the definitions. That then involves having the right version of protoc installed and also having the right version of protoc-gen-go. The go compiler can't help you there. go generate suffers from the same problem; it's not automatic, so you can pass it the wrong dependencies (generation tool flags, version of the generator, etc.). People are using makefiles as a convenient place to write all these extra instructions. "What flags to I pass to protoc?" "What flags do I pass to docker build?" Why document it when you can "make protos" or "make container"? Unfortunately, make isn't actually good at this. It doesn't version the generation tools (or itself), so you will end up with vastly different results on different machines. The result is things like a 300 line diff to a generated protobuffer because the second engineer to work on that file happened to have protoc 0.7.7 instead of protoc 0.6.42, or they installed protoc-gen-go@master instead of protoc-gen-go@v1.2.3. Make doesn't care. It exited with exit status 0, so it must have worked. What started as a nice way to write down some instructions for hacking the code has now become a giant mess. Reasonable makefiles can only ever work for one person on one computer at one point in time. At that point, they might as well be a README.md. At least the README can mention the version numbers of the dependencies, and, most importantly, can wish the reader luck. There are two long-term solutions. One is to only use go. Write a program that reads the protos at runtime. Write a program that runs your Typescript through a hand-written compiler whose source code lives in your project at runtime. Now you only need "go build". This is... impractical, though. It's a nice ideal, but you'll never get anything done in the real world. So what you really need is a real build system that captures every dependency, knows about high level tasks ("make foo.proto available to a go program"), and knows every dependency between files like the go compiler does. With such a tool, you can get a working build on every computer with no instructions or manual setup. And since it is carefully written to understand what it's doing, you can get reliable incremental builds. (The full build and your incremental build should have the exact same md5sum of the resulting binary.) Such a tool does not exist. bazel is close. If you have to build more than just go files, you probably want to look into it. It's crazy. It's a lot of work. Don't do it if you're the only developer on the project. But if you want 10 random people to be able to build a project that's written in more than one language, you have to invest in some sort of tooling. Make is good for a one person team. Make can be scaled to do crazy things poorly (hi, Buildroot!). But it's probably not what you want to be using. If a README isn't good enough, you need a real build system.
- ajross 7y ago> The first is that make was developed for building C projects This is sort of a misconception. The C compiler was developed for building C projects. Make exists because those projects had to build other stuff and needed a way to stitch the files together. Make's only built-in support for "C" amounts to some default rules for building .o files out of .c files. If all you want from your build system is to compile a big unified blob of source in a single language into some kind of output file (like the examples you cite) you don't need make, just use whatever it is that your local language provides. When you have requirements that go beyond that, where you have programs (often themselves built locally, and often in variant languages or runtimes) generating custom intermediates and need to track that madness, that's when you need a more complicated build manager than your compiler provides. And that's when you start to understand why, despite four decades now of attempts to replace it, some of us still reach for make.
- ascotan 7y agoIf you're using windows for development you're doing it wrong.