14 ms·
Makefiles – Best Practices
- dilawar 8y agoThis is pretty cool summary. Thanks.
- jedimastert 8y agoFor those interested, there's also this writeup of making portable makefiles on nullprogram[1] [1]: https://nullprogram.com/blog/2017/08/20/ https://nullprogram.com/blog/2017/08/20/
- girzel 8y agoI recently decided it was time to get a better understanding of how makefiles work, and after reading a few tutorials, ended up just reading the manual. It's long, but it's very, very well written (a good example of one of the Gnu projects biggest strengths), to the point where just starting at the top and reading gives an almost tutorial-like effect. Just read the manual!
- jgrahamc 8y agoYes. It is highly recommended that you read that book (buy it to support the FSF). If you want more I wrote a book on GNU Make (https://nostarch.com/gnumake https://nostarch.com/gnumake) which takes things further than the GNU Make manual. A large amount of the content of the book came from a sequence of blog posts on GNU Make by me: https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-art...
- deng 8y agoYour book is awesome, thanks for writing it! I converted our whole build at $DAYJOB from recursive to non-recursive, and your book helped immensely. It was a lot of work, but in the end it was worth it.
- jgrahamc 8y agoThat's very kind. Weirdly, I have never done a conference presentation about GNU make. If I did what would people want to hear about?
- deng 8y agoI think the main point nowadays is how to write Makefiles for highly parallel builds. It's surely the thing that gave me the most headaches. In the end such a talk will probably mostly be about how to avoid calling Make recursively. Another talk I'd like to hear, which would be more meta: does it still make sense to bother with Make at all, aside from maintaining legacy systems? Are there features or paradigms which still make it worthwhile, compared to other popular build systems?
- LukeShu 8y agoI don't think the FSF is currently selling physical copies of the Make manual.
- jgrahamc 8y agoAh. I guess I am old and like books. Give them some money instead: https://my.fsf.org/donate/ https://my.fsf.org/donate/
- Nicksil 8y agoI really like the GNU Make documentation, as well. Links to the docs in various form: https://www.gnu.org/software/make/manual/ https://www.gnu.org/software/make/manual/
- chubot 8y agoFWIW I also read the GNU Make manual, and based some code for automatic deps off a profoundly ugly example it had. Then later people on HN showed me a better/simpler way to do it. https://news.ycombinator.com/item?id=15060149 https://news.ycombinator.com/item?id=15060149 https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html https://www.gnu.org/software/make/manual/html_node/Automatic... After reading the manual and writing 3 substantial Makefiles from scratch, I still think Make is ugly and, by modern standards, not very useful. In my mind, the biggest problem is that it offers you virtually no help in writing correct parallel and incremental builds. I care about build speed because I want my collaborators to be productive, and those two properties are how you get fast builds. GNU Make makes it easy to write a makefile that works well for a clean serial build, but has bugs once you add -j or when your repository is in some intermediate state.
- klodolph 8y ago> In my mind, the biggest problem is that it offers you virtually no help in writing correct parallel and incremental builds. That’s interesting, because in my mind, parallel and incremental builds are the main features of Make, the features that it is best at, and all other features are secondary. It sounds like your problem is with the correctness part. Make gives you no tools to enforce that your build rules are actually correct. Very few build systems provide any help here. What you are looking for is hermeticity. Bazel does this by sandboxing execution of all the build rules, and only the specified rule inputs are included in the sandbox. I recommend Bazel. Otherwise, it is up to you to get your build rules correct, and switching build systems won’t help in general (although they may help in specific cases). I find it surprising that you talk about using Make for a clean serial build, because if you want a clean serial build, you might as well use a shell script. Make’s only real purposes are to give you tools for incremental and parallel builds. Nearly any other tool you replace Make with will either have you sacrifice incremental/parallel builds or will give you the same hermeticity problems you would encounter with Make. Replacing Make, the main paths I see are towards improved versions of the same thing (e.g. Ninja, tup), completely redesigned versions of the same thing (Ant, SCons), systems and languages which generate makefiles (e.g. autotools, CMake), and the new wave of build systems which provide hermeticity (Bazel, Buck, Pants, Please). This last group is a very recent addition. Mind you, Make is old and not especially well-designed, and it has plenty of limitations, but it’s good enough at incremental/parallel builds that it has stuck around for so many decades. Make is good enough at what it provides that replacements like Ant, SCons, Tup, Ninja, etc. don’t seem like much of an improvement.
- pbalau 8y agoWhat if I want to use makefiles on a mac? Make of macs is not gnu make. /edit: bugger, after a make --version on a mac, i've got: GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. Back to the begining, why is this bloody thing not working...
- kawera 8y agoFYI, homebrew installs version 4.2.1. Invoke it as gmake.
- rlue 8y agoCan you humor a dumb question and tell me how you got the manual in your terminal? I'm on Debian, and when I `man make`, I just get a typical GNU CLI util manpage that's five paragraphs of intro and a line-by-line explanation of each command line option. Same for `info make`.
- aaaaaaaaaaab 8y agoCFLAGS := ${CFLAGS} CFLAGS += -ansi -std=99 What's the point of the first line? Why not just: CFLAGS += -ansi -std=99
- pooya13 8y agoMy guess is that the left hand side is a make variable while the right hand side is an environment variable. However the make manual says “Every environment variable that make sees when it starts up is transformed into a make variable with the same name and value.“ so maybe it’s not necessary?...
- pooya13 8y agoThis reminds me why I stopped visiting HN. People down vote you for trying to help! Screw whoever asshole that downvoted me even though I prefixed my answer with “My GUESS is...”.
- dewhelmed 8y agoBecause CFLAGS may not be set in the environment, resulting in concatenation to an undefined variable error.
- jgrahamc 8y agoNo such error occurs. $ cat Makefile FOO += foo all: ; @echo $(FOO) $ make foo
- dewhelmed 8y agoMy bad, I felt like I've seen that error pop up before in my Makefiles, but I should have double-checked before commenting.
- jschwartzi 8y agoThere's no such thing as an undefined variable in Make. The variable expands to text. Variables which are not defined by definition expand to no text.
- DblPlusUngood 8y agoA particularly tricky task with GNU make is automatically adding target dependencies on header files and handling updates to them (gcc's -M and -MMD switches). It would be great if the article explained those best practices, too.
- jgrahamc 8y agohttps://www.cmcrossroads.com/article/tips-and-tricks-automatic-dependency-generation-masters https://www.cmcrossroads.com/article/tips-and-tricks-automat...
- AceJohnny2 8y agoThanks John, your book is the most-referred to on my office shelf ;) Edit: https://nostarch.com/gnumake https://nostarch.com/gnumake (via https://blog.jgc.org/2015/04/the-gnu-make-book-probably-more-than.html https://blog.jgc.org/2015/04/the-gnu-make-book-probably-more...)
- jgrahamc 8y agoThat'a great. Glad it was helpful. If you have any of your own GNU Make tips and tricks please do blog about them!
- nrclark 8y ago+1 for a great book, I bought that book too! I use the patterns in @jgrahamc's article "Escaping: A Walk on the Wild Side" pretty often. Especially the \n definition, which works great to put on the end of a $(foreach) loop inside of a recipe. @jgrahamc: Thanks for all you've written on Make over the years.
- mturmon 8y agoI also recommend this book. The series on “meta programming make” is also quite good: http://make.mad-scientist.net/category/metaprogramming/ http://make.mad-scientist.net/category/metaprogramming/
- tedmiston 8y agoThere are so many gotcha with makefiles yet we still use them regularly. I'm considering moving on to something like Taskfile [1], though I haven't tried it yet. [1]: https://taskfile.org/ https://taskfile.org/
- igor47 8y agoI've been using invoke [1] which has allowed me to give my projects nice UIs while remaining in the primary language. If I were working on Ruby I would stick with Rake, even though I think invoke is better. [1] http://www.pyinvoke.org/ http://www.pyinvoke.org/
- learned 8y agoI'll second invoke. I use it for all of my python projects now and I love working with it. It makes it very clean to manage more complex tasks that have a lot of conditionals involved.
- tedmiston 8y agoI really like Invoke / Fabric for Python projects, though it feels weird to use in non-Python projects when it comes to dependencies. [Now I have to explain to non-Python people things like virtual envs, pipenv vs pip vs pipsi, pyenv, etc.] Just curious if you install Invoke individually for each project or keep one global install? I suppose I could always git exclude the tasks.py Invoke files.
- learned 8y agoI usually keep Invoke globally installed that way I can use it for setting up and interacting with pipenvs externally. Most of my calls beside the initial setup make use of 'pipenv run <command>' that way I never actually have to navigate inside of the virtualenv for most cases.
- SamWhited 8y agoI decided a long time ago that it's not worth having to install and learn a different tool for every ecosystem just to avoid make's quirks. Yes, make isn't perfect (it can even be quite annoying at times), but neither is everything else so it's worth it to me (from a personal and business perspective) to just make my developers use one thing for all projects, especially since it's already installed (or easily installable if not, there are packages for everything which may not be true of all the less popular alternatives) on most of their systems. Not that I couldn't ever see making an exception to that, I'm sure there are some things out there that don't work with the make model and really do need their own build system, but in general make is "good enough" and it's not worth using anything else.
- umvi 8y ago"Makefile, Best Practices: Use CMake to generate them for you"
- nomel 8y agoYou joke, but I've personally never seen a hand made make build systems, in any company I've worked at, that could handle "fuzzing" by deleting or modifying random files in the build system and having the expected output. They all required a "clean" to get back in order. For those that maintain handmade make based build system of reasonable complexity, I challenge you to "fuzz" it before each build. It will not be a comfortable experience. On the other hand, I've seen many generated makefiles that behaved correctly that are totally unreadable.
- Orphis 8y agoMakefiles, Best Practices: Don't write Makefiles, use a higher level language to describe your goal and some tool to execute it (either directly or through generating a Ninja file). The Make language is the assembly language of build systems. Do you really enjoy writing assembly all the time?
- jopsen 8y agoMake is simple.. it gets me started. I can do multiple targets with dependencies and incremental compilation.. Sure, you should be careful of scaling it far. BUT, please enlighten me on a sane high-level build system? cmake, bazel and the like all seems to rely on unintuitive macros. And let's not talk about auto tools :) What other mainstream build system will get you started quickly without unintuitive macros. (I won't claim that makefiles are often intuitive, hehe)
- Orphis 8y agoWhat's more simple than "add_executable(foo bar.cpp)"? It's concise, cross-platform and hides all the boilerplate for you. Sure, everything in CMake is not super intuitive, but I've led workshops were people got the hang of it pretty quickly for simple to advanced cases (including multiple targets and some scripting). Keep it simple, most of the time, you don't need the advanced features at all! Can't say the same for Make, apparently, people need to write about it still today! And most of the Makefiles are just plain wrong and won't make any reproducible and reliable builds. That's a big no in my book for a build system!
- ghthor 8y agoIts here I look to KISS to guide me. A Makefile is simple and you can expect it to be on most systems, so it's a good place to start stringing things together. It helps you start building a CLI UI with relative ease, once you know you should mark all ephemeral targets as .PHONY. As a project grows and gains users or requires more system support, the growth of Make usage to declare interdependence should always be discussed. At some point the team will need to start using a different tool, perhaps isolating Makefile usage to bootstrapping the bigger tool. The reason I reach for Make is I know it, and it's rather intuitive for someone to glance at my Makefiles and get an overview of where to look next.
- ufo 8y agoDoes the "?=" operator behave like ":=" or like "=" ?
- jgrahamc 8y agoBoth $ cat Makefile $(info FOO $(flavor FOO)) FOO ?= foo $(info FOO $(flavor FOO)) BAR := $(info BAR $(flavor BAR)) BAR ?= bar $(info BAR $(flavor BAR)) BAZ = $(info BAZ $(flavor BAZ)) BAZ ?= baz $(info BAZ $(flavor BAZ)) $ make FOO undefined FOO recursive BAR simple BAR simple BAZ recursive BAZ recursive If undefined then becomes recursive, otherwise the flavour is preserved.
- wahern 8y agoMake variable expansions are always recursive. For what you're describing I think the proper term as used by the GNU Manual is deferred. In other languages the phrase lazy evaluation is common. The one thing I've always had trouble remembering with GNU Make is precedence. Variables defined as command-line arguments (make FOO=bar) override assignments, but environment variables (FOO=bar make) only override ?= assignments. != is both immediate (the shell expansion) and deferred (the result of the shell expansion). I feel like there's some inconsistency when variables are inherited across recursive invocations, but in a simple test with make 3.81 (macOS) I couldn't find any. Maybe my suspicion is just a byproduct of my weird inability to remember the precedence rules. I normally try to stick to portable make these days, anyhow. Between the ancient version of GNU Make on macOS, OpenBSD Make, and NetBSD Make, you're mostly stuck with POSIX syntax. If you add Solaris' default make, and especially if you add AIX' make, you can't rely on any extensions at all. For me sticking with portable make is easier than installing and maintaining GNU Make on every flavor of operating system I test on, and definitely easier than installing autotools or installing and maintaining the most recent version of CMake.
- jgrahamc 8y agoThe GNU Make manual defines two variable types: recursively expanded and simple (see: https://www.gnu.org/software/make/manual/html_node/Flavors.html https://www.gnu.org/software/make/manual/html_node/Flavors.h...).
- brookhaven_dude 8y agoWhat about using Python (with system calls) to build the project? Much more understandable than Makefiles.
- dbcurtis 8y agoOK, so I am as much of a hard core Pythonista as you will find. But I don't think it makes any sense to build C projects (or many other languages) with Python instead of makefiles. C programmers know make, the makefile idiom has been evolving for 40+ years. A seasoned C programmer is going to look at an idiomatic makefile and find it much more readable than some rando doing some one-off Python script to control a build. And getting build dependencies resolved correctly so that you can only build what is need is not trivial. What I truly hate is all the IDE's that think they can do a better job than make. This is the curse of the embedded world. Building an embedded project requires calling particular tools, with peculiar flags, and I hate with a passion IDE's that obscure all of that in some crappy XML file that is git-unfriendly. A well-structured makefile is your friend in many ways. make is a powertool. Learn it and your life will be better.
- htfy96 8y agoAs a C developer I have to say there exists no idiomatic makefile - configuring header file dependencies with -MMD/auto rebuild targets affected by FLAGS change already makes your Makefile look like magic. As the project grows, eventually you will need some kind of flexibility that makefile cannot do well. In this case, explicit Python scripts become more useful that a bunch of possibly implicit makefile rules.
- kingosticks 8y agoPerhaps we need some kind of make8 tool to check for best practices and prevent us writing these magic make files.
- self_awareness 8y agoMy experiences with Makefiles are different. I can understand mine up to 3 months after I write them. And if I read someone else's Makefile, it's a binary blob, full of hacks, optimalizations, walkarounds and compatibility quirks between GNU make, BSD make, with sometimes different Makefiles for FreeBSD, Linux, OpenBSD, DOS, and Visual Studio uses its own project file in the win32 directory. It's a mess.
- SamWhited 8y agoI really wish blog posts like this would at least mention that they are going to use GNUMake extensions. There's some good stuff in here, but just calling it "make" and leaving it at that is misleading. It's not going to work on my minimal (non-GNU) Linux boxes that mostly run NetBSD make (bmake in many distros) or busybox style tools. I know a very small minority of people do something like that (or maybe not, what does Alpine run in containers?), but it seems worth at least a quick footnote to say that most things in here won't be portable.
- tynorf 8y ago/ # make --version GNU Make 4.2.1 Built for x86_64-alpine-linux-musl Copyright (C) 1988-2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
- SamWhited 8y agoGood to know; thanks! I'm kind of suprised; given how tiny Alpine is supposed to be and how they're muscl based I assumed they would have taken that philosophy into the userspace tools and used busybox or something. Not that I really know anything about Alpine except that it's nice when you just want something small.
- rcfox 8y agoI doubt most people even know that there are different versions. (I didn't!)
- dmix 8y agoIt's pretty common knowledge that there are differences from the GNU package set and linux package sets. I've been exposed to this multiple times: from downloading GNU packages using Homebrew on OSX, downloading different packages on Android, the variety of options available to ArchLinux users, etc. Anyone with basic Linux knowledge should be well aware of this. From a minimum of following tutorials and having the basic command flags not work on common terminal programs.
- corndoge 8y agoNext up: Automake, best practices Publication date 2025 5000 pages
- jandrese 8y agoOr a single page with a single word: No
- kstenerud 8y agoI've been using makefiles for decades, and finally decided to try something more modern. It turns out that CMake has come a LOOOONG way, and is almost nice to use, aside from the fact that it's horribly complicated. But with some basic templates to work from, it's actually pretty easy to get a project started. https://github.com/kstenerud/modern-cmake-templates https://github.com/kstenerud/modern-cmake-templates
- shrimp_emoji 8y agoOr just use Rust. :3 Cargo handles makes with very simple TOML scripts, and it's the more modern language, to boot.
- kstenerud 8y agoI've been planning to, but unfortunately it still lacks 128 bit float and decimal types, which I need.
- burk96 8y agoYou may be interested in this crate which adds decimal types https://crates.io/crates/rust_decimal https://crates.io/crates/rust_decimal. Here is another that I have not used but appears to add a f128 type https://crates.io/crates/f128 https://crates.io/crates/f128.
- phyllostachys 8y agoEvery time I start a little thing in C or C++ and I write a corresponding Makefile I get annoyed by it and write the thing in Rust. Cargo is really nice. I've considered using a Cargo.toml and build.rs file just to build C and C++ files before.
- int_19h 8y agoTake a look at Meson, too.
- hzhou321 8y agoBest practice is to keep it simple. For example, if the Makefile is written in such a way that lazy set or immediate set doesn't matter, then it is not complex at all. But these facilities are there for achieve logic. Complex logics are complex, especially when they are entangled. For example, we often use build tools to generate Makefile. Some of the logic is implemented in the build tool; some of the logic is implemented using Makefile -- of course the result is complex. There is no rule of thumb on how to organize complex logic, that is what programmers are paid (so much) to do. Respect your job and treat complex logic carefully, then you won't make too much a mess -- at least do not complain about it after making one.
- nrclark 8y agoGNU Make is an awesome task-runner. I use it all the time, for all sorts of things. It's my shell-script replacement, and I write one-off Makefiles frequently. It's got a nice macro-processor, some helpful string manipulation tools, and expresses task dependencies perfectly. The manual is also very well written. But if you're writing a software package, consider using CMake, Meson, Autotools, or something similar. Unless you're a superhuman, any of them will handle the corner cases better than you can. Especially the cross-compilation corner-cases. It's extra work, but the people who need to build and package your software down the road will thank you. Your artisan Makefile might be an elegant work of art, but what does it do when somebody wants to build from a different sysroot? Does it handle DESTDIR correctly?
- scottlocklin 8y agoI haven't found a reason to not just use Make yet when starting from green fields, including some very big projects. In fact, I have often been annoyed at some projects picking something like CMake for no reason beyond it's "more advanced," but which ends up just being an extra dependency I have to fetch and install. If I were to pick one of the above for building large projects on linux variants, which one would you pick?
- nrclark 8y agoFor me, I mostly work in embedded Linux - that's Linux for routers, custom hardware, sometimes a Docker image, etc. That means lots of packaging work. Sometimes it means build machines that can't even run the compiled binary. I compile libraries for platforms that the original author would never have imagined. In that space, there are lots and lots of details that need to be "just so". Lots of specifics about the cflags and linker flags. Requirements on where 'make install' puts files. Sometimes restrictions where an output binary wants to search for config files (spoiler: use sysconfdir). Out of all the libraries/programs that I've had to package, autotoolized projects are the easiest to handle by far, followed by CMake projects. Hand-written Makefiles are usually painful to a packager in one way or another. The most common ones being hard-coded CC/CXX/CFLAGS/CXXFLAGS, non-standard target names, and non-standard/incomplete usage of directory variables. I know that Autotools is crusty in a lot of ways, and that the learning curve is steep. It's a nasty mix of Perl and m4, and runs lots of checks that don't matter at all. It's also what I use for all of my own libraries and programs, because the result is worth the pain (for me). So for any program that I expect more than two humans to ever compile, I'd recommend Autotools if you're a perfectionist and CMake if you're not. Within Autotools, I'd recommend Automake and Autoconf if the language is C/C++, and just Autoconf/Make otherwise. (But be sure to follow the Makefile conventions: https://www.gnu.org/software/make/manual/html_node/Makefile-Conventions.html https://www.gnu.org/software/make/manual/html_node/Makefile-...). I wouldn't recommend a hand-rolled Makefile unless you're the only one using it, the build is really weird for some reason, or unless it's a wrapper around some other build tool.
- vbernat 8y ago> Using the assignment operator = will instead override CC and LDD values from the environment; it means that we choose the default compiler and it cannot be changed without editing the Makefile. People can use `make CC=clang` to override assignments. That's a pretty common use.
- BeeOnRope 8y agoCame here to say this. The "strength" of user specified variables depends on where they come from: the environment is weaker than those specified as command line arguments - which can lead to confusion.
- swills 8y agoPlease add to this list "If you use GNU Make specific syntax, use the name GNUmakefile". https://www.gnu.org/software/make/manual/html_node/Makefile-Names.html https://www.gnu.org/software/make/manual/html_node/Makefile-...
- jjeaff 8y agoI have never worked much with compiled languages but I recently started creating gnu makefiles in all my projects. I read someone's article, maybe from here, that gave me the idea. I have been putting everything in it. From the terminal commands needed to spin up the cloud infrastructure to all the docker build commands and of course all the build tasks like gulp and composer. Even my docker compose commands have been replaced with "make up" and "make down". It is really handy and has really streamlined a lot of processes when it comes to local dev environment issues with different developers. I basically just use it as an easy way to organize shell scripts. The next step is to convert my ci/cd pipeline to just use specific makefile commands from the same makefile so that everything is more portable and I can easily execute the exact same process locally when desired.
- heyjudy 8y agoIf you're using GNU make, as the author appears to be doing, makefiles should be named GNUmakefile instead of Makefile so as to not confuse the user or the tools by conflating GNU make with BSD make or other make dialects.
- solatic 8y agoMake is a very underappreciated tool outside C/C++ circles. I'm currently using it in an environment which uses Concourse as the CI tooling - Concourse takes care of version and dependency management at its level, and Make fills in the gaps for fetching the latest versions for local development environments. Because Make won't re-download files for already-made targets, local build cycles are fast after initial setup. Concourse can then re-use these Makefiles by symlinking the resources it manages before calling make, and Make won't try to re-download the resources. If you're not concerned with portability, you can even use make to fetch (and clean) Docker images since they live in predictable directories on Linux (but not OS X). More people ought to approach the tool with an open mind.