9 ms·
Make: Theory and Practice
- osivertsson 12y agoThis seems like an excellent and concise piece on how to best use Make, and one I will share with other developers. Thank you for writing it and posting it here!
- rspeer 12y agoI'm currently using Make as part of a data-building pipeline. It's nice that it's a build system that doesn't assume my goal is to build a binary or a JS file, and that it's remarkably easy to parallelize. One deficiency I've come across, which seems to be well-known, is the M:N problem -- where one step takes M input files and has N output files. Make rules seem to expect to have only 1 output, and the workarounds like .SECONDARY prevent some features of Make from working correctly. I've also seen this limitation in many of the fancy new build systems that get posted here on HN. Is there a build system, a modification to Make, or anything out there that does keep track of builds with multiple outputs? Not an I/O-guzzling MapReduce framework, please.
- jgrahamc 12y agoSee also: "Atomic rules in GNU make" http://www.cmcrossroads.com/article/atomic-rules-gnu-make http://www.cmcrossroads.com/article/atomic-rules-gnu-make
- pornel 12y agoBroccoli.js uses directory tree as the primitive http://www.solitr.com/blog/2014/02/broccoli-first-release/ http://www.solitr.com/blog/2014/02/broccoli-first-release/
- tomn 12y agoI've used SCons for processing audio files recently and it has worked pretty well for me; multiple targets are handled just fine. I did have one percieved speed problem, but this scons calculating file checksums to see if it needed to re-build targets; this is usually a good thing, but if you're working with large files it might be better to use the MD5-timestamp Decider instead.
- oso2k 12y agoHave you read the Make pattern rules examples [0]? There's the specific example of yacc/bison doing something similar to as your describing (at least for 1:N case). %.tab.c %.tab.h: %.y bison -d $< [0] http://www.gnu.org/savannah-checkouts/gnu/make/manual/html_node/Pattern-Examples.html#Pattern-Examples http://www.gnu.org/savannah-checkouts/gnu/make/manual/html_n...
- DSMan195276 12y agoIIRC, I got around the M:N problem with GNU make by defining a pattern-matching rule to do what I want. For example, I was using yacc, so I did this: %.tab.h %.tab.c: %.y stuff Using pattern matching forces make (GNU make) to correctly treat this as one rule which outputs multiple files, instead of the default of it being multiple rules which each output one file. It's not perfect, but it is a simple fix for some of those issues.
- ploxiln 12y agoThat's actually one rule which applies to multiple targets. If you run without parallelism (the default, -j1), it'll work fine, since make will run the rule for one target, and then when it comes to the second one, make will notice that it's already up-to-date (based on file timestamp). If you run with -j4 or something, it make will likely run the rule twice, simultaneously. That may or may not be a bad problem depending on how the rule writes the output file. EDIT: man I must have really rushed reading your post... as you say, this does appear to be a heuristic for GNU Make when it's a pattern rule. clearly make is tricky and I end up learning stuff every time I use make...
- evmar 12y agoThe automake manual has a description of how to get the dependencies right for multiple outputs: http://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html http://www.gnu.org/software/automake/manual/html_node/Multip... . (Disclaimer: I'm no expert, and the Make book mentioned elsewhere in this thread may have a better solution.)
- oavdeev 12y agoMultiple outputs are tricky, what you going to do if you run a step and it somehow produces 2 out of 3 expected outputs? I mean, you need a way to decide whether whole thing failed or not, easiest way is indeed to have dummy targets. Anyway, take a look at https://github.com/spotify/luigi https://github.com/spotify/luigi, it is basically make-like tool geared toward data pipelines. We are experimenting with it for our data pipeline (which is quite sizeable, ~30TB/day gzipped, thousands of files, a bunch of processing steps) and while it lacks some things, especially in the UI, this approach seems to work quite well.
- stormbrew 12y agoWhy would you not want to consider the step failed if it didn't produce all its expected targets?
- oavdeev 12y agoYeah, that's my point: if you're happy with this all-or-nothing logic, you can just use dummy target files. Allowing steps to partially succeed seems like adding quite a lot of complexity, and I'm not sure where it would be beneficial.
- grayclhn 12y agoDummy target files don't work if the files are deleted elsewhere --- i.e. not as part of the make pipeline.
- cafebeen 12y agoI usually create dummy targets for that kind of thing, and then include loops to create dependencies of each of the N on the dummy task. Alternatively, you can create a directory for the output and use that as a target, although you need to ensure the target directory only exists once the M:N task has finished.
- rspeer 12y agoI'm pretty sure everything you're suggesting here is a workaround that breaks the intended features of Make. If a multiple-output step fails, I want the outputs to be deleted, and the rest of the dependencies to keep working so I don't have to rebuild everything from scratch. Whatever you're suggesting with dummy targets sounds like it's certainly not going to track the dependencies correctly. I already know that directories aren't tracked correctly as dependencies.
- cafebeen 12y agoYeah, you can wrap it up in a script that only produces the expected outputs on a success. As for workarounds... this is hacker news, is it not?
- evmar 12y agoFWIW, my tool Ninja (was just on here again this week here: https://news.ycombinator.com/item?id=9282539 https://news.ycombinator.com/item?id=9282539 ) was designed to be more or less the same idea as make with some semantic fixes, and that includes supporting multiple outputs. You can see the brief list of improvements here: https://martine.github.io/ninja/manual.html#_comparison_to_make https://martine.github.io/ninja/manual.html#_comparison_to_m... (Within Ninja this manifests concretely as needing to represent the build graph as a bipartite graph between files and commands. Though I don't know make's internal representation, I imagine the straightforward implementation is as a graph between files and I can imagine why it'd be difficult to make multiple outputs work in that case.)
- ploxiln 12y agoI really like the minimal and solid design of ninja; I'd agree that if make isn't cutting it, ninja + (your own script to generate all the correct rules) is probably the best way to go.
- JoshTriplett 12y agoIs there some fundamental reason why the approach used in Ninja couldn't be used to accelerate a substantial subset of Makefiles? If you dropped Make's built-in rules, more complex functions, VPATH, conditionals, and similar items the Ninja manual describes as slowing down Make (all of which are non-portable features of GNU Make), while mostly keeping compatibility with the portable subset of Make, is there some fundamental reason that couldn't be accelerated to the same degree as Ninja?
- evmar 12y agoNo, they should be equivalent. The Ninja manual goes so far as to encourage you to continue using Make if Make is fast enough for you. However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all. Because we cared more about performance than other things we maybe were a little crazier about shaving off milliseconds than other systems, so it might be hard to make make fast enough. But (as the Ninja manual suggests) this may only matter for very large (Chrome-sized) projects. You can read a chapter about some of the performance work done on Ninja here: http://www.aosabook.org/en/posa/ninja.html http://www.aosabook.org/en/posa/ninja.html
- serve_yay 12y ago> One deficiency I've come across, which seems to be well-known, is the M:N problem -- where one step takes M input files and has N output files. Make rules seem to expect to have only 1 output, and the workarounds like .SECONDARY prevent some features of Make from working correctly. Jeez, you know, this is the sort of thing that I would hate to get somewhat deep into using make, only to then discover. I often see hackers lamenting that a plethora of build tools exist instead of everyone just using make, but I'm not so sure that this state of affairs is less preferable. Maybe it's better if the build tool understands some things about the platform it's building for.
- emelski 12y agoYou can actually articulate that relationship in GNU make, as long as your outputs can be coerced into a pattern rule (like "%.c %.h: %.y"). Otherwise there's no way to do it that's reliable in the face of both incremental and full builds. There's been discussion of adding syntax to GNU make for this purpose on-and-off over the past several years, but nobody can seem to agree on what the syntax should be. With Electric Make you can mark the rule as producing multiple outputs by adding "#pragma multi" above the declaration. See http://blog.melski.net/2013/01/01/pragma-multi-and-rules-with-multiple-outputs-in-gnu-make/ http://blog.melski.net/2013/01/01/pragma-multi-and-rules-wit.... Disclaimer: I'm the Chief Architect of Electric Make
- jhallenworld 12y agoShould read this along with it: http://www.conifersystems.com/whitepapers/gnu-make/ http://www.conifersystems.com/whitepapers/gnu-make/ In my current very huge (takes hours to build) project at work there is a lot of voodoo around the amount of parallelism you can use. "make -j 4" seems OK, but "make -j 20" fails. Of course, nobody wants to work on improving the build system.
- jgrahamc 12y agoSee also: The Pitfalls and Benefits of GNU Make Parallelization http://www.cmcrossroads.com/article/pitfalls-and-benefits-gnu-make-parallelization http://www.cmcrossroads.com/article/pitfalls-and-benefits-gn...
- hyc_symas 12y agoWhat's sad is that I wrote about this about 25 years ago when I first wrote gmake's -j feature. http://highlandsun.com/hyc/#Make http://highlandsun.com/hyc/#Make The Chrome/Ninja discussion reminds me of the old mess of X11 IMakefiles. Apparently nothing has really improved since then.
- jgrahamc 12y agoNot a bad introduction. I spent years working on GNU make stuff (including a complete emulation) at Electric Cloud and wrote an "Ask Mr Make" column. Everything I wrote about GNU make can be found here: http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-arti... [blatant advertising] And also I have a new book on GNU make coming out this month: http://www.nostarch.com/gnumake http://www.nostarch.com/gnumake if you really want to get into make deeply.
- EliRivers 12y agoI have a (signed and with some nice cloudflare stickers, thank you very much) copy of your previous GNU Make book. How much better is this one? (Disclaimer; Mr GC is totally not paying me to ask this)
- jgrahamc 12y agoThe previous book was self-published, self-edited, etc. This is a properly edited and updated version of that book. Covers up to GNU make 4.0. If you are one of the few who bought the old one then it might not be worth buying the new one. But this book does have exactly 2^8 pages :-)
- angersock 12y ago...do we have to get into make deeply? ;) Like, can't we just get into a little make, and call it square?
- jackmaney 12y agoGNU make is one of the (many) things that I've meant to get a grasp on. I picked up an electronic copy of the book. Very nice! It's a lot more example-driven than the GNU Make Manual.
- teddyh 12y agoIf anyone wants to get into GNU Make deeply, one would think that the obvious thing to do would be to read the official manual for GNU Make. Anyone who writes their own, not freely available, manual to a free software program, is dividing the world into two halves: Those readers who can afford this new book, and will pay the author (not the writers of the software, mind you) for a nice manual, and those readers who can’t afford to pay for this new manual, and will have to do with the inferior¹ official manual. Those readers who can afford the new manual will not be incentivised to improve the official manual, since they have the new, better, one. Therefore, this act (of creating new manuals which are not freely available) is reducing the number of readers who can be expected to help improve the official manual to those readers who cannot afford the new manual. These effects, i.e. reduced funding and reduced help to the original software authors, is not, to me, worth it to have a new, even if better, manual. ① Presumably, by their own standards, since they wrote a replacement manual.
- amelius 12y agoIncremental and parallel computations, like building software in a development environment, is better done using some kind of purely functional system, imho.
- cafebeen 12y agoI'm curious--any pointers to example packages?
- ambrop7 12y agoNix & NixOS - http://nixos.org/ http://nixos.org/ But, in practice it is used more on the level of packages and configuration than as a build system within a package. I remember reading an article demonstrating its use for building C code but I can't find it.
- jayridge 12y agomakes my day
- q2 12y agoThinking of make as an example of "declarative programming" helps in appreciating its functionality. This aspect needs to be highlighted to those who have knowledge of imperative/object oriented programming paradigms only. Otherwise, understanding how those makefiles actually work can become confusing and painful.
- thinkingkong 12y agoThe reason I love make so much isn't because it's a super reasonable way to build projects, it's because it has the best interface for building projects in a way that can apply to all languages. I wrap lots of other build tools in make, so that the way things happen always follows make setup && make build && make (deploy || install) No matter the underlying tools. It just makes getting started with one of the many dozens of repos we worth with easier.
- Animats 12y agomake is a good idea gone bad. It's supposed to be dependency-based, but it has no mechanism for finding dependencies. It has to be told about them manually. So kludges have been written to find dependencies, but they have to overwrite makefiles. Its analysis of what has changed is purely based on time ordering, so it can get confused. This leads to too much "make clean". Although it wasn't intended as a macro processor, it's turned into one. Then, of course, there's "./configure". The alternatives tend to be bundled with some giant IDE, or are language-specific. The trend seems to be towards the latter; Go has "go", and rust has "cargo".