14 ms·

Why Use Make

358 points by pie 14y ago

mcginleyr1 14y ago
I agree Make is great at what it does but too many people really abuse their makefiles and don't use dependency management correctly. I suggest taking a look at http://www.amazon.com/Managing-Projects-Make-Nutshell-Handbooks/dp/0596006101/ref=sr_1_1?ie=UTF8&qid=1361732761&sr=8-1&keywords=gnu+make http://www.amazon.com/Managing-Projects-Make-Nutshell-Handbo... Really well written book. You don't have to agree with everything but its a great look at some of the better ways of using Make.
fafner 14y ago
Using make gets quickly really ugly. I wrote many Makefiles in my life and if you have to try to stick to GNU Make. At least it provides some basic functionality. But even that gets nasty quite soon when you need a simple thing like conditions with "and" or "or". If you have to write portable Makefiles ... well fsck. http://www.conifersystems.com/whitepapers/gnu-make/ http://www.conifersystems.com/whitepapers/gnu-make/ There were recent discussions about adding GNU Guile to GNU Make. I think this is a brilliant idea. Not only would a major GNU project not only use Guile. But it would make writing Makefiles much more enjoyable because you have a real programming language at hand.
- eropple 14y ago
  Personally, I don't want a programming language in my makefiles. Unless working in a very codified environment (Java and Maven, for example), I think it is vital to keep your build system as simple as you can. If I absolutely need nontrivial logic in my makefile, I would much, much rather be forced to call out to a script file (because it encourages not doing that). And if you insist on parking a programming language into it, it should probably be something with a reasonable amount of existing knowledge and users. I like Scheme, but unless the project was written in a Lisp I'd seriously question the judgment of anybody whacking Scheme into a build file multiple people had to work with and understand.
  - __david__ 14y ago
    After using Rake (ruby with some dependency DSLish stuff) on a project for a solid year, I appreciate Make much more. Rake is too powerful and makes it too easy to keep a bunch of complex logic inside the Rakefile. Our project, left unchecked, turned the Rakefile into a pile of spaghetti. Make, on the other hand, being kind of crufty and inconvenient, encourages moving complex logic out of itself and into external scripts earlier than Rake. On top of that it encourages you to have reasonable interfaces to those scripts so that they fit back into the Makefile gracefully. So ironically, Make, by being obtuse, ends up encouraging better overall project build structure.
    - ufo 14y ago
      What sort of issue did you run into. I am wondering if its osmething that could be solved by better modularization/sandboxing or if its more fundamental problems.
      - __david__ 14y ago
        It's not a programming issue per se, it's a human issue. Rake provides a nice baseline for throwing all your quickie scripts into little functions that end up with reasonable (if basic) command line interfaces for free. Those quickie scripts inevitably grow more complicated as the project goes on (usually because the project itself gets more complicated over time) and before long you've outgrown Rake. Except that since Rake is just Ruby it tricks you into thinking you haven't outgrown it! I started noticing it when I was taking a bunch of time deciding in what order I should be putting the optional parameters to my Rake tasks such that it was most convenient to the user and spending way to much code validating those arguments and setting defaults when I realized that I could've written a script using 'optparse' that would easier to document, easier to use, and easier to write and modify. There's a graph you could make where the X axis is size of the script and the Y axis is complexity (or maybe "effort"). The Rake line, drawn on this graph, starts near 0,0 but climbs and a nice steep rate. Make starts at basically the same spot as Rake but climbs way faster. A standalone command-line script starts a bit higher on the Y axis, but is flatter over all. The point at which the Rake (or Make) and the script lines meet is where you should switch to a standalone script. With Make, this happens fairly early on when things are still relatively simple. So you convert your bash commands into a Ruby script and you end up better off in the long run. With Rake it happens so late that converting to a standalone script becomes a very large undertaking and nobody wants to do it (because it still works--why mess with it?). Over the long haul it becomes a pain point.
- udp 14y ago
  The stupid thing is that conditions are supported just fine in BSD make, but the syntax is slightly different than GNU make. So you're forced to write for one or the other, and GNU make is more widely used - so makefiles end up being GNU make only, even though they don't require any GNU-specific features.
- rdtsc 14y ago
  Any thoughts on CMake for portable Makefile generation. It seems a lot projects use that.
  - fafner 14y ago
    CMake is a great example on problems with DSLs. Simply not everybody is a good language designer. CMake uses a pretty horrible language and comes with its own kinds of strangeness. I have used it quite extensively and it's probably the way to go if you want to support MSVC. But I'm not really happy with it.
- zwegner 14y ago
  Here's a very simple but powerful build system by the author of that paper: http://code.google.com/p/make-py/ http://code.google.com/p/make-py/ It's just a single ~400 line python script. The only major feature that's missing IMO is hash-based rebuild detection, which is essential if you're using much auto-generated code.
  - ycombobreaker 14y ago
    > The only major feature that's missing IMO is hash-based rebuild detection, which is essential if you're using much auto-generated code. I'm guessing that your goal here is to avoid a recompile if the auto-generated content did not change. This is not a problem for timestamp-based build tools if you add a single step: auto-generate your output to a temp file, and only replace the target with the temp file if the file shows a difference. The program `install` will do the 'copy iff file contents have changed' part for you, so it's really just two extra lines in the Makefile recipe.
    - zwegner 14y ago
      Interesting idea, I hadn't thought of that before. This only increases the complexity of the Makefile though, and for not much reason. Hashing can help in other circumstances (such as possibly skipping a linking step if the object files don't change), and it would really be much cleaner to have it as part of the build system. The last time I played around with generating code through make, it got real ugly real fast, and I don't think your 'install' trick will help much in that regard...
      - ycombobreaker 14y ago
        > ... for not much reason There's actually a good reason to do this, but it is an edge case. A timestamp-based build system can do a no-op check with very little I/O. A hash-based build system has to read all of the file contents in order to determine that nothing has changed. Depending on the latency and bandwidth of your storage, this can make a big difference in incremental builds.
mynegation 14y ago
My problem with make for data pipelines is that a lot if decisions have to be content based instead of timestamp based. Multiple platforms is also an issue, I usually don't bother installing the whole Cygwin just for gnu make. I end up with custom python scripts which may be more verbose but more flexible and almost always cross-platform.
- mbostock 14y ago
  Have you tried Waf? http://code.google.com/p/waf/ http://code.google.com/p/waf/ Seems like it might be a good fit.
  - ahelwer 14y ago
    How does Waf compare to SCons? I'm a big fan of SCons, although it annoyingly does not have transitive dependency resolution for libraries. Waf has this functionality, correct?
    - cschmidt 14y ago
      I haven't used Waf either, but some information is here: www.scons.org/wiki/SconsVsOtherBuildTools
    - ArchD 14y ago
      I found it hard to make scons build outside my source directory (instead of polluting it). This is important for example when you don't want random files showing up in your git clone. With waf, this is easy. Subjectively, waf code looks more Pythonic than scons, though both are supposedly Python.
    - wrl 14y ago
      Waf is also much nicer for end-users, if you're the sort who distributes source packages. It mimics the "./configure; make; make install" sequence as "./waf configure; ./waf; ./waf install".
under2x 14y ago
You should use CMake.
- dkhenry 14y ago
  CMake generates Makefiles and it really is only good for building software while Make can be used for any number of complex tasks. I agree that if you need a build system use CMake it will make your life easy.
  - rcfox 14y ago
    > use CMake it will make your life easy At the expense of your users. Okay, I know next to nothing about CMake. All I know is that it is extremely frustrating when I get the source to a project that uses CMake and then have to modify the CFLAGS or figure out why it can't find some include file. Of course, the same applies to makefiles generated by automake. Let's just write our makefiles by hand, people.
    - dkhenry 14y ago
      Thats is the worse idea ever, hand written make files are extremly prone to failure, moreso then CMake files. There is a way to verify that you have all your dependencies and everything is discovered rather then hard coded, but some people really like to hard code things ( CMake and make both let you do this ) The thing CMake gives you is that it will _automatically_ handle almost everything involved with creating correct make files if you tell it to. Why write the same code to discover dependencies and to have external build directories and staging directories when we can do it once and have a computer write it out for us. I see it this way Make files are great and reduce you typing at the computer by a factor of 1000. CMake is also great and reduces your typing over make by a factor of 10 to 20. Things like Go or SBT/Maven are even better and they reduce typing to almost nothing at all ( SBT doesn't require any typing to build most standalone programs )
      - krakensden 14y ago
        "everything is discovered rather than hard coded" I take it you've never used or read any of the Find.cmake files, or tried to figure out which variables you need to set in all the spaghetti. Is it _LIBRARIES or *_LIBRARY_DIR? Is the comment at the top related to reality? The awful homegrown language makes it all worse.
emddudley 14y ago
This article isn't so much "Why Use Make" as it is "Use a Build System". It even says so in the sidebar.
- mbostock 14y ago
  Yep! I wrote about Make because it's what I use and it's the most ubiquitous. But aside from the section on syntax, the post applies to nearly any build system. (And as another poster mentioned, other build systems might offer advantages over GNU Make, such as using content hashes instead of modification times, or more elegant syntax.)
mprovost 14y ago
Drake was just released, it's a make replacement specifically for data processing and seems to have some nice features. http://blog.factual.com/introducing-drake-a-kind-of-make-for-data http://blog.factual.com/introducing-drake-a-kind-of-make-for...
Peaker 14y ago
If your project sports auto-generated code, or any sort of build targets that depend on inputs only known after generation -- Make simply cannot handle this. There are ugly workarounds but they don't work well. Make is: * Slow (e.g: when compared with tup[1]) * Very easy to get it wrong (under-specify dependencies), with cryptic bugs (or over-building) as a result. Pretty difficult to get it right (e.g: doing proper #include scanning, especially with the above-mentioned generated code). * Terrible scripting language (mentioned by many others) * Lacks useful features of other build systems (e.g: tup, or shake) of useful profile reports and queries about the build process. It is just an antiquated, poor tool that ought to be replaced by betters that already exist, whenever possible. [1]: http://gittup.org/tup/ http://gittup.org/tup/
- chubot 14y ago
  Have you used tup? I have encountered on the web, and it looks very well done. But I never hear about anyone using it. Is it just under-publicized? People sometimes talk about waf as an alternative. I took a look at waf awhile aback and I felt disappointed, I forget why. I think node.js switched off waf for some reason I don't recall; either that or they didn't like it.
  - Peaker 14y ago
    I used tup, and it has its shortcomings. I would never prefer Make to tup though, even with its shortcomings. I am now working with Shake, and it seems to be the nicest I've used yet. It is not as robust as tup at verifying there are no under-specified dependencies, nor at detecting what needs to be rebuilt when a build script changes. But it is much more flexible (you get to write a build script with the full power of Haskell), it generates much nicer reports as a result, and it has some other interesting features that tup lacks. Unlike tup, you don't need to specify all the dependencies statically before building anything. Since tup has some useful features that Shake lacks, I don't think there's a clear winner here. But either one is far preferable to Make in every setting.
    - chubot 14y ago
jgrahamc 14y ago
If you use GNU Make it's worth using my GNU Make Standard Library: http://gmsl.sourceforge.net/ http://gmsl.sourceforge.net/ And also reading everything I wrote as Mr. Make: http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-arti... Or buy my book: http://www.lulu.com/shop/john-graham-cumming/gnu-make-unleashed/paperback/product-2937580.html;jsessionid=853955F416C443C6FCCCF7E7700BF62E http://www.lulu.com/shop/john-graham-cumming/gnu-make-unleas...
- gatherknwldg 14y ago
  gmsl is a wonderful, well considered library. I had to buy the book after finding it. Thanks!
- eliben 14y ago
  I'll have to respectfully disagree. IMHO, the utility of DSLs like Make is high when used very sparingly, for simple flows like the one the linked article describes. The more complex your Makefile, the less you should be using Make. If you need to write complete programs in Make, just stop bending over backwards and pick a real programming language. FWIW most non-trivial projects nowadays that use Make don't write makefiles directly but use make generators like CMake and Gyp.
  - malkia 14y ago
    Make is the assembly/C language of build system. Lots of other systems just generate makefile compatible files (qmake, cmake, premake, others), but often these fall into overly complex traps trying to explain everything both with simple unique words, and then clashing into these. Make is very optimal at the stage it is, everything above or below it it's just not that optimal.
    - eliben 14y ago
      As the assembly language of build systems, I agree that Make works well. But there, IMHO, its utility ends.
kostya-kow 14y ago
This is funny to read, especially after spending several hours trying to get a project to compile with make, while tracking down an annoying and cryptic error that doesn't even make sense. Make is a horrible tool, and is extremely hard to learn. I think it should only be used when generated by other build systems.
largesse 14y ago
Created in 1977, Make has its quirks. But whether you prefer GNU Make or a more recent alternative, consider the benefits of capturing your workflow in a machine-readable format. Like a programming language?
- stephth 14y ago
  Exactly what I was thinking. Use a scripting language you're proficient at. This article doesn't make Make look worth the trouble if you don't have a more specific reason to use it. Too much complexity, and quirks like you’ll need to delete the previously-downloaded zip file before running make can easily be avoided with a scripting language. These days any relatively convoluted task that I may need to do more than once, I cook it in Ruby (and sprinkle some AppleScript/appscript if it involves UI). Languages like Ruby or Python offer a good balance of flexibility and abstraction for this kind of automation.
  - kragen 14y ago
    If you're using a series of shell commands, each of which takes seconds to hours to run, you will be happy with make's ability to (1) run shell commands easily and (2) not run them again when they don't need to be run. You could write a dependency tracking thingy in any programming language, but at that point you might as well be using Rake or some other make replacement.
- NickPollard 14y ago
  Yes. Make is just a DSL for file-based dependency management. You could certainly write makefiles in another language, but chances are it would be a lot more verbose and probably less portable. Of course there are other make tools apart from Make, so feel free to use whatever you feel is best suited.
- triplepoint217 14y ago
  Agreed, I generate documents with graphs and calculations in them for my research, but I tend to document and automate them with python scripts rather than makefiles, because python does not make my brain hurt.
ttty 14y ago
can't I use js with Node.js instead?
- ricardobeat 14y ago
  Cakefiles are a nice CoffeeScript based alternative.
- nailer 14y ago
  Pardon? I think you might be missing a project name eg, foo.js
cgrubb 14y ago
Most systems I use have GNU make 3.81 installed. GNU make 3.81 added the special target .ONESHELL which makes code like the following possible. I'm not sure if it is a good idea, but it does remove the dependency that make has on shell programming. .ONESHELL: SHELL = /usr/bin/python VAR := lorem ipsum all: @ import re n = 3 print('make variable: $(VAR)') print('local variable n: {}'.format(n)) rx = re.compile('\d+') s = 'foo 17' m = rx.search(s) if m: print('has number: {}'.format(s)) else: print('no number: {}'.format(s))
- subleq 14y ago
  .ONESHELL was actually added in 3.82: http://cvs.savannah.gnu.org/viewvc/make/NEWS?view=annotate&root=make#l153 http://cvs.savannah.gnu.org/viewvc/make/NEWS?view=annotate&#...
coldskull 14y ago
scons is a good alternative to make. Make is ok for small projects, but it gets ugly pretty quick...and it allows you to make mistakes easily...
mseebach 14y ago
Absolutely do not use make for any new project. If you love make, it's a big, red, burning flag that you're not demanding enough of your tools and that you're not keeping up with changes in your ecosystem. There are many, many way better alternatives to make. Which one is better depends on the platform you're on. The majority of them throws in automatic dependency management for free. Yes, I know that the essence of the post is "use a build system". I agree completely. In fact, script everything. Then script your scripts. Then refactor your scripts because they are getting messy. But don't give impressionable souls the idea that "make" is anywhere near an acceptable (generalised, default) choice today.
- falcolas 14y ago
  Lots of "Don't do this" without suggesting alternatives doesn't do anyone much good. If you're on a *nix system, building C/C++ programs (or automating one-off builds like the example), what would you recommend in make's place?
  - lmm 14y ago
    CMake
    - craftkiller 14y ago
      Personally I'd suggest Premake because it uses Lua instead of rolling its own scripting language. The world would be a better place if we could all agree on one scripting language for stuff like this so no one has to look up the syntax for things like creating arrays for every individual tool. Feature-wise premake isn't entirely caught up to CMake but it has everything important. Also the premake files look a lot cleaner and more readable compared to cmake files.
      - leif 14y ago
        http://xkcd.com/927 http://xkcd.com/927
      - quotemstr 14y ago
wildgift 14y ago
To everyone with alternatives to make: Make is easy to learn. It simplifies a slightly complex task. Other build tools have a steeper learning curve. If they're more complex than the problem being solved, people won't want to adopt them. When things get complex, there's the GNU make manual and libraries. When you have to target multiple platforms, there's autotools, which is really complex and intimidating, but less complex than targeting multiple platforms. Alternatives to make seem to fit in between make and autotools. Make sucks. I've also heard that Unix sucks, and C sucks, too. Despite this alleged suckyness, these things not only persist, but accumulate improvements over the years.
- narzac 14y ago
  Definitely agree, unfortunately people always know better, even when they don't have any clue. My philosophy is "If the development environment provide you a modern, native build system(ex: go build, get), go with it, otherwise stick to make unless using autotools etc. will give you a reasonable advantage."
minsight 14y ago
This is a sloppily phrased call for automation. I can wholeheartedly endorse automation as a goal, but the idea that make can be your go-to tool for this is embarrassing. Make will get you into trouble the moment your task becomes non-trivial. Everyone has their favourite tools and no single tool is best for every job (although grep and find are often your friend). As I often say in job interviews, I'm lazy. I protect the lazy guy inside of me, because he's the one who cries out "Didn't we do this manually before? Shouldn't we automate it?" He's a good guy to have around.
jgrahamc 14y ago
Three years ago I wrote a blog post arguing in favour of Make : http://blog.jgc.org/2010/11/things-make-got-right-and-how-to-make.html http://blog.jgc.org/2010/11/things-make-got-right-and-how-to... I'm not sure I still agree with everything in that post, but Make is very terse and expressive which I like. One of the biggest problems with Make is that it's a macro language and some people don't get along well with them.
jpollock 14y ago
Everyone points at make and says, "use something better". They point at Rake, they point at Maven. However, Make has one feature that I really need, that I rely on to speed up builds: parallel dependency construction. The last time I checked, neither Maven nor Rake do this properly. Maven runs everything inside a single VM. Who wants to have to worry about multithreading in their unit tests? Rake requires thread support in the underlying Ruby interpreter (why?). I've got 64+ hardware threads (Sparc T4-1) all sitting there waiting to be spun up and do my bidding. Please help me to convert electricity to heat!
- ArchD 14y ago
  Waf can also do parallel build with proper dependencies, and it's Python-based. Have you looked at it? The documentation is rather clear, but the initial learning curve is probably steeper than make.
  - jpollock 14y ago
    Thanks for the pointer, I'll go check it out.
- catenate 14y ago
  Inferno's <{} construct dispatches processes in parallel, and gathers output in coherent blocks. I use it in credo to parallelize all the dependencies for a target. I rely on mutual exclusion to avoid building a target twice at the same time, and on cached checksums to avoid rebuilding when there have been no changes. See the sh-inferno scripts map, credeper, and cresum in the project http://github.com/catenate/credo http://github.com/catenate/credo
fnordfnordfnord 14y ago
>>(Note: use tabs rather than spaces to indent the commands in your makefile. Otherwise Make will crash with a cryptic error.) Sheesh.
- csense 14y ago
  Requiring tabs alone is reason enough to ditch make.
  - nickzoic 14y ago
    ... or use an editor with syntax highlighting for Makefiles.
jcromartie 14y ago
Everybody seems to be missing the fact that he's not talking about building a big software project. He's talking about scripting a simple workflow around a few files. Make is probably better than anything else for this, because under these circumstances it's so simple it's hard to get it wrong. Something like Rake is obviously better when you need to really program your builds.
- jackalope 14y ago
  I depend on make for recording a workflow history of my projects. There's no point in creating an alias in my ~/.bashrc for an arcane command I'm only going to use once or twice a year to transform, sync or configure a group of files. When I check a project out of version control, it's nice to have a Makefile there to remind me how I accomplished something. Make allows you to consistently specify actions in a universal way. Typing 'make edit' will open the main file in vim for me in any project. Typing 'make update' will check for available updates on any platform using the appropriate package manager, and 'make upgrade' will download and install them. Typing 'make sync' will transfer my project files to the specified $SERVER as the specified $USER with the appropriate protocol. Typing 'make install' will copy an updated configuration file and restart all of the daemons necessary for it to work. It's a great timesaver. Even when I need to write shell scripts for something too elaborate for make, I tend to create a target that runs the script, instead of trying to remember its name and all of the options.
- pjungwir 14y ago
  I wanted to say exactly this. I wouldn't choose Make to build a software project these days (probably), but I use it all the time for exactly what the author describes: recording/managing a data processing workflow. For stats/ML exploratory work, it's invaluable, because everything you do is recorded in various targets. It makes my one-off scripts not quite so one-off, so when I revisit the code six months later, I can quickly figure out what I was doing. It also means less typing, because every command is just `make foo`, and that delegates out to R/Ruby/Python scripts, shell pipelines, or whatever I need. I'm delighted that someone else is (ab)using make the same way!
RexRollman 14y ago
From what the example accomplishes, it seems to be that it would have been easier to create a shell script. Is there something I am missing?
- pyxy 14y ago
  I think shell script can work as well as makefile in most cases. The pro of shell script is not having one more tool for running your commands. You have a shell, why do you need make for describing build steps? For the moment I see a makefile as a shell script that fails when a command fails. But you can use `-e` option with sh, bash, rc shells and expect the same behaviour. The only thing shell misses (comparing to makefile) is dependencies check, you have to write dependencies mechanism yourself if you need it. Also I don't like the .PHONY stuff, I just can't get it, it feels alien. Am I missing anything obvious?
  - RexRollman 14y ago
    It doesn't sound like it to me. Thanks for the reply.
- throwawayG9 14y ago
  I just started learning about Make today, but according to the other comments in this post, there are lots of differences, like incremental builds and parallelism. It also checks things like file modification dates and whatnot.
nohuck13 14y ago
I've started using make for production data generation (not building software) instead of pure Python. Mostly to tie together Python scripts doing the real work. Pro -dependency management for free -well-known paradigm makes it easy for someone other than me to figure out where to look if something went wrong. -scripts I call out to can be focused Cons -syntax -for processes that don't generate an output (think adding data to a file in place) I wind up creating placeholder files ("file.transformA.done"). I actually want my dependency management to be terse and declarative, which is the opposite of what I'm looking for in a programming language, so it feels like a pretty natural divide.
nathell 14y ago
Make is an excellent expert system / topological sorting tool with an abominable syntax.
ChuckMcM 14y ago
I find this a really interesting conversation. On the one hand I've built some really really complicated systems with make (like all of SunOS) and some even more complicated systems with a custom build tool (the google base infrastructure packages) and there are pluses and minuses to both approaches. If you're building something small, its hard to beat a simple make file. Its easy to write, it allows you to capture dependencies and refactor quickly, and it doesn't interrupt your coding flow. If you're building something quite large there are some real productivity benefits from building knowledge of what you are building into the build system. And computational build systems (which is to say build systems where the build spec file includes the capability to do local computation) can make retargeting the same build to different environments easier.
pedrocr 14y ago
Like everyone seems to be mentioning this post is really about "you should have automatic builds" and make is just an incidental. redo[1] is another interesting make replacement that keeps most of the good points of make (e.g., simple and shell based), while having a more powerful dependency mechanism. It's even compatible with make itself allowing you to move a part of a recursively built project to redo, while having it call into make for other subcomponents. What I've done before when doing data-driven blog posts[2][3] was to write the whole analysis end-to-end in ruby, including branching out to R for stats and graphing[4]. I ended up using rake (ruby's make-like tool) to tie everything together with dependency tracking but I could just as well have written a script that would call everything in order. Doing that gives you a way to quickly reproduce results (what's discussed here) but also, together with version management, a way to go back to previous versions and use things like "git bisect" to figure out when you introduced a bug. [1] https://github.com/apenwarr/redo/ https://github.com/apenwarr/redo/ [2] http://pedrocr.pt/text/how-much-gnu-in-gnu-linux http://pedrocr.pt/text/how-much-gnu-in-gnu-linux [3] http://pedrocr.pt/text/preliminary-results-open-source-evolution http://pedrocr.pt/text/preliminary-results-open-source-evolu... [4] https://github.com/pedrocr/codecomp https://github.com/pedrocr/codecomp
mcartyem 14y ago
Inevitably, any build system becomes part of the software. With make this means you are now using a far more inferior language to build software.
eliben 14y ago
I think what you were looking for here is actually a shell script, rather than a Makefile. A shell script is much more suitable for general purpose programming. Or better yet, use Python ;-) And, let's admit it, for small projects dependency analysis is overrated (although it can be implemented easily enough in a real programming language).
- apw 14y ago
  One thing make allows you to do is incremental runs. For example, if the first step in your shell script costs 1/2 hour, you really don't want to run the shell script over and over again when you are debugging the next step. Yes, you can implement your own caching logic in your shell script, but its usually much better to use a tool specifically designed for that job.
robot 14y ago
"Makefiles are machine-readable documentation ... " Yes it is machine readable but how about us humans? Makefiles are ugly and when working on someone else's project it is very hard to reverse engineer the build system. Build systems are themselves software projects and we need better tools to develop and maintain them. SCons was a promising project at one point, it improved things by capturing the build system in Python classes. I thought things would be more maintainable and readable. However for me it wasn't a well design, it obscures build system development with mixed declarative and iterative programming. Makefiles are the defacto standard today, but they're no where near beautiful, or maintainable, or readable. There are also variations of it such as gmake, imake, and so on, who only add their own quirks without solving the real problems.
sehugg 14y ago
I once used gmake to implement a multi-stage Mechanical Turk workflow. It was awful. The syntax sucks. But it worked consistently, and the core logic was only 110 lines of Makefile. It described the files and the data flow between them. Even now, I can read it and understand it with not too much effort. Make is a very simple functional language. It's restartable. If you type 'make -j 2' it becomes parallelizable. For almost anything for which you might write a shell script, you have to ask yourself: Why not make instead? I would like a cleaner, kinder make, but I also want it to retain make's essential make-iness. Nothing else seems to do that. I do feel bad for anyone downstream of my makefiles, though.
- tiziano88 14y ago
  actually, why not a shell script?
  - sehugg 14y ago
    Shell scripts are fine, but if your shell scripts generate files than makefiles might help you model the dependencies between them. You get parallel execution and restart ability almost for free. (and they detect errors a little better)
deleted 14y ago
[deleted]
qznc 14y ago
I like discussing build systems. However, personally I still come back to make all the time. There are certainly better alternatives (tup, shake, redo, etc), but make is simply available everywhere. It is available, if I use an OS which is a decade old. It will be available, if I use an OS in ten years from now. I am writing this on Ubuntu LTS (12.04) and none of those three advanced build systems above is available from the default repos. This means a serious dependency burden, if somebody wants to build my stuff. A few years ago, Jam was a promising build system. It seems to be dead now. Hopefully, tup and shake will stay longer. Until you have good reasons against make, it is a perfectly fine initial solution.
- Game_Ender 14y ago
  CMake is available absolutely everywhere and is used to build very large projects such as Boost, KDE, and ROS. Of course it's a C/C++ build system not a general purpose one.
g8oz 14y ago
What about Java based tools like Ant or Maven?
- viraptor 14y ago
  <description><sentence> <subject>It</subject> <verb>depends</verb> </sentence></description> Xml is an awful language to program in and every ant file grows with the software it builds => ant files that take more than 2 screens are very rarely maintainable.
- cgrubb 14y ago
  Ant doesn't conditionally re-execute a recipe when the prerequisites are newer than the target. Such logic isn't needed for Java projects since it is built into javac. Hence Ant is okay perhaps for Java projects but inefficient for others.
- Roboprog 14y ago
  Pro: they avoid starting another instance of the JVM for each "command"; they run on Windows. Con: they are grotesquely bloated for humans to read and write; (as indicated elsewhere) they make any custom file generation "tasks" a PITA. Personally, I'd like to strangle the bastard that invented Ant :-) It would have been so much nicer to have just copied make syntax, and predefined a few macros for java-specific tasks which actually spun off threads within a JVM based interpreter.
speaksense 14y ago
Man alive just use an IDE.
nickzoic 14y ago
I've used Make in a similar context: building documents. Raw simulation results (.log) -> Processed for plotting (.plot) -> Ugly Fig files (.x.fig) -> Pretty Fig files (.fig) -> EPS files (.eps) -> The final document (.pdf). By including the right dependencies in there, you can have individual figures update themselves when the raw data changes, and whole swathes of charts update themselves when the 'fixer' scripts get updated.
- philsnow 14y ago
  I used Make, inotifytools, pdflatex (or similar), "xpdf -remote", and $EDITOR to get an almost-instantly-updated view of papers and my thesis, including all figures and illustrations. When any of the dependencies of the final output changed, inotifywait noticed and kicked off a "make", and then notified the xpdf instance to refresh itself. xpdf was nice enough to try to stay on the same page when it did this.
- stevekemp 14y ago
  Me too, building a series of HTML/PDF output pages from a docbook source, for example. Make is a lovely tool that is useful for more than just compiling source code.
- mturmon 14y ago
  I've done this for a large scientific data reduction task. Each operation at the level of one month needed to be repeated several times to dial in parameters for tossing junk data. Once the per-month operations were done, all the results were combined into various plots and html pages. There were about 120 months, and running one month took several CPU hours. I put it all in a makefile. It saved a huge amount of time to not have to repeat all the data reduction (for each month) whenever the plots and subsequent analysis needed to be updated due to an underlying parameter change for a few months' data. I could run make and know that all the per-month changes would roll up correctly to the per-year and overall summaries. Also, the -j argument to make handled parallelizing the data reduction at zero cost to me.
nickzoic 14y ago
> You can approximate URL dependencies by checking the > Last-Modified header via curl -I. ... or better, use the '-z' option. counties.zip: curl -o counties.zip -z counties.zip 'http://whatever.zip'
norswap 14y ago
No you shouldn't. I know make inside and out, a result of being stubborn and lazy, and I am positive I can build a better replacement for it in a matter of days. Something like, but more general than, Rake. The way Make works is very complex with lot of implicit rules, special exceptions, etc. Debugging Makefile is a chore. If you are bent to use it, two pieces of advice: there is a handy debug flag that will tell you everything make is doing, and do disable all the implicit rules.
- quotemstr 14y ago
  > I couldn't disagree more: make is powerful and ubiquitous. Then why haven't you? > Debugging Makefile is a chore. Debugging make is a breeze. make -rd tells you exactly what make decided to do and why.
srean 14y ago
Posted a new link to Fbuild. Its a build system that has quite a different and interesting take on the build process. http://news.ycombinator.com/item?id=5276504 http://news.ycombinator.com/item?id=5276504
gimlids 14y ago
More than convincing me to use Make for dataviz, this really begs the question, "Is there a good dataflow manager for data visualization?" Something that can use URLs as dependencies, simpler syntax than Make, perhaps has a node/pipe dataflow GUI... Cascading.org comes to mind, but it is too complicated and Hadoop-oriented for this kind of dataviz.
runn1ng 14y ago
Make is fine, easy to learn and understand, but not very portable, while Autotools are scary giant monsters left from previous generations.
dllthomas 14y ago
I had a professor who would use make to manage document format conversion of his assignments and handouts.
jmount 14y ago
And don't forget how broken recursive make can in fact be: http://aegis.sourceforge.net/auug97.pdf http://aegis.sourceforge.net/auug97.pdf . I lot of builds are nasty "Heisenbuilds."
ericmoritz 14y ago
I can't tell you how many projects I've shelved for months and when I returned thought, "Damn it. How do I build this again? Oh look a Makefile. Thank you Past Eric." Even for projects that use more sophisticated build tools like rebar, leiningen or npm I write a Makefile so I don't have to remember those tools. Make provides a universal interface to those tools.
shurcooL 14y ago
This is a nice article, especially on a general level of writing scripts that automate work so you don't have to redo it manually. I want to point out one thing. targetfile: sourcefiles command This violates DRY, there is duplication between command and source files (and target file?). In theory, it should be possible to automatically deduct the source files from the command. It's far from easy in the general case, but it would save you from having to manually update source files when the command changes. (Any small inefficiency times lots of occurrences times lots of people really adds up.)
- spc476 14y ago
  I don't understand. In make, I can do: program : f1.o f2.o f3.o $(CC) -o $@ $^ f1.o : f1.c f1.h f.h $(CC) -c -o $@ $< Where "$@" is the target, "$^" is a list of the source files, and "$<" is the first source file. And that's if you want to be verbose.
zem 14y ago
all these "make for beginners" tutorials i've seen point out (correctly) how make is, at its most basic, just a dsl for specifying a dag. it seems to me that it would be pretty useful to have a tool that let you build up said dag graphically, perhaps dragging and dropping files from an explorer pane, and then generated a makefile under the hood. add in simple "infinite undo" git support that checkpointed every time you built with new inputs, and you'd have a dead simple way for non-programmers (who nonetheless have to do some programming to work with their data) to get the benefits of programming best practices. does such a tool exist?
cpressey 14y ago
As noted in many of the comments, make sucks, and the article is not promoting make per se. But make seems to be the topic of conversation, so: To me, make is a cruddy low-level declarative language that gets abused as a pseudo-imperative language, compounding the problem. Phony targets like "make <verb>" break the paradigm, because <verb> is not an artefact that can be tested for up-to-date-ness. But one advantage of make that I'm seeing underrepresented in the comments is its ubiquity. If I just want to try your project, and I need to build your project to try it, and your project is using the cool new SchnauzerBuild[1] tool, and there's no SchnauzerBuild package for my OS, I have to go install that from source... which might have its own build dependencies... ok, your project looks kind of cool but I have better things to do than this. I think it's great when build tools are able to write out a Makefile or sh[2] script that just builds everything, and when projects ship with that pregenerated. (and remember to keep it updated) [1] fictional [2] another technology that sucks but is ubiquitous
mncolinlee 14y ago
Isn't this just as true of Ant or MSBuild or Gradle (or s/.*/yourfavoritebuildtool/s)?

358 points by pie 14y ago

I agree Make is great at what it does but too many people really abuse their makefiles and don't use dependency management correctly. I suggest taking a look at http://www.amazon.com/Managing-Projects-Make-Nutshell-Handbooks/dp/0596006101/ref=sr_1_1?ie=UTF8&qid=1361732761&sr=8-1&keywords=gnu+make http://www.amazon.com/Managing-Projects-Make-Nutshell-Handbo... Really well written book. You don't have to agree with everything but its a great look at some of the better ways of using Make.

fafner 14y ago

Using make gets quickly really ugly. I wrote many Makefiles in my life and if you have to try to stick to GNU Make. At least it provides some basic functionality. But even that gets nasty quite soon when you need a simple thing like conditions with "and" or "or". If you have to write portable Makefiles ... well fsck. http://www.conifersystems.com/whitepapers/gnu-make/ http://www.conifersystems.com/whitepapers/gnu-make/ There were recent discussions about adding GNU Guile to GNU Make. I think this is a brilliant idea. Not only would a major GNU project not only use Guile. But it would make writing Makefiles much more enjoyable because you have a real programming language at hand.

eropple 14y ago
Personally, I don't want a programming language in my makefiles. Unless working in a very codified environment (Java and Maven, for example), I think it is vital to keep your build system as simple as you can. If I absolutely need nontrivial logic in my makefile, I would much, much rather be forced to call out to a script file (because it encourages not doing that). And if you insist on parking a programming language into it, it should probably be something with a reasonable amount of existing knowledge and users. I like Scheme, but unless the project was written in a Lisp I'd seriously question the judgment of anybody whacking Scheme into a build file multiple people had to work with and understand.
- __david__ 14y ago
  After using Rake (ruby with some dependency DSLish stuff) on a project for a solid year, I appreciate Make much more. Rake is too powerful and makes it too easy to keep a bunch of complex logic inside the Rakefile. Our project, left unchecked, turned the Rakefile into a pile of spaghetti. Make, on the other hand, being kind of crufty and inconvenient, encourages moving complex logic out of itself and into external scripts earlier than Rake. On top of that it encourages you to have reasonable interfaces to those scripts so that they fit back into the Makefile gracefully. So ironically, Make, by being obtuse, ends up encouraging better overall project build structure.
  - ufo 14y ago
    What sort of issue did you run into. I am wondering if its osmething that could be solved by better modularization/sandboxing or if its more fundamental problems.
    - __david__ 14y ago
      It's not a programming issue per se, it's a human issue. Rake provides a nice baseline for throwing all your quickie scripts into little functions that end up with reasonable (if basic) command line interfaces for free. Those quickie scripts inevitably grow more complicated as the project goes on (usually because the project itself gets more complicated over time) and before long you've outgrown Rake. Except that since Rake is just Ruby it tricks you into thinking you haven't outgrown it! I started noticing it when I was taking a bunch of time deciding in what order I should be putting the optional parameters to my Rake tasks such that it was most convenient to the user and spending way to much code validating those arguments and setting defaults when I realized that I could've written a script using 'optparse' that would easier to document, easier to use, and easier to write and modify. There's a graph you could make where the X axis is size of the script and the Y axis is complexity (or maybe "effort"). The Rake line, drawn on this graph, starts near 0,0 but climbs and a nice steep rate. Make starts at basically the same spot as Rake but climbs way faster. A standalone command-line script starts a bit higher on the Y axis, but is flatter over all. The point at which the Rake (or Make) and the script lines meet is where you should switch to a standalone script. With Make, this happens fairly early on when things are still relatively simple. So you convert your bash commands into a Ruby script and you end up better off in the long run. With Rake it happens so late that converting to a standalone script becomes a very large undertaking and nobody wants to do it (because it still works--why mess with it?). Over the long haul it becomes a pain point.
udp 14y ago
The stupid thing is that conditions are supported just fine in BSD make, but the syntax is slightly different than GNU make. So you're forced to write for one or the other, and GNU make is more widely used - so makefiles end up being GNU make only, even though they don't require any GNU-specific features.
rdtsc 14y ago
Any thoughts on CMake for portable Makefile generation. It seems a lot projects use that.
- fafner 14y ago
  CMake is a great example on problems with DSLs. Simply not everybody is a good language designer. CMake uses a pretty horrible language and comes with its own kinds of strangeness. I have used it quite extensively and it's probably the way to go if you want to support MSVC. But I'm not really happy with it.
zwegner 14y ago
Here's a very simple but powerful build system by the author of that paper: http://code.google.com/p/make-py/ http://code.google.com/p/make-py/ It's just a single ~400 line python script. The only major feature that's missing IMO is hash-based rebuild detection, which is essential if you're using much auto-generated code.
- ycombobreaker 14y ago
  > The only major feature that's missing IMO is hash-based rebuild detection, which is essential if you're using much auto-generated code. I'm guessing that your goal here is to avoid a recompile if the auto-generated content did not change. This is not a problem for timestamp-based build tools if you add a single step: auto-generate your output to a temp file, and only replace the target with the temp file if the file shows a difference. The program `install` will do the 'copy iff file contents have changed' part for you, so it's really just two extra lines in the Makefile recipe.
  - zwegner 14y ago
    Interesting idea, I hadn't thought of that before. This only increases the complexity of the Makefile though, and for not much reason. Hashing can help in other circumstances (such as possibly skipping a linking step if the object files don't change), and it would really be much cleaner to have it as part of the build system. The last time I played around with generating code through make, it got real ugly real fast, and I don't think your 'install' trick will help much in that regard...
    - ycombobreaker 14y ago
      > ... for not much reason There's actually a good reason to do this, but it is an edge case. A timestamp-based build system can do a no-op check with very little I/O. A hash-based build system has to read all of the file contents in order to determine that nothing has changed. Depending on the latency and bandwidth of your storage, this can make a big difference in incremental builds.

mynegation 14y ago

My problem with make for data pipelines is that a lot if decisions have to be content based instead of timestamp based. Multiple platforms is also an issue, I usually don't bother installing the whole Cygwin just for gnu make. I end up with custom python scripts which may be more verbose but more flexible and almost always cross-platform.

mbostock 14y ago
Have you tried Waf? http://code.google.com/p/waf/ http://code.google.com/p/waf/ Seems like it might be a good fit.
- ahelwer 14y ago
  How does Waf compare to SCons? I'm a big fan of SCons, although it annoyingly does not have transitive dependency resolution for libraries. Waf has this functionality, correct?
  - cschmidt 14y ago
    I haven't used Waf either, but some information is here: www.scons.org/wiki/SconsVsOtherBuildTools
  - ArchD 14y ago
    I found it hard to make scons build outside my source directory (instead of polluting it). This is important for example when you don't want random files showing up in your git clone. With waf, this is easy. Subjectively, waf code looks more Pythonic than scons, though both are supposedly Python.
  - wrl 14y ago
    Waf is also much nicer for end-users, if you're the sort who distributes source packages. It mimics the "./configure; make; make install" sequence as "./waf configure; ./waf; ./waf install".

under2x 14y ago

You should use CMake.

dkhenry 14y ago
CMake generates Makefiles and it really is only good for building software while Make can be used for any number of complex tasks. I agree that if you need a build system use CMake it will make your life easy.
- rcfox 14y ago
  > use CMake it will make your life easy At the expense of your users. Okay, I know next to nothing about CMake. All I know is that it is extremely frustrating when I get the source to a project that uses CMake and then have to modify the CFLAGS or figure out why it can't find some include file. Of course, the same applies to makefiles generated by automake. Let's just write our makefiles by hand, people.
  - dkhenry 14y ago
    Thats is the worse idea ever, hand written make files are extremly prone to failure, moreso then CMake files. There is a way to verify that you have all your dependencies and everything is discovered rather then hard coded, but some people really like to hard code things ( CMake and make both let you do this ) The thing CMake gives you is that it will _automatically_ handle almost everything involved with creating correct make files if you tell it to. Why write the same code to discover dependencies and to have external build directories and staging directories when we can do it once and have a computer write it out for us. I see it this way Make files are great and reduce you typing at the computer by a factor of 1000. CMake is also great and reduces your typing over make by a factor of 10 to 20. Things like Go or SBT/Maven are even better and they reduce typing to almost nothing at all ( SBT doesn't require any typing to build most standalone programs )
    - krakensden 14y ago
      "everything is discovered rather than hard coded" I take it you've never used or read any of the Find.cmake files, or tried to figure out which variables you need to set in all the spaghetti. Is it _LIBRARIES or *_LIBRARY_DIR? Is the comment at the top related to reality? The awful homegrown language makes it all worse.

emddudley 14y ago

This article isn't so much "Why Use Make" as it is "Use a Build System". It even says so in the sidebar.

mbostock 14y ago
Yep! I wrote about Make because it's what I use and it's the most ubiquitous. But aside from the section on syntax, the post applies to nearly any build system. (And as another poster mentioned, other build systems might offer advantages over GNU Make, such as using content hashes instead of modification times, or more elegant syntax.)

mprovost 14y ago

Drake was just released, it's a make replacement specifically for data processing and seems to have some nice features. http://blog.factual.com/introducing-drake-a-kind-of-make-for-data http://blog.factual.com/introducing-drake-a-kind-of-make-for...

Peaker 14y ago

If your project sports auto-generated code, or any sort of build targets that depend on inputs only known after generation -- Make simply cannot handle this. There are ugly workarounds but they don't work well. Make is: * Slow (e.g: when compared with tup[1]) * Very easy to get it wrong (under-specify dependencies), with cryptic bugs (or over-building) as a result. Pretty difficult to get it right (e.g: doing proper #include scanning, especially with the above-mentioned generated code). * Terrible scripting language (mentioned by many others) * Lacks useful features of other build systems (e.g: tup, or shake) of useful profile reports and queries about the build process. It is just an antiquated, poor tool that ought to be replaced by betters that already exist, whenever possible. [1]: http://gittup.org/tup/ http://gittup.org/tup/

chubot 14y ago
Have you used tup? I have encountered on the web, and it looks very well done. But I never hear about anyone using it. Is it just under-publicized? People sometimes talk about waf as an alternative. I took a look at waf awhile aback and I felt disappointed, I forget why. I think node.js switched off waf for some reason I don't recall; either that or they didn't like it.
- Peaker 14y ago
  I used tup, and it has its shortcomings. I would never prefer Make to tup though, even with its shortcomings. I am now working with Shake, and it seems to be the nicest I've used yet. It is not as robust as tup at verifying there are no under-specified dependencies, nor at detecting what needs to be rebuilt when a build script changes. But it is much more flexible (you get to write a build script with the full power of Haskell), it generates much nicer reports as a result, and it has some other interesting features that tup lacks. Unlike tup, you don't need to specify all the dependencies statically before building anything. Since tup has some useful features that Shake lacks, I don't think there's a clear winner here. But either one is far preferable to Make in every setting.
  - chubot 14y ago

jgrahamc 14y ago

If you use GNU Make it's worth using my GNU Make Standard Library: http://gmsl.sourceforge.net/ http://gmsl.sourceforge.net/ And also reading everything I wrote as Mr. Make: http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-arti... Or buy my book: http://www.lulu.com/shop/john-graham-cumming/gnu-make-unleashed/paperback/product-2937580.html;jsessionid=853955F416C443C6FCCCF7E7700BF62E http://www.lulu.com/shop/john-graham-cumming/gnu-make-unleas...

gatherknwldg 14y ago
gmsl is a wonderful, well considered library. I had to buy the book after finding it. Thanks!
eliben 14y ago
I'll have to respectfully disagree. IMHO, the utility of DSLs like Make is high when used very sparingly, for simple flows like the one the linked article describes. The more complex your Makefile, the less you should be using Make. If you need to write complete programs in Make, just stop bending over backwards and pick a real programming language. FWIW most non-trivial projects nowadays that use Make don't write makefiles directly but use make generators like CMake and Gyp.
- malkia 14y ago
  Make is the assembly/C language of build system. Lots of other systems just generate makefile compatible files (qmake, cmake, premake, others), but often these fall into overly complex traps trying to explain everything both with simple unique words, and then clashing into these. Make is very optimal at the stage it is, everything above or below it it's just not that optimal.
  - eliben 14y ago
    As the assembly language of build systems, I agree that Make works well. But there, IMHO, its utility ends.

kostya-kow 14y ago

This is funny to read, especially after spending several hours trying to get a project to compile with make, while tracking down an annoying and cryptic error that doesn't even make sense. Make is a horrible tool, and is extremely hard to learn. I think it should only be used when generated by other build systems.

largesse 14y ago

Created in 1977, Make has its quirks. But whether you prefer GNU Make or a more recent alternative, consider the benefits of capturing your workflow in a machine-readable format. Like a programming language?

stephth 14y ago
Exactly what I was thinking. Use a scripting language you're proficient at. This article doesn't make Make look worth the trouble if you don't have a more specific reason to use it. Too much complexity, and quirks like you’ll need to delete the previously-downloaded zip file before running make can easily be avoided with a scripting language. These days any relatively convoluted task that I may need to do more than once, I cook it in Ruby (and sprinkle some AppleScript/appscript if it involves UI). Languages like Ruby or Python offer a good balance of flexibility and abstraction for this kind of automation.
- kragen 14y ago
  If you're using a series of shell commands, each of which takes seconds to hours to run, you will be happy with make's ability to (1) run shell commands easily and (2) not run them again when they don't need to be run. You could write a dependency tracking thingy in any programming language, but at that point you might as well be using Rake or some other make replacement.
NickPollard 14y ago
Yes. Make is just a DSL for file-based dependency management. You could certainly write makefiles in another language, but chances are it would be a lot more verbose and probably less portable. Of course there are other make tools apart from Make, so feel free to use whatever you feel is best suited.
triplepoint217 14y ago
Agreed, I generate documents with graphs and calculations in them for my research, but I tend to document and automate them with python scripts rather than makefiles, because python does not make my brain hurt.

ttty 14y ago

can't I use js with Node.js instead?

ricardobeat 14y ago
Cakefiles are a nice CoffeeScript based alternative.
nailer 14y ago
Pardon? I think you might be missing a project name eg, foo.js

cgrubb 14y ago

Most systems I use have GNU make 3.81 installed. GNU make 3.81 added the special target .ONESHELL which makes code like the following possible. I'm not sure if it is a good idea, but it does remove the dependency that make has on shell programming. .ONESHELL: SHELL = /usr/bin/python VAR := lorem ipsum all: @ import re n = 3 print('make variable: $(VAR)') print('local variable n: {}'.format(n)) rx = re.compile('\d+') s = 'foo 17' m = rx.search(s) if m: print('has number: {}'.format(s)) else: print('no number: {}'.format(s))

subleq 14y ago
.ONESHELL was actually added in 3.82: http://cvs.savannah.gnu.org/viewvc/make/NEWS?view=annotate&root=make#l153 http://cvs.savannah.gnu.org/viewvc/make/NEWS?view=annotate&#...

coldskull 14y ago

scons is a good alternative to make. Make is ok for small projects, but it gets ugly pretty quick...and it allows you to make mistakes easily...

mseebach 14y ago

Absolutely do not use make for any new project. If you love make, it's a big, red, burning flag that you're not demanding enough of your tools and that you're not keeping up with changes in your ecosystem. There are many, many way better alternatives to make. Which one is better depends on the platform you're on. The majority of them throws in automatic dependency management for free. Yes, I know that the essence of the post is "use a build system". I agree completely. In fact, script everything. Then script your scripts. Then refactor your scripts because they are getting messy. But don't give impressionable souls the idea that "make" is anywhere near an acceptable (generalised, default) choice today.

falcolas 14y ago
Lots of "Don't do this" without suggesting alternatives doesn't do anyone much good. If you're on a *nix system, building C/C++ programs (or automating one-off builds like the example), what would you recommend in make's place?
- lmm 14y ago
  CMake
  - craftkiller 14y ago
    Personally I'd suggest Premake because it uses Lua instead of rolling its own scripting language. The world would be a better place if we could all agree on one scripting language for stuff like this so no one has to look up the syntax for things like creating arrays for every individual tool. Feature-wise premake isn't entirely caught up to CMake but it has everything important. Also the premake files look a lot cleaner and more readable compared to cmake files.
    - leif 14y ago
      http://xkcd.com/927 http://xkcd.com/927
    - quotemstr 14y ago

wildgift 14y ago

To everyone with alternatives to make: Make is easy to learn. It simplifies a slightly complex task. Other build tools have a steeper learning curve. If they're more complex than the problem being solved, people won't want to adopt them. When things get complex, there's the GNU make manual and libraries. When you have to target multiple platforms, there's autotools, which is really complex and intimidating, but less complex than targeting multiple platforms. Alternatives to make seem to fit in between make and autotools. Make sucks. I've also heard that Unix sucks, and C sucks, too. Despite this alleged suckyness, these things not only persist, but accumulate improvements over the years.

narzac 14y ago
Definitely agree, unfortunately people always know better, even when they don't have any clue. My philosophy is "If the development environment provide you a modern, native build system(ex: go build, get), go with it, otherwise stick to make unless using autotools etc. will give you a reasonable advantage."

minsight 14y ago

This is a sloppily phrased call for automation. I can wholeheartedly endorse automation as a goal, but the idea that make can be your go-to tool for this is embarrassing. Make will get you into trouble the moment your task becomes non-trivial. Everyone has their favourite tools and no single tool is best for every job (although grep and find are often your friend). As I often say in job interviews, I'm lazy. I protect the lazy guy inside of me, because he's the one who cries out "Didn't we do this manually before? Shouldn't we automate it?" He's a good guy to have around.

jgrahamc 14y ago

Three years ago I wrote a blog post arguing in favour of Make : http://blog.jgc.org/2010/11/things-make-got-right-and-how-to-make.html http://blog.jgc.org/2010/11/things-make-got-right-and-how-to... I'm not sure I still agree with everything in that post, but Make is very terse and expressive which I like. One of the biggest problems with Make is that it's a macro language and some people don't get along well with them.

jpollock 14y ago

Everyone points at make and says, "use something better". They point at Rake, they point at Maven. However, Make has one feature that I really need, that I rely on to speed up builds: parallel dependency construction. The last time I checked, neither Maven nor Rake do this properly. Maven runs everything inside a single VM. Who wants to have to worry about multithreading in their unit tests? Rake requires thread support in the underlying Ruby interpreter (why?). I've got 64+ hardware threads (Sparc T4-1) all sitting there waiting to be spun up and do my bidding. Please help me to convert electricity to heat!

ArchD 14y ago
Waf can also do parallel build with proper dependencies, and it's Python-based. Have you looked at it? The documentation is rather clear, but the initial learning curve is probably steeper than make.
- jpollock 14y ago
  Thanks for the pointer, I'll go check it out.
catenate 14y ago
Inferno's <{} construct dispatches processes in parallel, and gathers output in coherent blocks. I use it in credo to parallelize all the dependencies for a target. I rely on mutual exclusion to avoid building a target twice at the same time, and on cached checksums to avoid rebuilding when there have been no changes. See the sh-inferno scripts map, credeper, and cresum in the project http://github.com/catenate/credo http://github.com/catenate/credo

fnordfnordfnord 14y ago

>>(Note: use tabs rather than spaces to indent the commands in your makefile. Otherwise Make will crash with a cryptic error.) Sheesh.

csense 14y ago
Requiring tabs alone is reason enough to ditch make.
- nickzoic 14y ago
  ... or use an editor with syntax highlighting for Makefiles.

jcromartie 14y ago

Everybody seems to be missing the fact that he's not talking about building a big software project. He's talking about scripting a simple workflow around a few files. Make is probably better than anything else for this, because under these circumstances it's so simple it's hard to get it wrong. Something like Rake is obviously better when you need to really program your builds.

jackalope 14y ago
I depend on make for recording a workflow history of my projects. There's no point in creating an alias in my ~/.bashrc for an arcane command I'm only going to use once or twice a year to transform, sync or configure a group of files. When I check a project out of version control, it's nice to have a Makefile there to remind me how I accomplished something. Make allows you to consistently specify actions in a universal way. Typing 'make edit' will open the main file in vim for me in any project. Typing 'make update' will check for available updates on any platform using the appropriate package manager, and 'make upgrade' will download and install them. Typing 'make sync' will transfer my project files to the specified $SERVER as the specified $USER with the appropriate protocol. Typing 'make install' will copy an updated configuration file and restart all of the daemons necessary for it to work. It's a great timesaver. Even when I need to write shell scripts for something too elaborate for make, I tend to create a target that runs the script, instead of trying to remember its name and all of the options.
pjungwir 14y ago
I wanted to say exactly this. I wouldn't choose Make to build a software project these days (probably), but I use it all the time for exactly what the author describes: recording/managing a data processing workflow. For stats/ML exploratory work, it's invaluable, because everything you do is recorded in various targets. It makes my one-off scripts not quite so one-off, so when I revisit the code six months later, I can quickly figure out what I was doing. It also means less typing, because every command is just `make foo`, and that delegates out to R/Ruby/Python scripts, shell pipelines, or whatever I need. I'm delighted that someone else is (ab)using make the same way!

RexRollman 14y ago

From what the example accomplishes, it seems to be that it would have been easier to create a shell script. Is there something I am missing?

pyxy 14y ago
I think shell script can work as well as makefile in most cases. The pro of shell script is not having one more tool for running your commands. You have a shell, why do you need make for describing build steps? For the moment I see a makefile as a shell script that fails when a command fails. But you can use `-e` option with sh, bash, rc shells and expect the same behaviour. The only thing shell misses (comparing to makefile) is dependencies check, you have to write dependencies mechanism yourself if you need it. Also I don't like the .PHONY stuff, I just can't get it, it feels alien. Am I missing anything obvious?
- RexRollman 14y ago
  It doesn't sound like it to me. Thanks for the reply.
throwawayG9 14y ago
I just started learning about Make today, but according to the other comments in this post, there are lots of differences, like incremental builds and parallelism. It also checks things like file modification dates and whatnot.

nohuck13 14y ago

I've started using make for production data generation (not building software) instead of pure Python. Mostly to tie together Python scripts doing the real work. Pro -dependency management for free -well-known paradigm makes it easy for someone other than me to figure out where to look if something went wrong. -scripts I call out to can be focused Cons -syntax -for processes that don't generate an output (think adding data to a file in place) I wind up creating placeholder files ("file.transformA.done"). I actually want my dependency management to be terse and declarative, which is the opposite of what I'm looking for in a programming language, so it feels like a pretty natural divide.

nathell 14y ago

Make is an excellent expert system / topological sorting tool with an abominable syntax.

ChuckMcM 14y ago

I find this a really interesting conversation. On the one hand I've built some really really complicated systems with make (like all of SunOS) and some even more complicated systems with a custom build tool (the google base infrastructure packages) and there are pluses and minuses to both approaches. If you're building something small, its hard to beat a simple make file. Its easy to write, it allows you to capture dependencies and refactor quickly, and it doesn't interrupt your coding flow. If you're building something quite large there are some real productivity benefits from building knowledge of what you are building into the build system. And computational build systems (which is to say build systems where the build spec file includes the capability to do local computation) can make retargeting the same build to different environments easier.

pedrocr 14y ago

Like everyone seems to be mentioning this post is really about "you should have automatic builds" and make is just an incidental. redo[1] is another interesting make replacement that keeps most of the good points of make (e.g., simple and shell based), while having a more powerful dependency mechanism. It's even compatible with make itself allowing you to move a part of a recursively built project to redo, while having it call into make for other subcomponents. What I've done before when doing data-driven blog posts[2][3] was to write the whole analysis end-to-end in ruby, including branching out to R for stats and graphing[4]. I ended up using rake (ruby's make-like tool) to tie everything together with dependency tracking but I could just as well have written a script that would call everything in order. Doing that gives you a way to quickly reproduce results (what's discussed here) but also, together with version management, a way to go back to previous versions and use things like "git bisect" to figure out when you introduced a bug. [1] https://github.com/apenwarr/redo/ https://github.com/apenwarr/redo/ [2] http://pedrocr.pt/text/how-much-gnu-in-gnu-linux http://pedrocr.pt/text/how-much-gnu-in-gnu-linux [3] http://pedrocr.pt/text/preliminary-results-open-source-evolution http://pedrocr.pt/text/preliminary-results-open-source-evolu... [4] https://github.com/pedrocr/codecomp https://github.com/pedrocr/codecomp

mcartyem 14y ago

Inevitably, any build system becomes part of the software. With make this means you are now using a far more inferior language to build software.

eliben 14y ago

I think what you were looking for here is actually a shell script, rather than a Makefile. A shell script is much more suitable for general purpose programming. Or better yet, use Python ;-) And, let's admit it, for small projects dependency analysis is overrated (although it can be implemented easily enough in a real programming language).

apw 14y ago
One thing make allows you to do is incremental runs. For example, if the first step in your shell script costs 1/2 hour, you really don't want to run the shell script over and over again when you are debugging the next step. Yes, you can implement your own caching logic in your shell script, but its usually much better to use a tool specifically designed for that job.

robot 14y ago

"Makefiles are machine-readable documentation ... " Yes it is machine readable but how about us humans? Makefiles are ugly and when working on someone else's project it is very hard to reverse engineer the build system. Build systems are themselves software projects and we need better tools to develop and maintain them. SCons was a promising project at one point, it improved things by capturing the build system in Python classes. I thought things would be more maintainable and readable. However for me it wasn't a well design, it obscures build system development with mixed declarative and iterative programming. Makefiles are the defacto standard today, but they're no where near beautiful, or maintainable, or readable. There are also variations of it such as gmake, imake, and so on, who only add their own quirks without solving the real problems.

sehugg 14y ago

I once used gmake to implement a multi-stage Mechanical Turk workflow. It was awful. The syntax sucks. But it worked consistently, and the core logic was only 110 lines of Makefile. It described the files and the data flow between them. Even now, I can read it and understand it with not too much effort. Make is a very simple functional language. It's restartable. If you type 'make -j 2' it becomes parallelizable. For almost anything for which you might write a shell script, you have to ask yourself: Why not make instead? I would like a cleaner, kinder make, but I also want it to retain make's essential make-iness. Nothing else seems to do that. I do feel bad for anyone downstream of my makefiles, though.

tiziano88 14y ago
actually, why not a shell script?
- sehugg 14y ago
  Shell scripts are fine, but if your shell scripts generate files than makefiles might help you model the dependencies between them. You get parallel execution and restart ability almost for free. (and they detect errors a little better)

deleted 14y ago

[deleted]

qznc 14y ago

I like discussing build systems. However, personally I still come back to make all the time. There are certainly better alternatives (tup, shake, redo, etc), but make is simply available everywhere. It is available, if I use an OS which is a decade old. It will be available, if I use an OS in ten years from now. I am writing this on Ubuntu LTS (12.04) and none of those three advanced build systems above is available from the default repos. This means a serious dependency burden, if somebody wants to build my stuff. A few years ago, Jam was a promising build system. It seems to be dead now. Hopefully, tup and shake will stay longer. Until you have good reasons against make, it is a perfectly fine initial solution.

Game_Ender 14y ago
CMake is available absolutely everywhere and is used to build very large projects such as Boost, KDE, and ROS. Of course it's a C/C++ build system not a general purpose one.

g8oz 14y ago

What about Java based tools like Ant or Maven?

viraptor 14y ago
<description><sentence> <subject>It</subject> <verb>depends</verb> </sentence></description> Xml is an awful language to program in and every ant file grows with the software it builds => ant files that take more than 2 screens are very rarely maintainable.
cgrubb 14y ago
Ant doesn't conditionally re-execute a recipe when the prerequisites are newer than the target. Such logic isn't needed for Java projects since it is built into javac. Hence Ant is okay perhaps for Java projects but inefficient for others.
Roboprog 14y ago
Pro: they avoid starting another instance of the JVM for each "command"; they run on Windows. Con: they are grotesquely bloated for humans to read and write; (as indicated elsewhere) they make any custom file generation "tasks" a PITA. Personally, I'd like to strangle the bastard that invented Ant :-) It would have been so much nicer to have just copied make syntax, and predefined a few macros for java-specific tasks which actually spun off threads within a JVM based interpreter.

speaksense 14y ago

Man alive just use an IDE.

nickzoic 14y ago

I've used Make in a similar context: building documents. Raw simulation results (.log) -> Processed for plotting (.plot) -> Ugly Fig files (.x.fig) -> Pretty Fig files (.fig) -> EPS files (.eps) -> The final document (.pdf). By including the right dependencies in there, you can have individual figures update themselves when the raw data changes, and whole swathes of charts update themselves when the 'fixer' scripts get updated.

philsnow 14y ago
I used Make, inotifytools, pdflatex (or similar), "xpdf -remote", and $EDITOR to get an almost-instantly-updated view of papers and my thesis, including all figures and illustrations. When any of the dependencies of the final output changed, inotifywait noticed and kicked off a "make", and then notified the xpdf instance to refresh itself. xpdf was nice enough to try to stay on the same page when it did this.
stevekemp 14y ago
Me too, building a series of HTML/PDF output pages from a docbook source, for example. Make is a lovely tool that is useful for more than just compiling source code.
mturmon 14y ago
I've done this for a large scientific data reduction task. Each operation at the level of one month needed to be repeated several times to dial in parameters for tossing junk data. Once the per-month operations were done, all the results were combined into various plots and html pages. There were about 120 months, and running one month took several CPU hours. I put it all in a makefile. It saved a huge amount of time to not have to repeat all the data reduction (for each month) whenever the plots and subsequent analysis needed to be updated due to an underlying parameter change for a few months' data. I could run make and know that all the per-month changes would roll up correctly to the per-year and overall summaries. Also, the -j argument to make handled parallelizing the data reduction at zero cost to me.

nickzoic 14y ago

> You can approximate URL dependencies by checking the > Last-Modified header via curl -I. ... or better, use the '-z' option. counties.zip: curl -o counties.zip -z counties.zip 'http://whatever.zip'

norswap 14y ago

No you shouldn't. I know make inside and out, a result of being stubborn and lazy, and I am positive I can build a better replacement for it in a matter of days. Something like, but more general than, Rake. The way Make works is very complex with lot of implicit rules, special exceptions, etc. Debugging Makefile is a chore. If you are bent to use it, two pieces of advice: there is a handy debug flag that will tell you everything make is doing, and do disable all the implicit rules.

quotemstr 14y ago
> I couldn't disagree more: make is powerful and ubiquitous. Then why haven't you? > Debugging Makefile is a chore. Debugging make is a breeze. make -rd tells you exactly what make decided to do and why.

srean 14y ago

Posted a new link to Fbuild. Its a build system that has quite a different and interesting take on the build process. http://news.ycombinator.com/item?id=5276504 http://news.ycombinator.com/item?id=5276504

gimlids 14y ago

More than convincing me to use Make for dataviz, this really begs the question, "Is there a good dataflow manager for data visualization?" Something that can use URLs as dependencies, simpler syntax than Make, perhaps has a node/pipe dataflow GUI... Cascading.org comes to mind, but it is too complicated and Hadoop-oriented for this kind of dataviz.

runn1ng 14y ago

Make is fine, easy to learn and understand, but not very portable, while Autotools are scary giant monsters left from previous generations.

dllthomas 14y ago

I had a professor who would use make to manage document format conversion of his assignments and handouts.

jmount 14y ago

And don't forget how broken recursive make can in fact be: http://aegis.sourceforge.net/auug97.pdf http://aegis.sourceforge.net/auug97.pdf . I lot of builds are nasty "Heisenbuilds."

ericmoritz 14y ago

I can't tell you how many projects I've shelved for months and when I returned thought, "Damn it. How do I build this again? Oh look a Makefile. Thank you Past Eric." Even for projects that use more sophisticated build tools like rebar, leiningen or npm I write a Makefile so I don't have to remember those tools. Make provides a universal interface to those tools.

shurcooL 14y ago

This is a nice article, especially on a general level of writing scripts that automate work so you don't have to redo it manually. I want to point out one thing. targetfile: sourcefiles command This violates DRY, there is duplication between command and source files (and target file?). In theory, it should be possible to automatically deduct the source files from the command. It's far from easy in the general case, but it would save you from having to manually update source files when the command changes. (Any small inefficiency times lots of occurrences times lots of people really adds up.)

spc476 14y ago
I don't understand. In make, I can do: program : f1.o f2.o f3.o $(CC) -o $@ $^ f1.o : f1.c f1.h f.h $(CC) -c -o $@ $< Where "$@" is the target, "$^" is a list of the source files, and "$<" is the first source file. And that's if you want to be verbose.

zem 14y ago

all these "make for beginners" tutorials i've seen point out (correctly) how make is, at its most basic, just a dsl for specifying a dag. it seems to me that it would be pretty useful to have a tool that let you build up said dag graphically, perhaps dragging and dropping files from an explorer pane, and then generated a makefile under the hood. add in simple "infinite undo" git support that checkpointed every time you built with new inputs, and you'd have a dead simple way for non-programmers (who nonetheless have to do some programming to work with their data) to get the benefits of programming best practices. does such a tool exist?

cpressey 14y ago

As noted in many of the comments, make sucks, and the article is not promoting make per se. But make seems to be the topic of conversation, so: To me, make is a cruddy low-level declarative language that gets abused as a pseudo-imperative language, compounding the problem. Phony targets like "make <verb>" break the paradigm, because <verb> is not an artefact that can be tested for up-to-date-ness. But one advantage of make that I'm seeing underrepresented in the comments is its ubiquity. If I just want to try your project, and I need to build your project to try it, and your project is using the cool new SchnauzerBuild[1] tool, and there's no SchnauzerBuild package for my OS, I have to go install that from source... which might have its own build dependencies... ok, your project looks kind of cool but I have better things to do than this. I think it's great when build tools are able to write out a Makefile or sh[2] script that just builds everything, and when projects ship with that pregenerated. (and remember to keep it updated) [1] fictional [2] another technology that sucks but is ubiquitous

mncolinlee 14y ago

Isn't this just as true of Ant or MSBuild or Gradle (or s/.*/yourfavoritebuildtool/s)?