17 ms·

The Unreasonable Effectiveness of Makefiles

256 points by srijan4 4y ago

thinkingkong 4y ago
I like make. But these days to me the best part about it is that it’s a common entry point. Most popular languages come with their own make-esque tools that provide the same experience to developers and systems. Tying together multiple projects, source from different locations, etc Id probably use make or a script.
Mister_Snuggles 4y ago
My most horrible abuse of `make` was to write a batch job runner. Most of the targets in the Makefile had a command to kick off the job and wait for it to finish (this was accomplished with a Python script since kicking off a job involved telling another application to run the job) followed by a `touch $@` so that make would know which jobs it had successfully run. If a process had dependencies these were declared as you'd expect. The other targets in the Makefile lashed those together into groups of processes, all the way up to individual days and times. So "monday-9pm" might run "daily-batch", "daily-batch" would have "daily-batch-part-1" (etc), and each "daily-batch-part-..." would list individual jobs. It was awful. It still is awful because it works so well that there's been no need to replace it. I keep having dreams of replacing it, but like they say there's nothing more permanent than a temporary solution. All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order.
- scott_s 4y ago
  Congrats! You invented Apache Airflow: https://airflow.apache.org https://airflow.apache.org
  - MonkeyMalarky 4y ago
    With the added bonus of not having to learn or maintain Apache Airflow!
  - Mister_Snuggles 4y ago
    Interesting: > Airflow was started in October 2014 by Maxime Beauchemin at Airbnb. It was open source from the very first commit and officially brought under the Airbnb GitHub and announced in June 2015. I believe I starting building my tool somewhere around 2010, possibly 2011. The core mechanism has been completely unchanged in that time. If Airflow was a thing at the time, I'd have hopefully looked into it. I looked at a handful of similar products and didn't find anything that was a good fit. Based on a really quick skim of the Airflow docs it seems like it checks all of the boxes. Off the top of my head: * LocalExecutor (with some degree of parallelism, assuming the dependencies are all declared properly) seems to do exactly what I want. * I could write an Operator to handle the interaction with the system where the processes actually run. The existing Python script that does this interaction can probably get me 90% of the way there. Due to the nature of what I'm running, any job scheduler will have to tell the target system to do a thing then poll it to wait for the thing to be done. To do this without any custom code, I could just use BashOperator to call my existing script. * It's written in Python, so the barrier to entry (for me) is fairly low. * Converting the existing Makefile to an Airflow DAG is likely something that can be done automatically. We deliberately keep the Makefile very consistent, so a conversion program can take advantage of that. I think my dream of replacing this might have new life!
    - zomglings 4y ago
      But... why would you want to spend energy replacing something that has been running stable for over a decade?
      - Mister_Snuggles 4y ago
        There are a number of deficiencies with the current system that aren't showstoppers, but are pain points nonetheless. Off the top of my head: * There's no reasonable way to do cross-batch dependencies (e.g., if process X in batch A fails, don't run process Y in batch B). I've got a few ideas on how I could add this in, but nothing has been implemented yet. * There's no easy way to visualize what's going on. Airflow has a Gantt view that looks very useful for this purpose, our business users would absolutely LOVE the task duration graph, and the visualization of the DAG looks really helpful too. * Continuing a failed batch is a very manual process. None of these are showstoppers because, as you said, this has been running fine for over a decade. These are mostly quality-of-life improvements.
  - rhacker 4y ago
    But airflow is an abomination... that I am forced to use at my current job.
    - ncmncm 4y ago
      It says "Apache" right there on the tin.
    - leaflets2 4y ago
      What don't you like about it?
- deleted 4y ago
  [deleted]
- cerved 4y ago
  That's beautiful, not awful
- josteink 4y ago
  > All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order. Sometimes the most interesting thing is not the story itself, but the story behind the story. This has my interest peaked. Is there anywhere else I can read about this?
  - pjot 4y ago
    Just a friendly tip that it’s _piqued_ :)
    - shawabawa3 4y ago
      You never know, maybe it is the peak of their interest and it's all downhill from here
    - ncmncm 4y ago
      Which, by the way, means "sharpened". Probably "peak" has the same root.
      - adhesive_wombat 4y ago
        pique comes from (Vulgar) Latin piccare, which means "prick with a sword". The route to English is via French. peak comes from Old English pīc, meaning just "peak" (e.g. of a mountain). The two are completely different words that just sound similar. pique may possibly go back to Proto-Germanic, and peak does, but the two go back to two separate words (*pīkaz, *pikkāre) though both are then related to sharp things and possibly onomatopoeic.
  - deleted 4y ago
    [deleted]
  - mturmon 4y ago
    This is just the “job server” which you can read about in the Gnu make documentation, which is excellent. https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... Basically you can specify a “-j 24” (e.g.) option to make, and it will run as many as 24 build steps in parallel. If your Makefile is correct, that’s all you need. Because make knows the dependency graph, it can correctly handle cases where some build steps have to be done serially, while others can be fully parallelized. E.g., a: b; b: c; In which builds of b and c are serial, versus x: y; x: z; for which builds of y and z are parallel. It’s quite a powerful capability and it feels great to see 24 build-steps start in parallel from one simple command.
- ithkuil 4y ago
  My most horrible abuse of make was a distributed CI where I put a wrapper in the MAKE env var so that recursive make executions would invoke my wrapper which would enqueue jobs for remote workers to pick up
- jbboehr 4y ago
  > All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order. Any sufficiently complicated init system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of systemd.
  - felixgallo 4y ago
    including, most notably, systemd.
- enriquto 4y ago
  > My most horrible abuse of `make` was to (...) Heh. The text that follows this sentence is likely the most beautiful and elegant use of a Makefile ever. I love the humble bragging of this site.
- fmajid 4y ago
  I did the same. Parallelization thanks to GNU make's `-j`, recoverability (only rerun the failed steps, not from scratch). If you use the remake fork of GNU make, you also get debugging and profiling for free.
klodolph 4y ago
Make is fantastic at what it does well. These days, I would absolutely not use Make to compile code written in C, except for the smallest personal projects. It is just too fussy to construct the Makefile correctly, in a way that you can get correct incremental builds. Nearly any other build system is better at building projects written in C, in the sense that it is easy & straightforward to get your build system to do correct incremental builds.
- EddySchauHai 4y ago
  Absolutely! Basically every company I've worked with over the last couple years as a contractor followed this methodology and it's grown to be my default runner as it's language agnostic.
- properparity 4y ago
  > incremental builds I've found that to not matter that much these days - unless your projects is hundreds of thousands (or maybe millions even) of LOC, full builds are instant on modern machines.
  - klodolph 4y ago
    That's definitely not true for: - Most C++ or Rust projects - Medium size or larger C projects - Anything which is built with tools written in JavaScript (due to Node startup time overhead) Stuff where a full build is close enough to instant: - Most Java, C#, Go projects - Small C projects - Tiny/trivial C++ or Rust programs, or C++ programs written for embedded systems This is just my experience. YMMV.
    - stjohnswarts 4y ago
      any sizeable rust or c++ project is gonna take a while to compile in my experience? I tend to break things off into libraries that are faster to compile. Some of my coworkers don't like it but they learn to deal (c++)
      - 4y ago
bjourne 4y ago
The short article conflates popularity with quality. Windows 3.11 became the most sold os in history despite being utter trash. Make is popular because it was the first build system, not because it is not utter trash.
- BiteCode_dev 4y ago
  Just got out of a python training session with one of my student running w11. Can confirm. So many problems. Seems like one version out of 2 of windows being trouble stills stand.
- LtWorf 4y ago
  What's a good replacement for make that has feature parity?
  - kimmeld 4y ago
    Well, many of make’s features is what makes it trash.
    - LtWorf 4y ago
      So your solution is to burn the computer and yell at clouds?
  - bjourne 4y ago
    I like WAF. Scons is good too. Then there is Ninja and the whole ecosystem around that. Search the HN archives for many more suggestions.
throwaway787544 4y ago
Declarative isn't important. I wish people would stop talking about it. It's not even useful to think about most of the time because of how many ways it can be interpreted. It's a thereotical categorization, not a functional design principle. Just make the program do useful things for the user. Make the computer work for the human rather than the other way around.
- samatman 4y ago
  This is almost too devoid of content (opinion is not that) to usefully reply, and I suspect you're gesturing at some more specific point I might agree with, or at least understand. Declaration is absolutely a coherent and functional design principle. A declarative system is one in which the outcome is specified and the process is not. This has big payoffs in domains where it's natural. A good example being grammars. It also has hazards, a good example being the performance of algorithms to parse grammars. We can see where a declarative build system might be a mixed bag, because the process itself is imperative: make is an early attempt to reconcile imperative build processes with a declaration of what circumstances require their triggering. Basically every build system since make has improved on make, but they all do more-or-less what make does. The design, in short, is proven, as well as declarative but not purely so. And the ability to reason about the degree to which make is declarative shows that 'declarative' is in fact a coherent idea. But you can't declare software into existence, you must compile it.
- nimih 4y ago
  I dunno, the "declarative-ness" of `make` is a pretty important component of its usefulness. In particular, the property of `make`, wherein the structure/details of the computation is implicit in the provided configuration and invoked command rather than being explicitly written down somewhere, is central to its utility, since the alternative of just writing a shell script is anecdotally a much less popular option. If you want to propose a more appropriate word to describe such a property which is less buzzwordy, feel free, but in the context of "why does `make` have such enduring popularity", I think the article's author is being quite reasonable in bringing it up.
  - 4y ago
Nexialist 4y ago
Slightly tangential but I've worked for several companies now that use `make` as a simple command runner, and I have to say it's been a boon. Being able to drop into any repo at work and expect that `make init`, `make test` and `make start` will by convention always work no matter what the underlying language or technology is, has saved me a lot of time.
- ReadTheLicense 4y ago
  This is standard in Node.js ecosystem and I love it. Each package has scripts in package.json that you can run with npm run [name], and some of these like start, test or build (and more) are standardized. It's really great DX.
  - patrickthebold 4y ago
    But it's npm, so when you switch to a java project, for example, you have different commands.
    - r3trohack3r 4y ago
      Quite a few companies I've contracted with have lifted the pattern up into Bazel or Gnu Make - for node projects `make lint` can be a pass through. In the project repo, either work.
- txutxu 4y ago
  Conventions are great, but that doesn't look like anything specific to make, a shell wrapper could do that: #!/bin/sh case $1 in init) ... do whatever for each project init ;; start) ... do whatever for each project start ;; test) ... do whatever for each project tests ;; *) echo "Usage: $0 init|start|test" >&2 exit 1 ;; esac In my home/personal projects I use a similar convention (clean, deploy, update, start, stop, test...), I call those little sh scripts in the root of the repo "runme". The advantage could be, maybe, no need to install make if not present, and no need to learn make stuff if you don't know it. Sometimes they don't match the usual words (deploy, start, stop, etc) but then I know that if I don't remember them, I just type ./runme and get the help. For my scenario, it's perfect because of it's simplicity.
qbasic_forever 4y ago
For a task runner I really like just and its Justfile format: https://github.com/casey/just https://github.com/casey/just It is heavily inspired by make but doesn't focus on the DAG stuff (but does support tasks and dependencies). Crucially it has a much better user experience for listing and documenting tasks--just comment your tasks and it will build a nice list of them in the CLI. It also supports passing CLI parameters to task invocations so you can build simple CLI tools with it too (no need to clutter your repo with little one-off CLI tools written in a myriad of different languages). If most of your make usage is a bunch of .PHONY nonsense and tricks to make it so developers can run a simple command to get going, check out just. You will find it's not difficult to immediately switch over to its task format.
- kbd 4y ago
  Just seems neat, but except for the dependencies it's just a way to package multiple shell scripts into one file, no? I've thought about trying it out a few times but can never see its value over scripts in ./bin.
  - qbasic_forever 4y ago
    Scripts in bin have no documentation, no easy way to enumerate them, etc. There is definitely a time and a place for bin scripts, especially as things grow in complexity. However the beauty of just is that there's one file (the justfile) that defines all of your project's actions. You don't have to go spelunking into bin to figure out how to tweak a compiler flag, etc. And since just will run anything there's no reason why your complex bin scripts can't just be called from a simple one liner task in a justfile. Could your write a bash script that does stuff like enumerate all the bin scripts, pull out documentation comments, etc.? Absolutely, and people have followed that pattern for a while (see https://github.com/qrush/sub https://github.com/qrush/sub) but it's a bunch of boilerplate to copy between projects. Just pulls out that logic into a simpler config file.
- gurgeous 4y ago
  Seconded - I love just & Justfile. Such an upgrade after trying to force things into package.json scripts. Chaining commands, optional CLI arguments, comments, simple variables, etc. Very simple and a breath of fresh air.
JackFr 4y ago
Never loved make. First used it in the early nineties and found the syntax obscure and error messages cryptic. My response to this article would be, if make is so great why did they have to invent 'configure' and 'xmkmf'? And why do people continue to create new build tools every couple of years? Yeah, I mean I guess it worked, but unreasonably effective? Hardly.
- jhallenworld 4y ago
  Eh, they solve different problems. Make is too simple to customize your build to deal with system differences- it just builds your code.
- bch 4y ago
  > … why do people continue to create new build tools every couple of years? Seems like a rite[0] of passage to some degree. Perhaps similar to people talking a stab at The Next Actually Correct CMS, and The Next Object System That Doesn’t Suck, or The Next Linux Distro For Smart People. [0] edit: corrected “right V. rite” per https://news.ycombinator.com/item?id=32442473 https://news.ycombinator.com/item?id=32442473
  - erik_seaberg 4y ago
    (btw, it’s https://en.wikipedia.org/wiki/Rite_of_passage https://en.wikipedia.org/wiki/Rite_of_passage)
- falcolas 4y ago
  > why did they have to invent 'configure' Cross-architecture and linux distro compatibility, mostly.
  - compiler-guy 4y ago
    Err, pedantically, configure was not for cross Linux distro compatibility, but for cross unix compatibility. It existed long before Linux was a sparkle in Linus’s eye. And even then, it handled even some non unix environments as well.
PaulKeeble 4y ago
Its unfortunate that other build systems haven't taken over. Make is terrible for incremental builds and its reliance on binaries often means issues getting it to run and being very platform dependent. It is better than using a bat or shell file for the same purpose but its a long way behind many of the other language specific tools. I am surprised something better hasn't become popular, Make is the CVS of the build tools.
- dwheeler 4y ago
  I find it works well for incremental builds, that is what it's for.
BiteCode_dev 4y ago
Make dsl is terrible, it's the yaml of build systems. If you are a python dev, give doit a try.
- Koshkin 4y ago
  What is so "terrible" about it? I think the rule syntax is as simple as it gets.
  - vitiral 4y ago
    Required tabs, lines are each their own "script" instead of blocks (allowing variables), not allowing other executors (i.e. python, TCL, etc would be better than sh).
    - MRtecno98 4y ago
      Actually you can tell GNU Make not to create a new shell for every command using .ONESHELL[0] and you can also select a different shell executable to run said commands[1] [0]: https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... [1]: https://www.gnu.org/software/make/manual/html_node/Choosing-the-Shell.html https://www.gnu.org/software/make/manual/html_node/Choosing-...
    - philkrylov 4y ago
      Check (GNU Make) .RECIPEPREFIX, (GNU Make) .ONESHELL, (POSIX Make) SHELL=
    - blacksqr 4y ago
      Tclmake, a partial Make clone in pure Tcl: https://github.com/TclLab/tclmake https://github.com/TclLab/tclmake
- nottorp 4y ago
  Interesting that everyone who recommends an alternative to Make picks... something different. I doubt there will be two people recommending the same thing in the whole discussion.
jhallenworld 4y ago
To go along with Make, use this technique to automatically generate and include dependency information: https://scottmcpeak.com/autodepend/autodepend.html https://scottmcpeak.com/autodepend/autodepend.html And this, avoid recursive make: https://accu.org/journals/overload/14/71/miller_2004/ https://accu.org/journals/overload/14/71/miller_2004/
keepquestioning 4y ago
Now we need to banish CMake
- Koshkin 4y ago
  Yeah, there is something deeply wrong about the whole thing... But what do we replace it with? (Out of desperation, perhaps, I even dreamt of a build system based on a C library, with the usage being, say, tcc -run mybuild.c :)
- mdaniel 4y ago
  People keep saying that, but of the systems I've encountered that use CMake they've worked the most predictably, as opposed to autotools, meson, scons, bazel, or heavenforbid ./build.sh *WRITING* CMake is the 11th circle of hell, but CLion is gradually getting more support for it because they use(d?) it as the first-class project definition when it launched I would pay good money for CMake 4.x to switch to skylark/starlark
est31 4y ago
Make is solving many complicated tasks, like keeping in mind when to re-run some target (there is communication going on with the compilers that provide make .d files so that they know which source files influence a binary), or running job servers managing parallelism that also support nested sub-makes. But it also has many ugly warts. It's hard to design something that solves both those tasks as well as make does, and also as generalist as make does. Often they are solving a subset, what currently itches the main developer. But something that is both as general, and as comprehensive as make, those tools are rare. Ninja for example checks many of the boxes, but lacks make jobserver support.
- bXVsbGVy 4y ago
  Ninja looks clean because it is new. Give it some decades and it is likely inherit a few of warts make has.
  - aappleby 4y ago
    It's not that new, and in practice there aren't very many ways for it to get warty because it's basically a dependency graph with "run this command line if this node is dirty" attached to the edges. I like it very much.
  - LtWorf 4y ago
    Ninja is so cool… until you can't compile stuff because the author had a different ninja version than what's on your distribution…
falcolas 4y ago
I love Make. It's a terrible tool for quite a few things, but it's awesome at the thing I use it most for - abstracting away complex series of shell commands behind one or two words. It's like shell aliases that can follow a repo anywhere. make test make format make clean make docker-stack Fantastically useful stuff, even if all it's doing is calling language specific build systems in the background.
dijit 4y ago
A company I worked for used make to execute docker, which felt somewhat odd. Correct me if I'm wrong, but I always assumed `make` to be one of those build tools that could incrementally build targets based on dependencies. The arcane and esoteric language that constitutes `make` is almost universally avoided, surely if people need a simple task runner there could be better options?
- falcolas 4y ago
  Better? Absolutely. More ubiquitous? Not really. I use it for executing docker as well, because the docker command is actually about 5 commands with long lists of parameters. But it's just `make docker-stack` for everyone now.
  - LtWorf 4y ago
    But then the docker command is starting docker and running the same makefile again inside the container, which becomes hellish to reason about. It's probably better to just make a separate script for that instead.
    - falcolas 4y ago
      That is odd. Within the docker container, I agree, you probably shouldn't do that. Esp. since Dockerfiles themselves allow you to run a list of shell commands as your entrypoint.
jcoq 4y ago
This is a remarkably stupid comment but not everything is "unreasonably effective". Mathematics was noted as unreasonably effective for modeling the universe because most areas of mathematics were not invented for the applications they meet... like discovering that your coffee maker doubles as an exceptionally good waffle maker. Makefiles on the other hand, are not unreasonably effective in this sense. Makefiles are, in fact, _reasonably_ effective... like a coffee maker that brews coffee well.
- deleted 4y ago
  [deleted]
bXVsbGVy 4y ago
I'm trying to understand why so many people seems to hate make. I hate building system that don't use Makefile, or that use but don't respect the variable convention. It makes really quite annoying to do things like changing allocation library, add compilers flags, etc.
- TylerE 4y ago
  Because gnu autotools blows goats. Oh, and the tab thing.
  - gbrown_ 4y ago
    make != autoconf
    - TylerE 4y ago
      Yes, but 99% of the makefiles you encounter in the wild comes from autotools.
      - dima55 4y ago
        This isn't at all true, in my experience. If it's true for you, please consider that your issues are about autotools and not Make, and direct your complaints in that direction.
      - stjohnswarts 4y ago
        the vast majority of the ones I encounter are from cmake...
      - nrclark 4y ago
        I use Make for all kinds of general-purpose build automation tasks, and as kind of a top-level catch all. No Autotools to be found in them (except maybe if called from some recipe).
  - tpoacher 4y ago
    surely it blows gnus rather than goats?
jwilk 4y ago
It's 500 Internal Server Error for me. Archived copy: https://web.archive.org/web/20220812135641/https://matt-rickard.com/the-unreasonable-effectiveness-of-makefiles https://web.archive.org/web/20220812135641/https://matt-rick...
topspin 4y ago
My most elaborate use of make was ten years ago to build a bootable, reproducible embedded system image including the OS, application, media files and cryptographic signatures. It's still in active use and no one is complaining.
hyperupcall 4y ago
Honestly, I only find Makefiles useful when I have a tiny C/C++ project and need stuff just to compile quickly and easily without the overhead of a real build system. For literally everything else, I found myself using it more as a task runner - and Make doesn't do a great job at it. You end up mixing Bash and Make variables, string interpolation, and it becomes really messy, really fast. Not to mention the footguns associated with Make. I found bake (https://github.com/hyperupcall/bake https://github.com/hyperupcall/bake) to suit my needs (disclaimer: I wrote it). It's literally just a Bash script with all the boilerplate taken care of you - what a task runner is meant to be imo
- nrclark 4y ago
  Does Bake support dependency expression / parallel execution? Those are killer features that keep me on Make for general-purpose automation.
h4l 4y ago
I've found these guidelines for Makefiles make for a pretty good experience using make: https://tech.davis-hansson.com/p/make/ https://tech.davis-hansson.com/p/make/ The advice on output sentinel files for rules creating multiple files helps keep rebuilding dependencies reliable. Avoiding most of the cryptic make variables also helps Makefiles to remain easily understandable when you're not frequently working on them. And using .ONESHELL to allow multi-line statements (e.g. loops, conditional etc) is great. No need to contort things into one line. or escape line breaks. Seems like you could even use a more serious programming language instead of sh/bash by setting SHELL to Python or similar. That may be a road to madness though...
- zelphirkalt 4y ago
  The problem with .ONESHELL is, that it is for the whole file. I so wish it was per target. That would be really useful. But for the whole file? Maybe I need each line to be a separate shell anywhere in the file and that will make it impossible to use .ONESHELL for the entire file.
  - h4l 4y ago
    It is per target, that's how it works when I've used it. For example: # Makefile SHELL := bash .ONESHELL: .RECIPEPREFIX = > thing1: > FOO=bar > echo "$${FOO:?}" .PHONY: thing1 thing2: > echo "$${FOO:?}" .PHONY: thing2 Results in: $ make thing1 thing2 FOO=bar echo "${FOO:?}" bar echo "${FOO:?}" bash: line 1: FOO: parameter null or not set Note how the bash error for unset FOO is line 1 for the second target. Edit: Maybe I misinterpreted, do you mean you'd want to choose whether a given target is ONESHELL or not?
    - zelphirkalt 4y ago
      > [...] do you mean you'd want to choose whether a given target is ONESHELL or not? Yep, exactly! I would like to have some targets use ONESHELL and others in the same Makefile not use ONESHELL. So that I can choose the most appropriate for each target. So far I have managed by avoiding ONESHELL and doing the typical "backslash, next line continues" thingy. But it puts some people off.
philsnow 4y ago
> File-watcher or live-reloading. You can create a build/deploy loop fairly easily When I worked with latex more, I kept a ~/.Makefile-latex and a shell function that would pretty much just do inotifywait -e modify -e move --recursive --monitor . | while read; do make --makefile ~/.Makefile-latex; done and I kept emacs and xpdf in side-by-side windows. Whenever I'd save a file in emacs (or xfig or whatever), a second later xpdf would refresh the pdf (and stay on the same page). It took away some of the pain of working with latex. edit: I used this complicated setup instead of LyX or whatever other "(la)tex IDE" because I had ancillary files like xfig diagrams that would get compiled to .eps files and gnuplot scripts that would render data into graphs, and the makefile knew how to generate everything.
- kwantam 4y ago
  Very nice! It turns out that latexmk has this functionality: latexmk -pvc -pdf foo.tex (It can be configured to HUP your pdf reader if needed, too.) I usually add something like this command as the `auto` target in my latex Makefiles, which works pretty nicely.
  - philsnow 4y ago
    TIL about latexmk (which I gather is a perl script that handles a lot of the latex build deps, like running bibtex at the right time(s), for you automatically), thanks! I never really understood most of the TeX world. I cargo-culted the bits I needed to make: * articles for journal submissions (easy, they all provided packages or templates), * my thesis (somebody had written a latex class that took care of all the school's formatting requirements a ~decade prior), and * a resume, and never dug into the differences between tex, latex, miktex, texlive, auctex, lyx, pdftex, xetex, luatex etc etc etc etc etc. I was just to ask you if you knew of a 40000-foot overview of these kinds of things but stumbled across this article over at Overleaf: https://www.overleaf.com/learn/latex/Articles/What%27s_in_a_Name%3A_A_Guide_to_the_Many_Flavours_of_TeX https://www.overleaf.com/learn/latex/Articles/What%27s_in_a_... , it looks pretty apropos.
- earthscienceman 4y ago
  This is funny, because I just wrote a (fish) shell script that does this as well because all of the tex IDEs are so painful. Mostly because the entire efficiency of latex is that you're editing text and can do it in a text editor like emacs and move things around very quickly. I don't want a new interface! But. I'm kind of proud. My shell script monitors the tex files for character changes and then once a (configurable) threshold of changes is met, it kicks off the compilation process. But the real game changer is that every time it compiles, if the compile is successful it commits the recent edits to a git branch. Then if I want, I can go through the git branch and see the entire history of edits for only versions of the document that compiled. It's a game changer in a big way. When I finish a section, I squash the minor edits into one commit and give it a good message and then commit the full thing to the main branch. Then there is where I can make sure my manuscripts look the way they should and do major revisions or collaborative edits. The icing on the cake is that the fish script monitors for a special keypress and will take actions. So I hit "c" to compile the tex, "b" to rerun the massive bibliography compilations, "s" to squash the commits into the main branch, and "l" to display log errors. It's a dream! Now I don't think about compilation at all, or when I should commit something minor to git (and fiddle with the commands). I just type away and watch/request the pdf refresh when I need it... and _actually_ get work done. My god. So happy. I just finished this today.
belkarx 4y ago
There was a post on HN (I believe) a couple of months ago concerning a research paper that explores "What defines a makefile" concluding that by their definition, Excel can be used as one. I've looked around the internet and can't find it, so if anyone else remembers it and has it upvoted or otherwise has the link, posting it would add to this conversation. It had interesting discussion of graphs and requirements and was overall a worthwhile read.
- yaris 4y ago
  Was it “Build Systems a la Carte” paper? I could not find mentions of it on HN after 2020 though.
  - belkarx 4y ago
    Yes, thank you! Here's the PDF in case anyone else is interested: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems-5ab0f42d0f937.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... I can't find any actual conversation about it on HN so I won't post the HN link
- AceJohnny2 4y ago
  I'm pretty sure I found that seminal paper, "Build Systems a la Carte", via HN, but indeed can't find a highly-commented story on the topic. Perhaps the most upvoted story was this one about a blog post from one of the authors (itself very interesting): https://news.ycombinator.com/item?id=17494016 https://news.ycombinator.com/item?id=17494016 It is a very important paper, I think, mapping out the landscape of build systems designs. But I'm still waiting for something novel to emerge from it. I know the authors planned on integrating the improvements they found into (Cloud) Shake, but I don't know if that's production-ready. Perhaps Bazel is the best we can hope for at this time.
ogogmad 4y ago
Is there a Python library that can check a timestamp and update some files according to a lambda? I don't want to learn Make syntax again.
- Jtsummers 4y ago
  if os.stat(object).st_mtime < os.stat(dependency).st_mtime: ... Use a dict to contain each rule: rules["a.c"] = (["b.c", "c.c", "b.h", "c.h"], action) ... That can be simplified as well and then run a simple parser over it that takes a simpler representation and turns it into that dict. Then: def make(rules, rule): (deps, action) = rules[rule] run_rule = len(deps) == 0 # if there are no dependencies, the rule always runs for dep in deps: make(rules, dep) if os.stat(rule).st_mtime < os.stat(dep).st_mtime: run_rule = True if run_rule: action() Of course, this doesn't actually validate the DAG itself. ---------- An amendment: You'll probably want to pass both the rule name and the dependencies into the action function/lambda/object so that you can parameterize it and maybe reuse the action (like a common compiler command): if run_rule: action(rule, deps)
  - andreareina 4y ago
    Topological sorting is in the stdlib: https://docs.python.org/3/library/graphlib.html https://docs.python.org/3/library/graphlib.html
- TillE 4y ago
  SCons can be used as a simple Make-style tool, in addition to the more complex stuff it supports.
shepherdjerred 4y ago
I'd encourage anyone thinking of using make to look at alternatives. Make is great, but is quickly becomes a ball of duct-tape. Make works very well when you spend the time to express your dependency tree, but realistically that never happens and people tend to add hacks upon hacks for Makefiles. Not only that, but they don't scale well as your project adds more components, such as integration testing, documentation, etc. I found Earthly[0] to be a great replacement. Everything runs in Docker. Your builds are reproducible, cache-able, and parallelizable by default. I've heard Dagger[1] is another good tool in the space. [0]: https://earthly.dev/ https://earthly.dev/ [1]: https://dagger.io/ https://dagger.io/
davidpfarrell 4y ago
As these types of post often come around to make's sub-optimal use as general runner, I'd like to point out my project, Run: https://github.com/TekWizely/run https://github.com/TekWizely/run It feels like a makefile but is optimized managing and invoking small tasks and wrappers, and auto-generates help text from comments.
olliej 4y ago
I find a bunch of the decisions in Make to not be great - but it's all just syntax stuff. At its core all a Makefile is is a dependency graph, which is necessary in all the more advanced config managers, only those others are much more heavyweight than the vast majority of projects ever need. Most code doesn't need to vary arguments or parameters based on the target environment, and so that complexity is unneeded, in which case a Makefile can do it all.
eichin 4y ago
Aww, noone mentioned that `debian/rules` files are almost always makefiles? (To the point of starting with `#!/usr/bin/make -f` ...)
_8j50 4y ago
Autogen/autotools mess however... not so pleasant when it breaks. Curious for those who published source with autotools,etc... do you really sit down and figure that out. It's just a mystery that works or doesn't to me. I never have or would go beyond a simple make file. It really seems tedious on top of the actual code you write. A bit impressive tbh.
- jbboehr 4y ago
  Yes, I used autotools[0]. It's definitely hairier than plain make, but you get a lot of useful features on top of it. There's thousands of examples all over the internet so it's easy to reference them. I like the elegance of pure make, and do use it when appropriate, but I wouldn't really want to reimplement the things autotools does myself in it. [0]: https://github.com/jbboehr/handlebars.c/blob/master/configure.ac https://github.com/jbboehr/handlebars.c/blob/master/configur...
- LtWorf 4y ago
  What's the alternative? I don't know in advance if the system's seek() supports 64 bits directly or just 32. Or if the user wants a more minimal binary without a certain feature.
  - UncleEntity 4y ago
    Cmake? Once I figured out how to get it to build python extension modules my life became a lot simpler. The python way (distools?) is easy enough until it isn’t then I reach for cmake.
    - LtWorf 4y ago
      cmake is not as powerful as autotools.
shadowgovt 4y ago
Does make still basically fail to handle filenames with spaces in them? That was the deal-breaker for me last I checked.
LAC-Tech 4y ago
I wanted to learn makefiles to generate a static website. Quickly ran into some massive limitations - one of which is that it completely broke apart when filenames had spaces in them. "But why would you do that you're doing it wrong" - don't care wanted spaces in filenames. Ended up switching Rake (a make like DSL written in ruby) and never looked back. Not only can you do all the make declarative stuff, but you get the full power of ruby to mess around with strings.
- rightbyte 4y ago
  Next thing you ask for leading tabs in filenames ... And the feature bloat slippery slope is sliding towards Bazel!
deleted 4y ago
[deleted]
kazinator 4y ago
Anyone who finds make unreasonably effective must be working with GNU Make. If I had to use some barely POSIX conforming thing from BSD thing or wherever, I'd instead write a write a top-to-bottom linear shell script full of conditionals.
- makapuf 4y ago
  In that case,a good habit is to name those GNUMakefile. Works the same, but announce gnu make (which is the one worth it)
  - kazinator 4y ago
    GNU Make is so prevalent that nobody seems to bother with this. I can't recall the last time I've seen a GNUmakefile. (Note the lower case 'm', according to the manual.) It makes sense if you ship both a GNUmakefile and a Makefile.
    - dwheeler 4y ago
      I agree. I try to stick to just POSIX make when that's easy to do, but POSIX make is so impoverished that it usually isn't worth trying to stick to just POSIX make. In general, for most people, make is GNU make. And GNU make is actually pretty decent at a lot of tasks. Many ridiculous makefile problems I see stem from not using it well. I suggest: * Use compiler dependency generators (generate .d files and "include" them via make). This eliminates many lines and errors. * Don't use recursive makefiles. * Set macros with := or ::= * Use substitutions so definitions can be changed in just one place and the rest automatically works. There is no perfect tool for all circumstances. But GNU make is still useful for many circumstances.
      - kazinator 4y ago
        Here is an odd thing: GNU Make looks for the file in this order (according to the manual): GNUmakefile, makefile, Makefile. The last time I saw a lower case 'm' makefile was eons ago. The manual itself recommends Makefile rather than makefile, right in the same paragraphs where it tells you makefile is checked first. Why look for makefile before Makefile, a file next to nobody has. Maybe lower-case makefile was a predecessor to GNUmakefile; a way of having a GNU-specific makefile. (If so, that's a noteworthy thing the manual ought to mention.)
aappleby 4y ago
Recently I've been much more pleased by the "Unreasonable Effectiveness of Ninja Plus A Few Trivial Scripts". A raw Ninja file is verbose but easy to read and understand, and there is basically no magic happening behind the scenes (well, except for "deps = gcc", but that's minor). Any action that's too complicated to go directly in the Ninja file goes to a "rule command \ command = ${command}" that just runs a python or shell script. The only thing I'd like to add to my setup would be "Ninja with glob support", but again that can be handled in a few lines of Python that spit out another chunk of .ninja rules so it's not really a blocker.
evilotto 4y ago
The 2 biggest things I find missing in makefiles is the ability to have the "modified date" of a target be something other than the mtime of the target file, and for the dependencies of a target to be generated rather than statically specified. The latter there are workarounds for, but not the former. The author's suggestion of "better support for make docker" is completely the wrong thing to do - you don't need to support docker, you need to support mtimes and dependencies in a way that docker could use.
acdw 4y ago
my website was a gnu makefile for a minute : https://gitlab.com/acdw/acdw.net https://gitlab.com/acdw/acdw.net
kwertyoowiyop 4y ago
Please, no more “unreasonable effectiveness” headlines.
k0k0r0 4y ago
Pretty much discussion about make as a task runner, and maybe the following is just a very specific use case: make test I combined learning make with learning TDD, or rather develope some individual style of TDD. I am astonished how effective make is in a rapidly evolving code base with not insificant amounts of unit, component and integration tests. That said I try hard to use the makefiles both, heavily recursive and highly structured. Of course, I am learning, so my set up is far from optimal. But wow.... Makefiles are so powerful for testing. Its amazing. On the other hand, that the first time I am properly using a build system.
kelseyfrog 4y ago
The fact that it's likely to be pre-installed on any given linux machine is a huge plus. I use make for providing a uniform interface for my team's data science project workflow and while it has a lot of downsides, it also has a upsides too. It's probably the choice I've struggled the most to justify, but I have a harder time justifying the alternatives.
dwheeler 4y ago
I once used GNU make to manage a large data pipeline on an 18-person project and it worked well. We developed a lot of Python scripts. To manage them I created some helper tools to integrate them via make. People are welcome to reuse it, I got it released it as open source software. I named it make-booster: https://github.com/david-a-wheeler/make-booster https://github.com/david-a-wheeler/make-booster From its readme: "This project (contained in this directory and below) provides utility routines intended to greatly simplify data processing (particularly a data pipeline) using GNU make. It includes some mechanisms specifically to help Python, as well as general-purpose mechanisms that can be useful in any system. In particular, it helps reliably reproduce results, and it automatically determines what needs to run and runs only that (producing a significant speedup in most cases)." "For example, imagine that Python file BBB.py says include CC, and file CC.py reads from file F.txt (and CC.py declares its INPUTS= as described below). Now if you modify file F.txt or CC.py, any rule that runs BBB.py will automatically be re-run in the correct order when you use make, even if you didn't directly edit BBB.py." This is NOT functionality directly provided by Python, and the overhead with >1000 files was 0.07seconds which we could live with :-). Make provides a way to handle dependencies as DAGs. Using it at scale requires that you call or write mechanisms to provide that DAG dependency info, but any such tool needs that info. Some compilers already come with generators for make, and in those cases it's especially convenient to use make.
- mturmon 4y ago
  I really like using make for data pipelines as you suggest, and thanks for pointing out your package. In this pipeline use case, you have base data, and a series of transformations that massage it into usable results. You are always revising the pipeline, usually at the output end (but not always) so you want to skip as many preprocessing steps as possible. Make automates all that. This works great for image processing pipelines, science data pipelines, and physical simulators for a few examples. There have been a few blog posts and ensuing HN discussions about this use pattern for make. The discussion generally gets mixed up between make’s use as a build system for code, alas.
  - dwheeler 4y ago
tomohawk 4y ago
Even more powerful: rake
quickthrower2 4y ago
I was wondering, if you don’t build a single timestamped artefact (i.e. everything is phony) is there any reason to use makefile over a set of bash scripts?
bandrami 4y ago
Years and years ago (before the init wars) I made an init system that was just a makefile run by a statically compiled make binary. While this was fun it outlined what I dislike about declarative run control systems (you don't see what's actually going on without having it be pointlessly verbose). But as far as that went it ran my silly micro distro pretty well.
Pinus 4y ago
I skimmed the comments quite quickly so I may have missed it, but it seems that nobody has mentioned what I think is one of the biggest issues with make (as actually used, as opposed to some ideal): recursive runs. I have seen several packages (and I used to work with one rather large code base) which were subdivided into multiple directories, sometimes in several levels, each with its own Makefile and some rule that did for d in $(SUBDIRS); do cd $d && $(MAKE); done A top-level make run would take ages just to recurse through everything to decide that nothing needed to be done, and dependency tracking did not work across modules, so the only way of making sure you got everything rebuilt was to ‘touch’ everything. (The advantage of the setup was that you could check out just part of the tree and build it separately.) It would be really nice if make could read all subdirectory Makefiles, build one global dependency graph and then have at it.
fmajid 4y ago
Like democracy, make is the worst build system to the exception of every other other. Like Maven. What demented mind thought XML was a good idea compared to the simple and clean DSL of Make (or Ninja)?

256 points by srijan4 4y ago

I like make. But these days to me the best part about it is that it’s a common entry point. Most popular languages come with their own make-esque tools that provide the same experience to developers and systems. Tying together multiple projects, source from different locations, etc Id probably use make or a script.

Mister_Snuggles 4y ago

My most horrible abuse of `make` was to write a batch job runner. Most of the targets in the Makefile had a command to kick off the job and wait for it to finish (this was accomplished with a Python script since kicking off a job involved telling another application to run the job) followed by a `touch $@` so that make would know which jobs it had successfully run. If a process had dependencies these were declared as you'd expect. The other targets in the Makefile lashed those together into groups of processes, all the way up to individual days and times. So "monday-9pm" might run "daily-batch", "daily-batch" would have "daily-batch-part-1" (etc), and each "daily-batch-part-..." would list individual jobs. It was awful. It still is awful because it works so well that there's been no need to replace it. I keep having dreams of replacing it, but like they say there's nothing more permanent than a temporary solution. All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order.

scott_s 4y ago
Congrats! You invented Apache Airflow: https://airflow.apache.org https://airflow.apache.org
- MonkeyMalarky 4y ago
  With the added bonus of not having to learn or maintain Apache Airflow!
- Mister_Snuggles 4y ago
  Interesting: > Airflow was started in October 2014 by Maxime Beauchemin at Airbnb. It was open source from the very first commit and officially brought under the Airbnb GitHub and announced in June 2015. I believe I starting building my tool somewhere around 2010, possibly 2011. The core mechanism has been completely unchanged in that time. If Airflow was a thing at the time, I'd have hopefully looked into it. I looked at a handful of similar products and didn't find anything that was a good fit. Based on a really quick skim of the Airflow docs it seems like it checks all of the boxes. Off the top of my head: * LocalExecutor (with some degree of parallelism, assuming the dependencies are all declared properly) seems to do exactly what I want. * I could write an Operator to handle the interaction with the system where the processes actually run. The existing Python script that does this interaction can probably get me 90% of the way there. Due to the nature of what I'm running, any job scheduler will have to tell the target system to do a thing then poll it to wait for the thing to be done. To do this without any custom code, I could just use BashOperator to call my existing script. * It's written in Python, so the barrier to entry (for me) is fairly low. * Converting the existing Makefile to an Airflow DAG is likely something that can be done automatically. We deliberately keep the Makefile very consistent, so a conversion program can take advantage of that. I think my dream of replacing this might have new life!
  - zomglings 4y ago
    But... why would you want to spend energy replacing something that has been running stable for over a decade?
    - Mister_Snuggles 4y ago
      There are a number of deficiencies with the current system that aren't showstoppers, but are pain points nonetheless. Off the top of my head: * There's no reasonable way to do cross-batch dependencies (e.g., if process X in batch A fails, don't run process Y in batch B). I've got a few ideas on how I could add this in, but nothing has been implemented yet. * There's no easy way to visualize what's going on. Airflow has a Gantt view that looks very useful for this purpose, our business users would absolutely LOVE the task duration graph, and the visualization of the DAG looks really helpful too. * Continuing a failed batch is a very manual process. None of these are showstoppers because, as you said, this has been running fine for over a decade. These are mostly quality-of-life improvements.
- rhacker 4y ago
  But airflow is an abomination... that I am forced to use at my current job.
  - ncmncm 4y ago
    It says "Apache" right there on the tin.
  - leaflets2 4y ago
    What don't you like about it?
deleted 4y ago
[deleted]
cerved 4y ago
That's beautiful, not awful
josteink 4y ago
> All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order. Sometimes the most interesting thing is not the story itself, but the story behind the story. This has my interest peaked. Is there anywhere else I can read about this?
- pjot 4y ago
  Just a friendly tip that it’s _piqued_ :)
  - shawabawa3 4y ago
    You never know, maybe it is the peak of their interest and it's all downhill from here
  - ncmncm 4y ago
    Which, by the way, means "sharpened". Probably "peak" has the same root.
    - adhesive_wombat 4y ago
      pique comes from (Vulgar) Latin piccare, which means "prick with a sword". The route to English is via French. peak comes from Old English pīc, meaning just "peak" (e.g. of a mountain). The two are completely different words that just sound similar. pique may possibly go back to Proto-Germanic, and peak does, but the two go back to two separate words (*pīkaz, *pikkāre) though both are then related to sharp things and possibly onomatopoeic.
- deleted 4y ago
  [deleted]
- mturmon 4y ago
  This is just the “job server” which you can read about in the Gnu make documentation, which is excellent. https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... Basically you can specify a “-j 24” (e.g.) option to make, and it will run as many as 24 build steps in parallel. If your Makefile is correct, that’s all you need. Because make knows the dependency graph, it can correctly handle cases where some build steps have to be done serially, while others can be fully parallelized. E.g., a: b; b: c; In which builds of b and c are serial, versus x: y; x: z; for which builds of y and z are parallel. It’s quite a powerful capability and it feels great to see 24 build-steps start in parallel from one simple command.
ithkuil 4y ago
My most horrible abuse of make was a distributed CI where I put a wrapper in the MAKE env var so that recursive make executions would invoke my wrapper which would enqueue jobs for remote workers to pick up
jbboehr 4y ago
> All of this was inspired by someone who replaced the rc scripts in their init system with a Makefile in order to allow processes to start in parallel while keeping the dependencies in the right order. Any sufficiently complicated init system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of systemd.
- felixgallo 4y ago
  including, most notably, systemd.
enriquto 4y ago
> My most horrible abuse of `make` was to (...) Heh. The text that follows this sentence is likely the most beautiful and elegant use of a Makefile ever. I love the humble bragging of this site.
fmajid 4y ago
I did the same. Parallelization thanks to GNU make's `-j`, recoverability (only rerun the failed steps, not from scratch). If you use the remake fork of GNU make, you also get debugging and profiling for free.

klodolph 4y ago

Make is fantastic at what it does well. These days, I would absolutely not use Make to compile code written in C, except for the smallest personal projects. It is just too fussy to construct the Makefile correctly, in a way that you can get correct incremental builds. Nearly any other build system is better at building projects written in C, in the sense that it is easy & straightforward to get your build system to do correct incremental builds.

EddySchauHai 4y ago
Absolutely! Basically every company I've worked with over the last couple years as a contractor followed this methodology and it's grown to be my default runner as it's language agnostic.
properparity 4y ago
> incremental builds I've found that to not matter that much these days - unless your projects is hundreds of thousands (or maybe millions even) of LOC, full builds are instant on modern machines.
- klodolph 4y ago
  That's definitely not true for: - Most C++ or Rust projects - Medium size or larger C projects - Anything which is built with tools written in JavaScript (due to Node startup time overhead) Stuff where a full build is close enough to instant: - Most Java, C#, Go projects - Small C projects - Tiny/trivial C++ or Rust programs, or C++ programs written for embedded systems This is just my experience. YMMV.
  - stjohnswarts 4y ago
    any sizeable rust or c++ project is gonna take a while to compile in my experience? I tend to break things off into libraries that are faster to compile. Some of my coworkers don't like it but they learn to deal (c++)
    - 4y ago

bjourne 4y ago

The short article conflates popularity with quality. Windows 3.11 became the most sold os in history despite being utter trash. Make is popular because it was the first build system, not because it is not utter trash.

BiteCode_dev 4y ago
Just got out of a python training session with one of my student running w11. Can confirm. So many problems. Seems like one version out of 2 of windows being trouble stills stand.
LtWorf 4y ago
What's a good replacement for make that has feature parity?
- kimmeld 4y ago
  Well, many of make’s features is what makes it trash.
  - LtWorf 4y ago
    So your solution is to burn the computer and yell at clouds?
- bjourne 4y ago
  I like WAF. Scons is good too. Then there is Ninja and the whole ecosystem around that. Search the HN archives for many more suggestions.

throwaway787544 4y ago

Declarative isn't important. I wish people would stop talking about it. It's not even useful to think about most of the time because of how many ways it can be interpreted. It's a thereotical categorization, not a functional design principle. Just make the program do useful things for the user. Make the computer work for the human rather than the other way around.

samatman 4y ago
This is almost too devoid of content (opinion is not that) to usefully reply, and I suspect you're gesturing at some more specific point I might agree with, or at least understand. Declaration is absolutely a coherent and functional design principle. A declarative system is one in which the outcome is specified and the process is not. This has big payoffs in domains where it's natural. A good example being grammars. It also has hazards, a good example being the performance of algorithms to parse grammars. We can see where a declarative build system might be a mixed bag, because the process itself is imperative: make is an early attempt to reconcile imperative build processes with a declaration of what circumstances require their triggering. Basically every build system since make has improved on make, but they all do more-or-less what make does. The design, in short, is proven, as well as declarative but not purely so. And the ability to reason about the degree to which make is declarative shows that 'declarative' is in fact a coherent idea. But you can't declare software into existence, you must compile it.
nimih 4y ago
I dunno, the "declarative-ness" of `make` is a pretty important component of its usefulness. In particular, the property of `make`, wherein the structure/details of the computation is implicit in the provided configuration and invoked command rather than being explicitly written down somewhere, is central to its utility, since the alternative of just writing a shell script is anecdotally a much less popular option. If you want to propose a more appropriate word to describe such a property which is less buzzwordy, feel free, but in the context of "why does `make` have such enduring popularity", I think the article's author is being quite reasonable in bringing it up.
- 4y ago

Nexialist 4y ago

Slightly tangential but I've worked for several companies now that use `make` as a simple command runner, and I have to say it's been a boon. Being able to drop into any repo at work and expect that `make init`, `make test` and `make start` will by convention always work no matter what the underlying language or technology is, has saved me a lot of time.

ReadTheLicense 4y ago
This is standard in Node.js ecosystem and I love it. Each package has scripts in package.json that you can run with npm run [name], and some of these like start, test or build (and more) are standardized. It's really great DX.
- patrickthebold 4y ago
  But it's npm, so when you switch to a java project, for example, you have different commands.
  - r3trohack3r 4y ago
    Quite a few companies I've contracted with have lifted the pattern up into Bazel or Gnu Make - for node projects `make lint` can be a pass through. In the project repo, either work.
txutxu 4y ago
Conventions are great, but that doesn't look like anything specific to make, a shell wrapper could do that: #!/bin/sh case $1 in init) ... do whatever for each project init ;; start) ... do whatever for each project start ;; test) ... do whatever for each project tests ;; *) echo "Usage: $0 init|start|test" >&2 exit 1 ;; esac In my home/personal projects I use a similar convention (clean, deploy, update, start, stop, test...), I call those little sh scripts in the root of the repo "runme". The advantage could be, maybe, no need to install make if not present, and no need to learn make stuff if you don't know it. Sometimes they don't match the usual words (deploy, start, stop, etc) but then I know that if I don't remember them, I just type ./runme and get the help. For my scenario, it's perfect because of it's simplicity.

qbasic_forever 4y ago

For a task runner I really like just and its Justfile format: https://github.com/casey/just https://github.com/casey/just It is heavily inspired by make but doesn't focus on the DAG stuff (but does support tasks and dependencies). Crucially it has a much better user experience for listing and documenting tasks--just comment your tasks and it will build a nice list of them in the CLI. It also supports passing CLI parameters to task invocations so you can build simple CLI tools with it too (no need to clutter your repo with little one-off CLI tools written in a myriad of different languages). If most of your make usage is a bunch of .PHONY nonsense and tricks to make it so developers can run a simple command to get going, check out just. You will find it's not difficult to immediately switch over to its task format.

kbd 4y ago
Just seems neat, but except for the dependencies it's just a way to package multiple shell scripts into one file, no? I've thought about trying it out a few times but can never see its value over scripts in ./bin.
- qbasic_forever 4y ago
  Scripts in bin have no documentation, no easy way to enumerate them, etc. There is definitely a time and a place for bin scripts, especially as things grow in complexity. However the beauty of just is that there's one file (the justfile) that defines all of your project's actions. You don't have to go spelunking into bin to figure out how to tweak a compiler flag, etc. And since just will run anything there's no reason why your complex bin scripts can't just be called from a simple one liner task in a justfile. Could your write a bash script that does stuff like enumerate all the bin scripts, pull out documentation comments, etc.? Absolutely, and people have followed that pattern for a while (see https://github.com/qrush/sub https://github.com/qrush/sub) but it's a bunch of boilerplate to copy between projects. Just pulls out that logic into a simpler config file.
gurgeous 4y ago
Seconded - I love just & Justfile. Such an upgrade after trying to force things into package.json scripts. Chaining commands, optional CLI arguments, comments, simple variables, etc. Very simple and a breath of fresh air.

JackFr 4y ago

Never loved make. First used it in the early nineties and found the syntax obscure and error messages cryptic. My response to this article would be, if make is so great why did they have to invent 'configure' and 'xmkmf'? And why do people continue to create new build tools every couple of years? Yeah, I mean I guess it worked, but unreasonably effective? Hardly.

jhallenworld 4y ago
Eh, they solve different problems. Make is too simple to customize your build to deal with system differences- it just builds your code.
bch 4y ago
> … why do people continue to create new build tools every couple of years? Seems like a rite[0] of passage to some degree. Perhaps similar to people talking a stab at The Next Actually Correct CMS, and The Next Object System That Doesn’t Suck, or The Next Linux Distro For Smart People. [0] edit: corrected “right V. rite” per https://news.ycombinator.com/item?id=32442473 https://news.ycombinator.com/item?id=32442473
- erik_seaberg 4y ago
  (btw, it’s https://en.wikipedia.org/wiki/Rite_of_passage https://en.wikipedia.org/wiki/Rite_of_passage)
falcolas 4y ago
> why did they have to invent 'configure' Cross-architecture and linux distro compatibility, mostly.
- compiler-guy 4y ago
  Err, pedantically, configure was not for cross Linux distro compatibility, but for cross unix compatibility. It existed long before Linux was a sparkle in Linus’s eye. And even then, it handled even some non unix environments as well.

PaulKeeble 4y ago

Its unfortunate that other build systems haven't taken over. Make is terrible for incremental builds and its reliance on binaries often means issues getting it to run and being very platform dependent. It is better than using a bat or shell file for the same purpose but its a long way behind many of the other language specific tools. I am surprised something better hasn't become popular, Make is the CVS of the build tools.

dwheeler 4y ago
I find it works well for incremental builds, that is what it's for.

BiteCode_dev 4y ago

Make dsl is terrible, it's the yaml of build systems. If you are a python dev, give doit a try.

Koshkin 4y ago
What is so "terrible" about it? I think the rule syntax is as simple as it gets.
- vitiral 4y ago
  Required tabs, lines are each their own "script" instead of blocks (allowing variables), not allowing other executors (i.e. python, TCL, etc would be better than sh).
  - MRtecno98 4y ago
    Actually you can tell GNU Make not to create a new shell for every command using .ONESHELL[0] and you can also select a different shell executable to run said commands[1] [0]: https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... [1]: https://www.gnu.org/software/make/manual/html_node/Choosing-the-Shell.html https://www.gnu.org/software/make/manual/html_node/Choosing-...
  - philkrylov 4y ago
    Check (GNU Make) .RECIPEPREFIX, (GNU Make) .ONESHELL, (POSIX Make) SHELL=
  - blacksqr 4y ago
    Tclmake, a partial Make clone in pure Tcl: https://github.com/TclLab/tclmake https://github.com/TclLab/tclmake
nottorp 4y ago
Interesting that everyone who recommends an alternative to Make picks... something different. I doubt there will be two people recommending the same thing in the whole discussion.

jhallenworld 4y ago

To go along with Make, use this technique to automatically generate and include dependency information: https://scottmcpeak.com/autodepend/autodepend.html https://scottmcpeak.com/autodepend/autodepend.html And this, avoid recursive make: https://accu.org/journals/overload/14/71/miller_2004/ https://accu.org/journals/overload/14/71/miller_2004/

keepquestioning 4y ago

Now we need to banish CMake

Koshkin 4y ago
Yeah, there is something deeply wrong about the whole thing... But what do we replace it with? (Out of desperation, perhaps, I even dreamt of a build system based on a C library, with the usage being, say, tcc -run mybuild.c :)
mdaniel 4y ago
People keep saying that, but of the systems I've encountered that use CMake they've worked the most predictably, as opposed to autotools, meson, scons, bazel, or heavenforbid ./build.sh *WRITING* CMake is the 11th circle of hell, but CLion is gradually getting more support for it because they use(d?) it as the first-class project definition when it launched I would pay good money for CMake 4.x to switch to skylark/starlark

est31 4y ago

Make is solving many complicated tasks, like keeping in mind when to re-run some target (there is communication going on with the compilers that provide make .d files so that they know which source files influence a binary), or running job servers managing parallelism that also support nested sub-makes. But it also has many ugly warts. It's hard to design something that solves both those tasks as well as make does, and also as generalist as make does. Often they are solving a subset, what currently itches the main developer. But something that is both as general, and as comprehensive as make, those tools are rare. Ninja for example checks many of the boxes, but lacks make jobserver support.

bXVsbGVy 4y ago
Ninja looks clean because it is new. Give it some decades and it is likely inherit a few of warts make has.
- aappleby 4y ago
  It's not that new, and in practice there aren't very many ways for it to get warty because it's basically a dependency graph with "run this command line if this node is dirty" attached to the edges. I like it very much.
- LtWorf 4y ago
  Ninja is so cool… until you can't compile stuff because the author had a different ninja version than what's on your distribution…

falcolas 4y ago

I love Make. It's a terrible tool for quite a few things, but it's awesome at the thing I use it most for - abstracting away complex series of shell commands behind one or two words. It's like shell aliases that can follow a repo anywhere. make test make format make clean make docker-stack Fantastically useful stuff, even if all it's doing is calling language specific build systems in the background.

dijit 4y ago

A company I worked for used make to execute docker, which felt somewhat odd. Correct me if I'm wrong, but I always assumed `make` to be one of those build tools that could incrementally build targets based on dependencies. The arcane and esoteric language that constitutes `make` is almost universally avoided, surely if people need a simple task runner there could be better options?

falcolas 4y ago
Better? Absolutely. More ubiquitous? Not really. I use it for executing docker as well, because the docker command is actually about 5 commands with long lists of parameters. But it's just `make docker-stack` for everyone now.
- LtWorf 4y ago
  But then the docker command is starting docker and running the same makefile again inside the container, which becomes hellish to reason about. It's probably better to just make a separate script for that instead.
  - falcolas 4y ago
    That is odd. Within the docker container, I agree, you probably shouldn't do that. Esp. since Dockerfiles themselves allow you to run a list of shell commands as your entrypoint.

jcoq 4y ago

This is a remarkably stupid comment but not everything is "unreasonably effective". Mathematics was noted as unreasonably effective for modeling the universe because most areas of mathematics were not invented for the applications they meet... like discovering that your coffee maker doubles as an exceptionally good waffle maker. Makefiles on the other hand, are not unreasonably effective in this sense. Makefiles are, in fact, _reasonably_ effective... like a coffee maker that brews coffee well.

deleted 4y ago
[deleted]

bXVsbGVy 4y ago

I'm trying to understand why so many people seems to hate make. I hate building system that don't use Makefile, or that use but don't respect the variable convention. It makes really quite annoying to do things like changing allocation library, add compilers flags, etc.

TylerE 4y ago
Because gnu autotools blows goats. Oh, and the tab thing.
- gbrown_ 4y ago
  make != autoconf
  - TylerE 4y ago
    Yes, but 99% of the makefiles you encounter in the wild comes from autotools.
    - dima55 4y ago
      This isn't at all true, in my experience. If it's true for you, please consider that your issues are about autotools and not Make, and direct your complaints in that direction.
    - stjohnswarts 4y ago
      the vast majority of the ones I encounter are from cmake...
    - nrclark 4y ago
      I use Make for all kinds of general-purpose build automation tasks, and as kind of a top-level catch all. No Autotools to be found in them (except maybe if called from some recipe).
- tpoacher 4y ago
  surely it blows gnus rather than goats?

jwilk 4y ago

It's 500 Internal Server Error for me. Archived copy: https://web.archive.org/web/20220812135641/https://matt-rickard.com/the-unreasonable-effectiveness-of-makefiles https://web.archive.org/web/20220812135641/https://matt-rick...

topspin 4y ago

My most elaborate use of make was ten years ago to build a bootable, reproducible embedded system image including the OS, application, media files and cryptographic signatures. It's still in active use and no one is complaining.

hyperupcall 4y ago

Honestly, I only find Makefiles useful when I have a tiny C/C++ project and need stuff just to compile quickly and easily without the overhead of a real build system. For literally everything else, I found myself using it more as a task runner - and Make doesn't do a great job at it. You end up mixing Bash and Make variables, string interpolation, and it becomes really messy, really fast. Not to mention the footguns associated with Make. I found bake (https://github.com/hyperupcall/bake https://github.com/hyperupcall/bake) to suit my needs (disclaimer: I wrote it). It's literally just a Bash script with all the boilerplate taken care of you - what a task runner is meant to be imo

nrclark 4y ago
Does Bake support dependency expression / parallel execution? Those are killer features that keep me on Make for general-purpose automation.

h4l 4y ago

I've found these guidelines for Makefiles make for a pretty good experience using make: https://tech.davis-hansson.com/p/make/ https://tech.davis-hansson.com/p/make/ The advice on output sentinel files for rules creating multiple files helps keep rebuilding dependencies reliable. Avoiding most of the cryptic make variables also helps Makefiles to remain easily understandable when you're not frequently working on them. And using .ONESHELL to allow multi-line statements (e.g. loops, conditional etc) is great. No need to contort things into one line. or escape line breaks. Seems like you could even use a more serious programming language instead of sh/bash by setting SHELL to Python or similar. That may be a road to madness though...

zelphirkalt 4y ago
The problem with .ONESHELL is, that it is for the whole file. I so wish it was per target. That would be really useful. But for the whole file? Maybe I need each line to be a separate shell anywhere in the file and that will make it impossible to use .ONESHELL for the entire file.
- h4l 4y ago
  It is per target, that's how it works when I've used it. For example: # Makefile SHELL := bash .ONESHELL: .RECIPEPREFIX = > thing1: > FOO=bar > echo "$${FOO:?}" .PHONY: thing1 thing2: > echo "$${FOO:?}" .PHONY: thing2 Results in: $ make thing1 thing2 FOO=bar echo "${FOO:?}" bar echo "${FOO:?}" bash: line 1: FOO: parameter null or not set Note how the bash error for unset FOO is line 1 for the second target. Edit: Maybe I misinterpreted, do you mean you'd want to choose whether a given target is ONESHELL or not?
  - zelphirkalt 4y ago
    > [...] do you mean you'd want to choose whether a given target is ONESHELL or not? Yep, exactly! I would like to have some targets use ONESHELL and others in the same Makefile not use ONESHELL. So that I can choose the most appropriate for each target. So far I have managed by avoiding ONESHELL and doing the typical "backslash, next line continues" thingy. But it puts some people off.

philsnow 4y ago

> File-watcher or live-reloading. You can create a build/deploy loop fairly easily When I worked with latex more, I kept a ~/.Makefile-latex and a shell function that would pretty much just do inotifywait -e modify -e move --recursive --monitor . | while read; do make --makefile ~/.Makefile-latex; done and I kept emacs and xpdf in side-by-side windows. Whenever I'd save a file in emacs (or xfig or whatever), a second later xpdf would refresh the pdf (and stay on the same page). It took away some of the pain of working with latex. edit: I used this complicated setup instead of LyX or whatever other "(la)tex IDE" because I had ancillary files like xfig diagrams that would get compiled to .eps files and gnuplot scripts that would render data into graphs, and the makefile knew how to generate everything.

kwantam 4y ago
Very nice! It turns out that latexmk has this functionality: latexmk -pvc -pdf foo.tex (It can be configured to HUP your pdf reader if needed, too.) I usually add something like this command as the `auto` target in my latex Makefiles, which works pretty nicely.
- philsnow 4y ago
  TIL about latexmk (which I gather is a perl script that handles a lot of the latex build deps, like running bibtex at the right time(s), for you automatically), thanks! I never really understood most of the TeX world. I cargo-culted the bits I needed to make: * articles for journal submissions (easy, they all provided packages or templates), * my thesis (somebody had written a latex class that took care of all the school's formatting requirements a ~decade prior), and * a resume, and never dug into the differences between tex, latex, miktex, texlive, auctex, lyx, pdftex, xetex, luatex etc etc etc etc etc. I was just to ask you if you knew of a 40000-foot overview of these kinds of things but stumbled across this article over at Overleaf: https://www.overleaf.com/learn/latex/Articles/What%27s_in_a_Name%3A_A_Guide_to_the_Many_Flavours_of_TeX https://www.overleaf.com/learn/latex/Articles/What%27s_in_a_... , it looks pretty apropos.
earthscienceman 4y ago
This is funny, because I just wrote a (fish) shell script that does this as well because all of the tex IDEs are so painful. Mostly because the entire efficiency of latex is that you're editing text and can do it in a text editor like emacs and move things around very quickly. I don't want a new interface! But. I'm kind of proud. My shell script monitors the tex files for character changes and then once a (configurable) threshold of changes is met, it kicks off the compilation process. But the real game changer is that every time it compiles, if the compile is successful it commits the recent edits to a git branch. Then if I want, I can go through the git branch and see the entire history of edits for only versions of the document that compiled. It's a game changer in a big way. When I finish a section, I squash the minor edits into one commit and give it a good message and then commit the full thing to the main branch. Then there is where I can make sure my manuscripts look the way they should and do major revisions or collaborative edits. The icing on the cake is that the fish script monitors for a special keypress and will take actions. So I hit "c" to compile the tex, "b" to rerun the massive bibliography compilations, "s" to squash the commits into the main branch, and "l" to display log errors. It's a dream! Now I don't think about compilation at all, or when I should commit something minor to git (and fiddle with the commands). I just type away and watch/request the pdf refresh when I need it... and _actually_ get work done. My god. So happy. I just finished this today.

belkarx 4y ago

There was a post on HN (I believe) a couple of months ago concerning a research paper that explores "What defines a makefile" concluding that by their definition, Excel can be used as one. I've looked around the internet and can't find it, so if anyone else remembers it and has it upvoted or otherwise has the link, posting it would add to this conversation. It had interesting discussion of graphs and requirements and was overall a worthwhile read.

yaris 4y ago
Was it “Build Systems a la Carte” paper? I could not find mentions of it on HN after 2020 though.
- belkarx 4y ago
  Yes, thank you! Here's the PDF in case anyone else is interested: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems-5ab0f42d0f937.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... I can't find any actual conversation about it on HN so I won't post the HN link
AceJohnny2 4y ago
I'm pretty sure I found that seminal paper, "Build Systems a la Carte", via HN, but indeed can't find a highly-commented story on the topic. Perhaps the most upvoted story was this one about a blog post from one of the authors (itself very interesting): https://news.ycombinator.com/item?id=17494016 https://news.ycombinator.com/item?id=17494016 It is a very important paper, I think, mapping out the landscape of build systems designs. But I'm still waiting for something novel to emerge from it. I know the authors planned on integrating the improvements they found into (Cloud) Shake, but I don't know if that's production-ready. Perhaps Bazel is the best we can hope for at this time.

ogogmad 4y ago

Is there a Python library that can check a timestamp and update some files according to a lambda? I don't want to learn Make syntax again.

Jtsummers 4y ago
if os.stat(object).st_mtime < os.stat(dependency).st_mtime: ... Use a dict to contain each rule: rules["a.c"] = (["b.c", "c.c", "b.h", "c.h"], action) ... That can be simplified as well and then run a simple parser over it that takes a simpler representation and turns it into that dict. Then: def make(rules, rule): (deps, action) = rules[rule] run_rule = len(deps) == 0 # if there are no dependencies, the rule always runs for dep in deps: make(rules, dep) if os.stat(rule).st_mtime < os.stat(dep).st_mtime: run_rule = True if run_rule: action() Of course, this doesn't actually validate the DAG itself. ---------- An amendment: You'll probably want to pass both the rule name and the dependencies into the action function/lambda/object so that you can parameterize it and maybe reuse the action (like a common compiler command): if run_rule: action(rule, deps)
- andreareina 4y ago
  Topological sorting is in the stdlib: https://docs.python.org/3/library/graphlib.html https://docs.python.org/3/library/graphlib.html
TillE 4y ago
SCons can be used as a simple Make-style tool, in addition to the more complex stuff it supports.

shepherdjerred 4y ago

I'd encourage anyone thinking of using make to look at alternatives. Make is great, but is quickly becomes a ball of duct-tape. Make works very well when you spend the time to express your dependency tree, but realistically that never happens and people tend to add hacks upon hacks for Makefiles. Not only that, but they don't scale well as your project adds more components, such as integration testing, documentation, etc. I found Earthly[0] to be a great replacement. Everything runs in Docker. Your builds are reproducible, cache-able, and parallelizable by default. I've heard Dagger[1] is another good tool in the space. [0]: https://earthly.dev/ https://earthly.dev/ [1]: https://dagger.io/ https://dagger.io/

davidpfarrell 4y ago

As these types of post often come around to make's sub-optimal use as general runner, I'd like to point out my project, Run: https://github.com/TekWizely/run https://github.com/TekWizely/run It feels like a makefile but is optimized managing and invoking small tasks and wrappers, and auto-generates help text from comments.

olliej 4y ago

I find a bunch of the decisions in Make to not be great - but it's all just syntax stuff. At its core all a Makefile is is a dependency graph, which is necessary in all the more advanced config managers, only those others are much more heavyweight than the vast majority of projects ever need. Most code doesn't need to vary arguments or parameters based on the target environment, and so that complexity is unneeded, in which case a Makefile can do it all.

eichin 4y ago

Aww, noone mentioned that `debian/rules` files are almost always makefiles? (To the point of starting with `#!/usr/bin/make -f` ...)

_8j50 4y ago

Autogen/autotools mess however... not so pleasant when it breaks. Curious for those who published source with autotools,etc... do you really sit down and figure that out. It's just a mystery that works or doesn't to me. I never have or would go beyond a simple make file. It really seems tedious on top of the actual code you write. A bit impressive tbh.

jbboehr 4y ago
Yes, I used autotools[0]. It's definitely hairier than plain make, but you get a lot of useful features on top of it. There's thousands of examples all over the internet so it's easy to reference them. I like the elegance of pure make, and do use it when appropriate, but I wouldn't really want to reimplement the things autotools does myself in it. [0]: https://github.com/jbboehr/handlebars.c/blob/master/configure.ac https://github.com/jbboehr/handlebars.c/blob/master/configur...
LtWorf 4y ago
What's the alternative? I don't know in advance if the system's seek() supports 64 bits directly or just 32. Or if the user wants a more minimal binary without a certain feature.
- UncleEntity 4y ago
  Cmake? Once I figured out how to get it to build python extension modules my life became a lot simpler. The python way (distools?) is easy enough until it isn’t then I reach for cmake.
  - LtWorf 4y ago
    cmake is not as powerful as autotools.

shadowgovt 4y ago

Does make still basically fail to handle filenames with spaces in them? That was the deal-breaker for me last I checked.

LAC-Tech 4y ago

I wanted to learn makefiles to generate a static website. Quickly ran into some massive limitations - one of which is that it completely broke apart when filenames had spaces in them. "But why would you do that you're doing it wrong" - don't care wanted spaces in filenames. Ended up switching Rake (a make like DSL written in ruby) and never looked back. Not only can you do all the make declarative stuff, but you get the full power of ruby to mess around with strings.

rightbyte 4y ago
Next thing you ask for leading tabs in filenames ... And the feature bloat slippery slope is sliding towards Bazel!

deleted 4y ago

[deleted]

kazinator 4y ago

Anyone who finds make unreasonably effective must be working with GNU Make. If I had to use some barely POSIX conforming thing from BSD thing or wherever, I'd instead write a write a top-to-bottom linear shell script full of conditionals.

makapuf 4y ago
In that case,a good habit is to name those GNUMakefile. Works the same, but announce gnu make (which is the one worth it)
- kazinator 4y ago
  GNU Make is so prevalent that nobody seems to bother with this. I can't recall the last time I've seen a GNUmakefile. (Note the lower case 'm', according to the manual.) It makes sense if you ship both a GNUmakefile and a Makefile.
  - dwheeler 4y ago
    I agree. I try to stick to just POSIX make when that's easy to do, but POSIX make is so impoverished that it usually isn't worth trying to stick to just POSIX make. In general, for most people, make is GNU make. And GNU make is actually pretty decent at a lot of tasks. Many ridiculous makefile problems I see stem from not using it well. I suggest: * Use compiler dependency generators (generate .d files and "include" them via make). This eliminates many lines and errors. * Don't use recursive makefiles. * Set macros with := or ::= * Use substitutions so definitions can be changed in just one place and the rest automatically works. There is no perfect tool for all circumstances. But GNU make is still useful for many circumstances.
    - kazinator 4y ago
      Here is an odd thing: GNU Make looks for the file in this order (according to the manual): GNUmakefile, makefile, Makefile. The last time I saw a lower case 'm' makefile was eons ago. The manual itself recommends Makefile rather than makefile, right in the same paragraphs where it tells you makefile is checked first. Why look for makefile before Makefile, a file next to nobody has. Maybe lower-case makefile was a predecessor to GNUmakefile; a way of having a GNU-specific makefile. (If so, that's a noteworthy thing the manual ought to mention.)

aappleby 4y ago

Recently I've been much more pleased by the "Unreasonable Effectiveness of Ninja Plus A Few Trivial Scripts". A raw Ninja file is verbose but easy to read and understand, and there is basically no magic happening behind the scenes (well, except for "deps = gcc", but that's minor). Any action that's too complicated to go directly in the Ninja file goes to a "rule command \ command = ${command}" that just runs a python or shell script. The only thing I'd like to add to my setup would be "Ninja with glob support", but again that can be handled in a few lines of Python that spit out another chunk of .ninja rules so it's not really a blocker.

evilotto 4y ago

The 2 biggest things I find missing in makefiles is the ability to have the "modified date" of a target be something other than the mtime of the target file, and for the dependencies of a target to be generated rather than statically specified. The latter there are workarounds for, but not the former. The author's suggestion of "better support for make docker" is completely the wrong thing to do - you don't need to support docker, you need to support mtimes and dependencies in a way that docker could use.

acdw 4y ago

my website was a gnu makefile for a minute : https://gitlab.com/acdw/acdw.net https://gitlab.com/acdw/acdw.net

kwertyoowiyop 4y ago

Please, no more “unreasonable effectiveness” headlines.

k0k0r0 4y ago

Pretty much discussion about make as a task runner, and maybe the following is just a very specific use case: make test I combined learning make with learning TDD, or rather develope some individual style of TDD. I am astonished how effective make is in a rapidly evolving code base with not insificant amounts of unit, component and integration tests. That said I try hard to use the makefiles both, heavily recursive and highly structured. Of course, I am learning, so my set up is far from optimal. But wow.... Makefiles are so powerful for testing. Its amazing. On the other hand, that the first time I am properly using a build system.

kelseyfrog 4y ago

The fact that it's likely to be pre-installed on any given linux machine is a huge plus. I use make for providing a uniform interface for my team's data science project workflow and while it has a lot of downsides, it also has a upsides too. It's probably the choice I've struggled the most to justify, but I have a harder time justifying the alternatives.

dwheeler 4y ago

I once used GNU make to manage a large data pipeline on an 18-person project and it worked well. We developed a lot of Python scripts. To manage them I created some helper tools to integrate them via make. People are welcome to reuse it, I got it released it as open source software. I named it make-booster: https://github.com/david-a-wheeler/make-booster https://github.com/david-a-wheeler/make-booster From its readme: "This project (contained in this directory and below) provides utility routines intended to greatly simplify data processing (particularly a data pipeline) using GNU make. It includes some mechanisms specifically to help Python, as well as general-purpose mechanisms that can be useful in any system. In particular, it helps reliably reproduce results, and it automatically determines what needs to run and runs only that (producing a significant speedup in most cases)." "For example, imagine that Python file BBB.py says include CC, and file CC.py reads from file F.txt (and CC.py declares its INPUTS= as described below). Now if you modify file F.txt or CC.py, any rule that runs BBB.py will automatically be re-run in the correct order when you use make, even if you didn't directly edit BBB.py." This is NOT functionality directly provided by Python, and the overhead with >1000 files was 0.07seconds which we could live with :-). Make provides a way to handle dependencies as DAGs. Using it at scale requires that you call or write mechanisms to provide that DAG dependency info, but any such tool needs that info. Some compilers already come with generators for make, and in those cases it's especially convenient to use make.

mturmon 4y ago
I really like using make for data pipelines as you suggest, and thanks for pointing out your package. In this pipeline use case, you have base data, and a series of transformations that massage it into usable results. You are always revising the pipeline, usually at the output end (but not always) so you want to skip as many preprocessing steps as possible. Make automates all that. This works great for image processing pipelines, science data pipelines, and physical simulators for a few examples. There have been a few blog posts and ensuing HN discussions about this use pattern for make. The discussion generally gets mixed up between make’s use as a build system for code, alas.
- dwheeler 4y ago

tomohawk 4y ago

Even more powerful: rake

quickthrower2 4y ago

I was wondering, if you don’t build a single timestamped artefact (i.e. everything is phony) is there any reason to use makefile over a set of bash scripts?

bandrami 4y ago

Years and years ago (before the init wars) I made an init system that was just a makefile run by a statically compiled make binary. While this was fun it outlined what I dislike about declarative run control systems (you don't see what's actually going on without having it be pointlessly verbose). But as far as that went it ran my silly micro distro pretty well.

Pinus 4y ago

I skimmed the comments quite quickly so I may have missed it, but it seems that nobody has mentioned what I think is one of the biggest issues with make (as actually used, as opposed to some ideal): recursive runs. I have seen several packages (and I used to work with one rather large code base) which were subdivided into multiple directories, sometimes in several levels, each with its own Makefile and some rule that did for d in $(SUBDIRS); do cd $d && $(MAKE); done A top-level make run would take ages just to recurse through everything to decide that nothing needed to be done, and dependency tracking did not work across modules, so the only way of making sure you got everything rebuilt was to ‘touch’ everything. (The advantage of the setup was that you could check out just part of the tree and build it separately.) It would be really nice if make could read all subdirectory Makefiles, build one global dependency graph and then have at it.

fmajid 4y ago

Like democracy, make is the worst build system to the exception of every other other. Like Maven. What demented mind thought XML was a good idea compared to the simple and clean DSL of Make (or Ninja)?