11 ms·
Show HN: Making GNU Make a better Task Runner
I know this could be considered blasphemous, but I constantly find myself using Make as a task runner. I have written my own task runner in the past, but somehow I always end up using make.
I went, and I put together 3 quality of life snippets I use all the time and put it in a single makext.mk file that can be included in other Makefiles and wrote a basic readme for it.
This is not meant to be a replacement for other task runners, but I do think it can be useful to some of you.
https://github.com/mitjafelicijan/makext https://github.com/mitjafelicijan/makext
Check it out and see if it makes sense to you.
Thanks for any feedback or comments.
Cheers
- smartmic 2y agoThanks, that is a good idea. I also prefer GNU Make for automatization if the number, diversity or dependencies of the tasks would exceed the benefits of a simple bash script. I stumbled over this statement in your README: > It is recommended to use .PHONY for targets that are not actual files. In the example below I am not doing that though, but it is wise to follow that rule. Please also give the example as you think it would be correct. Or write the understandable reason why you don't do it here. I think many people don't read the text between the code snippets but only scan the code examples. It would be helpful to find the ‘best practices’ right there.
- mitjafelicijan 2y agoYou are correct. I will add this. I was not clear enought on this.
- fmajid 2y agoremake is a more powerful tool, a fork of GNU Make that adds a debugger and profiler, among other extensions. https://remake.readthedocs.io/en/latest/ https://remake.readthedocs.io/en/latest/
- Galanwe 2y ago> I know this could be considered blasphemous, but I constantly find myself using Make as a task runner It's not, don't apologize, reusing battle tested tech is the way. I have a quite close "template Makefile", though I prefer to store it as a cookiecutter (or more recently "copier") template for easier per-project bootstrap. You could add a timestamped log function. I use that in mine so that targets have timestamped messages pre/post.
- mitjafelicijan 2y agoThis is a very good idea. I will add that. Thank you.
- wodenokoto 2y agoA while back someone linked to this relevant comment on HN, about using make as orchestration: https://news.ycombinator.com/item?id=32441602 https://news.ycombinator.com/item?id=32441602 I also believe I've seen someone setup tools that can draw the dependency graph for you, so you can view (not edit) it airflow style.
- tpoacher 2y agoVery nice. Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky. That, and the syntax is very flexible, which means it can easily be abused to write ugly/convoluted makefiles. If I had one thing I'd want to improve about make, it wouldn't be the tool itself, but its documentation, and a set of reasonable defaults/templates, like you have here. The documentation is amazing in one sense, in that it's very comprehensive, but very clunky and hard to master in another, in that it's organised in a very high-level manner that doesn't allow it to be used as a quick reference / point of truth, if you don't know what you're looking for. I found myself making anki notes for it, as it's not organised in the kind of way that would simply allow me to refer to the docs for your usecase and get things done. You really either know some functionality exists and can look it up or you don't, and unless you're someone who uses make in expert mode on a daily basis, it's hard to know all the specialised components and be able to combine them all together seamlessly. Hence my anki notes, hahah. But make really is an amazing tool. I wish people focused on improving make workflows instead of reinventing the wheel all the time with other tools (which are then forced on the user).
- frankjr 2y ago> Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky. If you're looking for a task runner, not a build tool, then make is not "far superior" in any sense and there are much better alternatives, the most prominent probably being "just". Even something as basic as accepting parameters to a task (make fetch <arg1> <arg2>) is awkward because make doesn't understand this syntax and interprets the arguments as targets. The only way to make it work is either through named arguments or empty target hacks and in both cases you need to write the validation logic yourself. In just it's simply: fetch PACKAGE VER: @echo fetching {{PACKAGE}} at version {{VER}} $ just fetch foo error: Recipe `fetch` got 1 argument but takes 2 usage: just fetch PACKAGE VER $ just fetch foo 1.2.3 fetching foo at version 1.2.3 https://just.systems/man/en/chapter_1.html https://just.systems/man/en/chapter_1.html
- keypusher 2y agoMay want to look at Just. It is heavily inspired by Make and shares much of the same syntax, but removes a lot of the workarounds necessary to use Make as a task runner and adds a few other features. https://github.com/casey/just https://github.com/casey/just
- nrclark 2y agoIf Just grew to have a large enough feature-set that it could replace GNU Make, it would collect an equal amount of warts and sharp-edges along the way. It's a little more user-friendly, but not enough to justify how much power you lose. I spent the last 20 minutes reading Just's documentation, and it seems like the only real wins are nicer argument parsing and && dependencies, and the --list argument. And in exchange for that, you lose file-based dependendicies and Make's powerful templating engine. Did I miss some other features that make the trade worth it?
- zokier 2y agoFor lots of uses not having file-based dependency mechanism is a benefit, not a con. Sometimes less is more
- nrclark 2y agoFor sure. But you also don't have to use file-based deps if you don't want. Make doesn't require it in any way. You can mix-and-match.
- bigstrat2003 2y agoI don't want file-based dependencies. I don't understand why anyone would, frankly. Just is an upgrade in that respect, not a downgrade.
- taeric 2y agoI'm intrigued. Without the file based dependency stuff, is it all incumbent on you to add the logic checking and stating if something has been done?
- barries11 2y agoNice, I will definitely use these techniques. Here are a few settings that make GNU make a bit faster, and enable multi-line, strict-mode bash scripts as recipes (make recipes are normally sequences of single-line sh invocations)--and enable quietude & tracing: https://github.com/barries/polling_state_machine_cpp/blob/main/make_config.mak https://github.com/barries/polling_state_machine_cpp/blob/ma...
- mitjafelicijan 2y agoThis is really nice. Thank you for this!
- woodrowbarlow 2y agopeople often turn to make as a taskrunner, but i've never understood why. i've heard people say "i don't want to add another tool/language/dependency to my project, so i'll just use make" but, usually, `make` is a new dependency for the project. `make` (realistically) excludes Windows users, most Linux distros don't ship it by default, it is a unique syntax (no, it's not shell). even in a C project, it's not a given that Make is already a dependency. in languages like python or javascript, with rich package ecosystem, `make` makes even less sense; why limit your users to one OS, and why not write your tasks in your project's native language? as a taskrunner, `make` has so many limitations -- because that's not what it's designed for. `make` is good for tracking dependencies between files that are generated from other files, and that's about it. a good taskrunner makes it easy to run a task while still exposing the tool's underlying flexibility. a good taskrunner lets me invoke the entire test suite with a short command, but also allows me to add custom options and arguments to, say, run a specific test case in an alternate environment. `make` fails to expose the tools' underlying flexibility. sure, you can write a .PHONY target to run the full test suite, but `make` can't handle passing options or arguments (besides cumbersome Makefile variables). a makefile tends to obscure the underlying tool, enshrining its launch arguments. anyone who has tried to cross-compile a makefile codebase authored by someone who didn't consider cross-compilation understands what i mean (you'll end up re-writing the Makefile 90% of the time).
- wodenokoto 2y agoThen what is a good task runner?
- woodrowbarlow 2y ago`just` is great, especially if you're using rust. if you're in python or javascript, it's well worth it to use a language-native runner (like pydoit / grunt). that way your "task" is just a function, it can take arguments (and you can define default values), you can even use something like `*args, **kwargs` to take any options/arguments, and then just pass them verbatim onto the subprocess.
- maccard 2y ago
- jjgreen 2y agoI saw MEX_LICENSE and thought: What, this requires a Matlab licence?!?
- mitjafelicijan 2y agoI did not knew they use this naming convention. :) I should probably choose something else :)
- jjgreen 2y agoMatlab mex files are what they call their plugins ("Matlab executable" I guess), so they're just .so files on Linux etc; I don't think they actually use "MEX_LICENSE" as the envar for their keys, but it's feasible that they might which is why it jumped out at me ...
- blacklion 2y agoI wonder, why GNU make is much more popular than BSD make/pmake. Both are extensions to classic make (makefiles without advanced features works with both implementations), but, to my eye, pmake extensions is much more simpler and nicer to use, than GNU make extensions. Simple ".if/.else/.endif" and true ".for" loop, like in sh is much more comprehensible than GNU make functional extensions to me. Is it only me? And BSD systems have extensive library for pmake already, which allows to build programs, libraries (both shared and static) and more in declarative way, without any custom rules. It is like these extensions, but battle-tested for 20+ years. With nice license, too.
- 0xbadcafebee 2y agoI've never found the need for a task runner since learning shell scripting. A lot of people recommend just; here's their sample file: alias b := build host := `uname -a` build: cc *.c -o main test-all: build ./test --all test TEST: build ./test --test {{TEST}} Here's how I would do that with shell: #!/usr/bin/env sh set -eu b="build" host=`uname -a` _cmd_build () { # build: Build main program cc *.c -o main } _cmd_test_all () { # test_all: Run all tests _cmd_build ./test --all } _cmd_test () { # test TEST: Run a test TEST _cmd_build ./test --test "$1" } _cmd_help () { echo "Available targets:" grep -E "^_cmd_[^[:space:]]+ \(\) {.+" "$0" | sed -E 's/.*#/ /' } [ $# -gt 0 ] || _cmd_help set -x "_cmd_$1" "$@" $ chmod 0755 run.sh $ ./run.sh Available targets: build: Build main program test_all: Run all tests test TEST: Run a test TEST ./foo.sh: line 23: $1: unbound variable $ ./run.sh build $ ./run.sh test_all $ ./run.sh test foo.t But this example (from just's homepage) is clearly a build process, so I would use Make for it instead. When I need to be able to run individual sets of commands with arguments and options, I make a shell script. Many more features available, more flexibility, I can tailor it to my use case.
- kstrauser 2y agoTwo things Just gives you beyond that: - It handles all that boilerplate for you. The only stuff in the justfile is code you want to execute, not the code to figure out which code to execute. - Out-of-the-box tab completion. "What did I name that recipe? `just t<tab>` Oh, `test-all`, that's right!" I like not reinventing those wheels. Let someone else manage that hassle for me.
- pama 2y agoThanks! Typically I don’t like solutions that involve extra environment variables as things inevitably get messy over time, but your use cases are clean, simple, and common enough that it’s nice to have them documented in one place in this extra makefile.
- crabique 2y agoBiggest showstopper for me personally is Make's inability to properly pass through arguments from `make run <arbitrary number and type of arguments>` to the underlying program. Some of the issues could be avoided by requiring a -- and using the .SILENT modifier, but some are extremely difficult: e.g. it word-splits strings and you can't just pass "some string" as a single argument, everything will be separated by spaces. In case the underlying program is a script-language wrapper that you also control, it is possible to hack around and let Make pass through its own $$PPID so that the underlying script could read the nul-terminated /proc/<Make's pid>/cmdline and pass it on to the actual program. This, of course, only works where procfs is a thing, so e.g. on macOS (if it's among your target platforms) you'd have to reinvent the wheel and learn a whole lot of Darwin sysctl dark magic to read arbitrary process' arguments, at which point you'd ask yourself if this is even something a sane person would ever do...
- simonmic 2y agoI used make as a task runner for probably 30 years; using all the tricks to do most of the things you want in that use case (arguments, help, portability, reliability..). I worked around all the idiosyncracies continually. That's enough time sunk into wasteful friction for me. Now just is here, and the people pushing it are right. For this job (task/script manager for more than trivial commands) it's better enough and reduces cognitive load enough that it's very often worth the install requirement and new learning curve. In time it will be more installed-by-default and I'm certain it'll acquire some form of make-like dependent building as well. [Apologies for contributing to the slightly off topic discussion. makext sounds great for people still using make for this and I totally would have used it in the past.]
- mitjafelicijan 2y agoI totally get it. I know the feeling. I have written some complex Makefiles in the past for C projects that looked completely horrid. For smaller projects I like make and people who work with me and know me always first check Makefile to see what is there and what the project is about. I use it as a launchpad for installing dependencies and things like that. And it's all about what is appropriate for that project. Something a shell script is better. Sometimes soemthing else. So I completely understand what you are saying.
- simonmic 2y agoAppreciate you saying so mitjafelicijan. You're right, it's all about what is appropriate for the project at a given time.
- simonmic 2y agoFor anyone interested (though to compare you really need to work with them over a period of time): Here's a fairly simple thousand line Makefile: https://github.com/simonmichael/hledger/blob/2d35b1051/Makefile https://github.com/simonmichael/hledger/blob/2d35b1051/Makef... (and https://github.com/simonmichael/hledger/blob/2d35b1051/Makefile.helpsys https://github.com/simonmichael/hledger/blob/2d35b1051/Makef...) that was converted to a Justfile: https://github.com/simonmichael/hledger/blob/43c93eb37/Justfile https://github.com/simonmichael/hledger/blob/43c93eb37/Justf... And here's a more powerful kind of makefile using a full programming language: https://github.com/simonmichael/hledger/blob/43c93eb37/Shake.hs https://github.com/simonmichael/hledger/blob/43c93eb37/Shake... And here are two multicommand shell scripts: https://github.com/simonmichael/hledger/blob/43c93eb37/bin/ft https://github.com/simonmichael/hledger/blob/43c93eb37/bin/f... https://github.com/simonmichael/hledger/blob/43c93eb37/bin/tt https://github.com/simonmichael/hledger/blob/43c93eb37/bin/t... that were converted to a justfile: https://github.com/simonmichael/hledger/blob/43c93eb37/bin/justfile https://github.com/simonmichael/hledger/blob/43c93eb37/bin/j...