11 ms·

Polyglot Makefiles

153 points by sciencerobot 6y ago

mehrdadn 6y ago
I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?
- sciencerobot 6y ago
  The shell is specific to each target. So doing `make ruby bash python docker` works. It even works in parallel if you do `make -j`. Edit: I'm the author.
  - mehrdadn 6y ago
    And what happens if there's overlap between the targets?
    - rhencke 6y ago
      Define 'overlap'?
      - mehrdadn 6y ago
        What if they have a common prerequisite?
brandmeyer 6y ago
That's interesting. Author probably wants to use `private` for those target-local variables, though. For example, R: .SHELLFLAGS := -e R: SHELL := Rscript R: greeting = "bonjour" message(paste0(greeting, ", R!")) Everything that target `R` depends on will also have SHELL and .SHELLFLAGS over-ridden. If `R` depends on some data generated by another program, it probably wants to be built and executed with the default SHELL (or another shell, perhaps). R: private .SHELLFLAGS := -e R: private SHELL := Rscript R: greeting = "bonjour" message(paste0(greeting, ", R!")) Now, `R`'s dependencies will be generated with the makefile's defaults. Usually I prefer to build up richer stages like this using the system shell anyway, though. Build a target which in turn is executed by the shell normally to traverse the next edge in the graph. But I can see how this mechanism has its uses. See also https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html https://www.gnu.org/software/make/manual/html_node/Target_00...
- sciencerobot 6y ago
  I didn't know about private. Thanks for the tip.
epistasis 6y ago
For complicated pipelines that I want to reuse multiple times, I have turned Makefiles into executables by putting this at the top: #!/usr/bin/make -f And then putting them in my $PATH. I run them with arguments like: $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished This makes me feel like I've sold my soul to the devil, and that I'm just living on borrowed time until it all fails and falls apart. It hasn't yet, however...
- haolez 6y ago
  In your scenario, "make processed_data" makes more sense semantically than rules commonly seem in the wild like "make run" or "make deploy"
- wahern 6y ago
  -f is guaranteed by POSIX, and #! is de facto portable.[1] My criteria for shame is, "will this silently break in the future?". I think you're good. It's not my style, but if it were something I came across at work, so long as it worked well it wouldn't even cross my mind to try to "fix" it. FWIW, using make -f in the shebang is also done for debian/rules in Debian package builds. I don't know if it serves any real purpose. I suppose it permits one to write a bespoke script for building targets without using make.[2] I guess I wouldn't be surprised if someone, somewhere depended on that capability, given how old and widespread Debian packages are. [1] /usr/bin/env make -f would be better, but then you run afoul of the problem that you can't portably pass more than a single explicit shebang command argument. [2] Which I see now is a bonus to your process-data.mk script. It could be replaced with a non-make version without effecting callers.
- asveikau 6y ago
  I can see two areas in which it might break. I might put #!/usr/bin/env make -f in case it's somewhere in else in PATH. Also some systems (BSD, old commercial Unix) have non-gnu-compatible make and sometimes call their gnu make port "gmake" or "gnumake".
  - tomjakubowski 6y ago
    That shebang is appealing but unfortunately more than one argument (past the initial command name) in a shebang is unportable: some OSes will coalesce the extra arguments into one, others make them separate arguments. There's also a special bonus papercut you might hit when /usr/bin/env is in the shebang with extra arguments: an infinite loop! Sorry for the plug: I wrote about it here. https://www.crystae.net/posts/2019/11/08/two-shebang-papercuts/ https://www.crystae.net/posts/2019/11/08/two-shebang-papercu...
ainar-g 6y ago
Note that this article (like many, many others) assumes GNU Make. POSIX Make has neither .ONESHELL nor local macros. Neither do most built-in Make implementations in other OSes, like OpenBSD's bmake.
- ddevault 6y ago
  POSIX shell is also notoriously obtuse and difficult to use. As a big advocate of POSIX as a target, I don't blame anyone for using GNU make - or perhaps BSD make is a better lowest common denominator. Personally, I try to use POSIX Makefiles, but I often find that they're most useful as a target for Makefile generators (in my case, these are usually a shell script called configure).
  - ainar-g 6y ago
    One person's obtuseness is other person's simplicity :-) . In all of my personal and some of my work projects I used nothing but portable features in Shell, Make, Sed, etc. Checking with multiple implementations where possible. As long as you use the right tool for the right job, there shouldn't be any problems. The most common mistake of that sort that I've seen is people trying to do complex conditionals inside their makefiles when they clearly would be better off in a Shell script. (I'm looking at you, fans of ifeq.)
    - wahern 6y ago
      You can implement conditionals semi-portably. See https://github.com/wahern/autoguess/blob/master/Makefile.guess.sub https://github.com/wahern/autoguess/blob/master/Makefile.gue..., which works with GNU Make, NetBSD/FreeBSD make, OpenBSD make, and Solaris make. Alas, it doesn't work with AIX's native make. Once POSIX standardizes "!=" then POSIX-portable conditionals will be possible using the same technique as above, replacing, e.g. OS = $(shell $(OS.exec))$(OS.exec:sh) with just OS != $(OS.exec). Though, you'd need to wait for Solaris, AIX, and macOS gmake[1] to add support for !=. Alternatively, if you add an extra level of indirection using .DEFAULT to capture and forward make invocations, you can simply pass OS, etc, as invocation arguments. Indirection solves everything, though, so that's cheating. [1] Apple's ancient GNU Make 3.81 predates != support. :(
gregwebs 6y ago
Make was designed for building dependencies. I think it is always problematic to use it as a command runner (for example there is no standard way to list out the available commands). [just](https://github.com/casey/just https://github.com/casey/just) is a tool that feels similar to make but is designed explicitly for the purpose of running commands. I think of this as a simple CLI for your project workflow. You still really want to avoid putting code into a Justfile and put it into scripts. But the Justfile helps provide a slightly nicer UX and automatically invoke dependencies.
- oso2k 6y ago
  When you say, "there is no standard way to list out the available commands", do you mean like a `make help`? https://gist.github.com/prwhite/8168133#gistcomment-3114855 https://gist.github.com/prwhite/8168133#gistcomment-3114855
  - lytedev 6y ago
    While you you could argue that this is a convention -- I'd say this isn't even a common one -- it's still a long way from being "standard".
  - jackewiehose 6y ago
    "make help" is definitely not a standard. For all I know "make help" builds help.exe. But there is a standard way to get available commands: Just look at the README or open the Makefile with a text-editor! The lack of a standard argument for getting help doesn't make it problematic for use as a command runner. You can't get atomatically all available commands from a Makefile just like you can't get all command-line flags from an executable. The program/Makefile has to provide it by itself.
- chubot 6y ago
  Yes I agree with this. I use shell instead of make, because make wrapps shell and its syntax collides very poorly with it. For example, the PID is now $$$$ and not $$. Most people forget to mark their targets .PHONY, so they have a subtle bug in their build (touch build; touch test). ---- But shell also suffers from the problem where it doesn't list the commands. I filed a bug for Oil shell here: https://github.com/oilshell/oil/issues/751 https://github.com/oilshell/oil/issues/751 I mentioned a couple other "frameworks" there like just, go, Taskfile, etc. But it should really just be built into the shell, since it's so common. And there should be command completion too, which I think bash-completion has for Makefiles on many distros. Apparently there is no standard name for this kind of "task runner". But I think shell makes a lot more sense than a custom format, because there are many instances where you need a simple loop or conditional. It scales better. (And Oil also fixes bad shell syntax while remaining compatible: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...) If anyone wants to help let me know :) The code is plain Python and pretty hackable. However it generates fast C++, so you get the best of both worlds (in progress)
adelarsq 6y ago
I added this to my list about Make at https://github.com/adelarsq/awesome-make https://github.com/adelarsq/awesome-make Pull requests are welcome.
IshKebab 6y ago
I wish someone would write a modern alternative to GNU Make. I've looked and there don't seem to be any. The closest is Ninja but it doesn't seem to be intended to be hand written.
- theshrike79 6y ago
  There are a lot of options, but make is just everywhere. Sometimes it's just simpler to bite the ancient bullet and go with a Makefile, with all its included pains and gotchas rather than try to figure out how to get the fancy new makefile replacement installed in all the relevant environments.
- epistasis 6y ago
  A lot of people in bioinformatics use SnakeMake. In this field you often want to restart analysis after something changes somewhere along a pipeline (for example the pipeline is under active development and changing frequently), and individual steps can take hours or more, so automatically rerunning just the right stuff is a great feature. However, SnakeMake, Nextflow, etc feels excessively verbose compared to standard make. And the prior workflow managers of last decades were far worse. With standard make, you type pretty much exactly what you would for shell commands, and not too much more. All other alternatives are going to be more verbose than make, and to me that's a negative.
- boris 6y ago
  https://build2.org/build2/doc/build2-build-system-manual.xhtml#intro https://build2.org/build2/doc/build2-build-system-manual.xht...
  - brandmeyer 6y ago
    > We believe, paraphrasing a famous quote, that those who do not understand make are condemned to reinvent it, poorly. Alas, I think the developers have in fact done just that. Make is not a build system. Make is a batch shell. Build2 didn't produce a better make, they produced yet another C++ build system.
purplezooey 6y ago
loving the python3 makefiles

153 points by sciencerobot 6y ago

I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?

sciencerobot 6y ago
The shell is specific to each target. So doing `make ruby bash python docker` works. It even works in parallel if you do `make -j`. Edit: I'm the author.
- mehrdadn 6y ago
  And what happens if there's overlap between the targets?
  - rhencke 6y ago
    Define 'overlap'?
    - mehrdadn 6y ago
      What if they have a common prerequisite?

brandmeyer 6y ago

That's interesting. Author probably wants to use `private` for those target-local variables, though. For example, R: .SHELLFLAGS := -e R: SHELL := Rscript R: greeting = "bonjour" message(paste0(greeting, ", R!")) Everything that target `R` depends on will also have SHELL and .SHELLFLAGS over-ridden. If `R` depends on some data generated by another program, it probably wants to be built and executed with the default SHELL (or another shell, perhaps). R: private .SHELLFLAGS := -e R: private SHELL := Rscript R: greeting = "bonjour" message(paste0(greeting, ", R!")) Now, `R`'s dependencies will be generated with the makefile's defaults. Usually I prefer to build up richer stages like this using the system shell anyway, though. Build a target which in turn is executed by the shell normally to traverse the next edge in the graph. But I can see how this mechanism has its uses. See also https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html https://www.gnu.org/software/make/manual/html_node/Target_00...

sciencerobot 6y ago
I didn't know about private. Thanks for the tip.

epistasis 6y ago

For complicated pipelines that I want to reuse multiple times, I have turned Makefiles into executables by putting this at the top: #!/usr/bin/make -f And then putting them in my $PATH. I run them with arguments like: $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished This makes me feel like I've sold my soul to the devil, and that I'm just living on borrowed time until it all fails and falls apart. It hasn't yet, however...

haolez 6y ago
In your scenario, "make processed_data" makes more sense semantically than rules commonly seem in the wild like "make run" or "make deploy"
wahern 6y ago
-f is guaranteed by POSIX, and #! is de facto portable.[1] My criteria for shame is, "will this silently break in the future?". I think you're good. It's not my style, but if it were something I came across at work, so long as it worked well it wouldn't even cross my mind to try to "fix" it. FWIW, using make -f in the shebang is also done for debian/rules in Debian package builds. I don't know if it serves any real purpose. I suppose it permits one to write a bespoke script for building targets without using make.[2] I guess I wouldn't be surprised if someone, somewhere depended on that capability, given how old and widespread Debian packages are. [1] /usr/bin/env make -f would be better, but then you run afoul of the problem that you can't portably pass more than a single explicit shebang command argument. [2] Which I see now is a bonus to your process-data.mk script. It could be replaced with a non-make version without effecting callers.
asveikau 6y ago
I can see two areas in which it might break. I might put #!/usr/bin/env make -f in case it's somewhere in else in PATH. Also some systems (BSD, old commercial Unix) have non-gnu-compatible make and sometimes call their gnu make port "gmake" or "gnumake".
- tomjakubowski 6y ago
  That shebang is appealing but unfortunately more than one argument (past the initial command name) in a shebang is unportable: some OSes will coalesce the extra arguments into one, others make them separate arguments. There's also a special bonus papercut you might hit when /usr/bin/env is in the shebang with extra arguments: an infinite loop! Sorry for the plug: I wrote about it here. https://www.crystae.net/posts/2019/11/08/two-shebang-papercuts/ https://www.crystae.net/posts/2019/11/08/two-shebang-papercu...

ainar-g 6y ago

Note that this article (like many, many others) assumes GNU Make. POSIX Make has neither .ONESHELL nor local macros. Neither do most built-in Make implementations in other OSes, like OpenBSD's bmake.

ddevault 6y ago
POSIX shell is also notoriously obtuse and difficult to use. As a big advocate of POSIX as a target, I don't blame anyone for using GNU make - or perhaps BSD make is a better lowest common denominator. Personally, I try to use POSIX Makefiles, but I often find that they're most useful as a target for Makefile generators (in my case, these are usually a shell script called configure).
- ainar-g 6y ago
  One person's obtuseness is other person's simplicity :-) . In all of my personal and some of my work projects I used nothing but portable features in Shell, Make, Sed, etc. Checking with multiple implementations where possible. As long as you use the right tool for the right job, there shouldn't be any problems. The most common mistake of that sort that I've seen is people trying to do complex conditionals inside their makefiles when they clearly would be better off in a Shell script. (I'm looking at you, fans of ifeq.)
  - wahern 6y ago
    You can implement conditionals semi-portably. See https://github.com/wahern/autoguess/blob/master/Makefile.guess.sub https://github.com/wahern/autoguess/blob/master/Makefile.gue..., which works with GNU Make, NetBSD/FreeBSD make, OpenBSD make, and Solaris make. Alas, it doesn't work with AIX's native make. Once POSIX standardizes "!=" then POSIX-portable conditionals will be possible using the same technique as above, replacing, e.g. OS = $(shell $(OS.exec))$(OS.exec:sh) with just OS != $(OS.exec). Though, you'd need to wait for Solaris, AIX, and macOS gmake[1] to add support for !=. Alternatively, if you add an extra level of indirection using .DEFAULT to capture and forward make invocations, you can simply pass OS, etc, as invocation arguments. Indirection solves everything, though, so that's cheating. [1] Apple's ancient GNU Make 3.81 predates != support. :(

gregwebs 6y ago

Make was designed for building dependencies. I think it is always problematic to use it as a command runner (for example there is no standard way to list out the available commands). [just](https://github.com/casey/just https://github.com/casey/just) is a tool that feels similar to make but is designed explicitly for the purpose of running commands. I think of this as a simple CLI for your project workflow. You still really want to avoid putting code into a Justfile and put it into scripts. But the Justfile helps provide a slightly nicer UX and automatically invoke dependencies.

oso2k 6y ago
When you say, "there is no standard way to list out the available commands", do you mean like a `make help`? https://gist.github.com/prwhite/8168133#gistcomment-3114855 https://gist.github.com/prwhite/8168133#gistcomment-3114855
- lytedev 6y ago
  While you you could argue that this is a convention -- I'd say this isn't even a common one -- it's still a long way from being "standard".
- jackewiehose 6y ago
  "make help" is definitely not a standard. For all I know "make help" builds help.exe. But there is a standard way to get available commands: Just look at the README or open the Makefile with a text-editor! The lack of a standard argument for getting help doesn't make it problematic for use as a command runner. You can't get atomatically all available commands from a Makefile just like you can't get all command-line flags from an executable. The program/Makefile has to provide it by itself.
chubot 6y ago
Yes I agree with this. I use shell instead of make, because make wrapps shell and its syntax collides very poorly with it. For example, the PID is now $$$$ and not $$. Most people forget to mark their targets .PHONY, so they have a subtle bug in their build (touch build; touch test). ---- But shell also suffers from the problem where it doesn't list the commands. I filed a bug for Oil shell here: https://github.com/oilshell/oil/issues/751 https://github.com/oilshell/oil/issues/751 I mentioned a couple other "frameworks" there like just, go, Taskfile, etc. But it should really just be built into the shell, since it's so common. And there should be command completion too, which I think bash-completion has for Makefiles on many distros. Apparently there is no standard name for this kind of "task runner". But I think shell makes a lot more sense than a custom format, because there are many instances where you need a simple loop or conditional. It scales better. (And Oil also fixes bad shell syntax while remaining compatible: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht...) If anyone wants to help let me know :) The code is plain Python and pretty hackable. However it generates fast C++, so you get the best of both worlds (in progress)

adelarsq 6y ago

I added this to my list about Make at https://github.com/adelarsq/awesome-make https://github.com/adelarsq/awesome-make Pull requests are welcome.

IshKebab 6y ago

I wish someone would write a modern alternative to GNU Make. I've looked and there don't seem to be any. The closest is Ninja but it doesn't seem to be intended to be hand written.

theshrike79 6y ago
There are a lot of options, but make is just everywhere. Sometimes it's just simpler to bite the ancient bullet and go with a Makefile, with all its included pains and gotchas rather than try to figure out how to get the fancy new makefile replacement installed in all the relevant environments.
epistasis 6y ago
A lot of people in bioinformatics use SnakeMake. In this field you often want to restart analysis after something changes somewhere along a pipeline (for example the pipeline is under active development and changing frequently), and individual steps can take hours or more, so automatically rerunning just the right stuff is a great feature. However, SnakeMake, Nextflow, etc feels excessively verbose compared to standard make. And the prior workflow managers of last decades were far worse. With standard make, you type pretty much exactly what you would for shell commands, and not too much more. All other alternatives are going to be more verbose than make, and to me that's a negative.
boris 6y ago
https://build2.org/build2/doc/build2-build-system-manual.xhtml#intro https://build2.org/build2/doc/build2-build-system-manual.xht...
- brandmeyer 6y ago
  > We believe, paraphrasing a famous quote, that those who do not understand make are condemned to reinvent it, poorly. Alas, I think the developers have in fact done just that. Make is not a build system. Make is a batch shell. Build2 didn't produce a better make, they produced yet another C++ build system.

purplezooey 6y ago

loving the python3 makefiles