8 ms·

Mk: A Successor to Make (1987) [pdf]

83 points by SllX 3y ago

vmsp 3y ago
For those curious, Mk is available in Plan 9 from User Space https://9fans.github.io/plan9port/ https://9fans.github.io/plan9port/
- jacobvosmaer 3y ago
  I tried plan9port's mk for a moment out of curiosity. I quickly ran into an annoying usability problem: it compares file mtimes with second accuracy. https://github.com/9fans/plan9port/blob/cc4571fec67407652b03d6603ada6580de2194dc/src/cmd/mk/unix.c#L346 https://github.com/9fans/plan9port/blob/cc4571fec67407652b03... With sub-second build times for individual targets, this causes mk to needlessly recompile files because the target may have the same mtime as the prerequisites.
  - jmrjmr 3y ago
    That looks like it would be relatively straightforward to fix. Attempted nerd snipe?
jylam 3y ago
Funny how building tools are reinvented again and again, with all the same culprits. I'm a programmer since the mid-90's, there are countless tools to do the job, and they are just as cumbersome and complicated the more they evolve. I'm personally sticking to the good old configure/make for my own C projects, but I understand that's a matter of taste and habits. I've written some CMakeLists.txt from time to time, anything you want to do requires some digging in some sort of stackoverflow answers (thanks to those who answer) for "not so common but yet I need them" weird features. I'm not criticizing anyone or any tool, just that we can't seem to find some good way to tool the compilation of both trivial and very complex projects, without taking an hour or two off. I also understand that I do not want to work on such an endeavor, that seems boring AF and kudos for people that like that. You are not enough it seem :)
- mananaysiempre 3y ago
  Plan 9’s mk isn’t really a radical departure from Make, it is more of an adaptation of it to the (decidedly non-POSIX) Plan 9 shell, rc, with modest extensions. One design mistake in make that mk fixes is variables in recipes: those are now passed as environment variables, with no prior substitution, so no more writing $$$$ to get the current PID in a recipe (and it’s written $pid in rc anyway). Unfortunately, mk inherits from rc the principle of having the list of strings as the fundamental datatype (also used by Jam). That works, but it’s noticeably more limiting than Tcl’s route of having everything be strings but with robust quoting and unquoting procedures for putting lists inside them—at which point Tcl starts to look like a Lisp-2 with a slight propensity for stringiness and a mildly unusual syntax.
  - mort96 3y ago
    Ohhh, passing variables as env vars is a good idea. That means things like "$foo" in a recipe just works even if 'foo' contains quotes.
    - throwawaaarrgh 3y ago
      Actually it doesn't really solve the $$foo problem. If your make target has an inline shell script with a variable reference, is it referring to a make variable or a shell variable? Is it set in make? If not, what will make do with that variable reference in the inline shell - nothing? Make it an empty string? Keep the literal '$foo' string? And what if both make and inline script have the variable set but they need to be different?
      - mananaysiempre 3y ago
        It does solve the $$foo problem in that mk just doesn’t do any variable substitution in recipes, at all. For each invocation of a recipe, it adds the values of mk variables it has computed to the parent environment, then execs a shell and passes it the entire recipe verbatim.
  - rollcat 3y ago
TheLocehiliosan 3y ago
An alternative I quite like is “just”. https://github.com/casey/just https://github.com/casey/just
- DanHulton 3y ago
  Was going to mention this. I've recently started converting my Makefiles to justfiles, and it's just nicer. Even being able to inline scripts to clean up "loose" files is a big win.
- packetlost 3y ago
  Just isn't really a replacement for make outside of the "simple command/task runner". Make is really a lot more powerful and has things like files system driven dependency handling.
  - zokier 3y ago
    Of course that is the whole point of just, to be simpler and more focused tool because a task runner is what people commonly want and the bloat/complexity of make is more of hindrance. This is clearly mentioned in the readme too > just has a ton of useful features, and many improvements over make: > • just is a command runner, not a build system, so it avoids much of make's complexity and idiosyncrasies. No need for .PHONY recipes!
- dimator 3y ago
  I've been using 'just' for running a set of commands that I can't be bothered to remember. It makes it easier to come back to a project after a few weeks, because you don't have to remember the N commands you were using to iterate/test, you only have to remember the 'just' invocation. For some reason, it feels like a more natural fit for this than 'make'
raddan 3y ago
Here’s another make alternative. You do not need to specify dependencies at all. https://www.usenix.org/conference/atc22/presentation/curtsinger https://www.usenix.org/conference/atc22/presentation/curtsin... (full disclosure: I am one of the authors)
- imran-iq 3y ago
  There's also tup: https://gittup.org/tup/ https://gittup.org/tup/
  - nmz 3y ago
    From https://gittup.org/tup/getting_started.html https://gittup.org/tup/getting_started.html > Make sure your source files are backed up, like in source control, or something. Tup is able to delete old files automatically, though it tries to prevent you from doing silly things like overwrite your hand-written C files. Still, it would suck if you got boned because tup has a bug or something. Hey, your hard disk can go at anytime, too. This is unacceptable, no thanks, I build first (test) and then commit, not the other way around.
    - vimax 3y ago
      I agree with you about that statement, but why don’t you commit to an unstable branch? Why risk losing something or pass up having a better development history?
      - nemetroid 3y ago
        The way the warning is worded suggests that any invocation of tup has a risk of deleting your files, implying that to be entirely safe you should commit before every single build, which would be ludicrous.
        rollcat 3y ago
        git commit --amend, rebase with squash/fixup, etc. I easily make a dozen commits per hour, it's like undo on steroids and you can always clean it up later.
ripe 3y ago
I like simple Makefiles. Keeping them POSIX compliant makes them portable, including on cygwin-type environments on Windows. I used this tutorial: https://nullprogram.com/blog/2017/08/20/ https://nullprogram.com/blog/2017/08/20/
throwawaaarrgh 3y ago
It seems to me that all build systems are just DAGs with syntactic sugar and functions. You could do away with build systems entirely if there were some Unix commands that manage an arbitrary DAG, where the state is kept out of band (in a file, in a database, etc) so you can operate on the DAG from any process. That way the shell (or any program, really) becomes your build system and you can compose any kind of logic that requires walking or manipulating a tree of dependencies. This could apply to anything where you need to execute arbitrary jobs with a DAG, not just builds.
- chriswarbo 3y ago
  Nix represents its builds in a way that's similar: each "derivation" is a text file like /nix/store/XXXXXXX-foo.drv, where XXXXXX is a hash of that file's content. This way each file can reference any others by their path, and there can never be cycles (unless we brute-forced SHA256 to find a pair of files which contain each others hashes!). This uses of hashing requires the files to be immutable, but that's good for caching/validation/reuse/etc. anyway. Note that Nix doesn't use a shell to execute things, it uses raw `exec` calls (each .drv file specifies the absolute path to an executable, a list of argument strings, and a set of environment variable strings). Though in practice, most .drv files specify a bash executable ;)
  - klysm 3y ago
    I didn’t realize that content addressing girls like that prevented circular deps by construction (with very high probability). That’s a super cool property to get as a bonus
- ur-whale 3y ago
  > It seems to me that all build systems are just DAGs Yeah, well, it's a little bit more than that. Two things come to mind that don't neatly fit the DAG mental framework: - dynamically generated dependencies (e.g. when you compile a C++ file only to discover that it #includes something and therefore has a dependency on that thing, and therefore the DAG has to be updated on the fly). Creating them by hand is horribly tedious, and/or borderline impossible (#includes that #include other #include ad infinitum) - reproducible builds, where a build system is capable of rebuilding a binary from scratch down to having not a single different bit in the final output assuming the leaves of the DAG haven't changed. A desirable feature that is darn near impossible to do unless you pair the DAG with something else.
adamgordonbell 3y ago
Here is my silly alternative. You just write bash, with preconditions and annotations to describe what can run in parallel. https://github.com/adamgordonbell/job-runner/blob/main/tests/test1.job https://github.com/adamgordonbell/job-runner/blob/main/tests...
gigatexal 3y ago
I like the simplified syntax of this mk, anyone know why it didn't catch on?
- nsajko 3y ago
  Research Unix wasn't easily available, AFAIK Bell Labs were trying to sell it as a product, so it lost to more free alternatives. It only got open-sourced recently AFAIK, years after being abandoned.
  - ggm 3y ago
    Also, and with no implied criticism, the people involved set the bar higher to capture their mindshare. I know, because I failed the bar (amicably I might add) Which meant they cared less about "winning" in some non-valuable game of "whose code is winning" and had less irritating conversations, "no thats not how it works, no I don't have to explain this idea called the plumber to you, yes rc is very different to other shells no thats not how it works"... ad infinitum.
vladxyz 3y ago
To add to the pile of task runners here, I made this one: https://code.ofvlad.xyz/v/lightning-runner https://code.ofvlad.xyz/v/lightning-runner . I like it, and currently use it for all personal projects. Mainly, posting here for some feedback!
- deleted 3y ago
  [deleted]
whiteinge 3y ago
I own a couple books on Make and it's a great tool, but I learned how to use Mk by just reading the manpage. It's a huge improvement and simplification at the same time. There's a solid, stand-alone implementation of mk in golang. No plan9 environment needed. https://github.com/henesy/mk https://github.com/henesy/mk

83 points by SllX 3y ago

vmsp 3y ago

For those curious, Mk is available in Plan 9 from User Space https://9fans.github.io/plan9port/ https://9fans.github.io/plan9port/

jacobvosmaer 3y ago
I tried plan9port's mk for a moment out of curiosity. I quickly ran into an annoying usability problem: it compares file mtimes with second accuracy. https://github.com/9fans/plan9port/blob/cc4571fec67407652b03d6603ada6580de2194dc/src/cmd/mk/unix.c#L346 https://github.com/9fans/plan9port/blob/cc4571fec67407652b03... With sub-second build times for individual targets, this causes mk to needlessly recompile files because the target may have the same mtime as the prerequisites.
- jmrjmr 3y ago
  That looks like it would be relatively straightforward to fix. Attempted nerd snipe?

jylam 3y ago

Funny how building tools are reinvented again and again, with all the same culprits. I'm a programmer since the mid-90's, there are countless tools to do the job, and they are just as cumbersome and complicated the more they evolve. I'm personally sticking to the good old configure/make for my own C projects, but I understand that's a matter of taste and habits. I've written some CMakeLists.txt from time to time, anything you want to do requires some digging in some sort of stackoverflow answers (thanks to those who answer) for "not so common but yet I need them" weird features. I'm not criticizing anyone or any tool, just that we can't seem to find some good way to tool the compilation of both trivial and very complex projects, without taking an hour or two off. I also understand that I do not want to work on such an endeavor, that seems boring AF and kudos for people that like that. You are not enough it seem :)

mananaysiempre 3y ago
Plan 9’s mk isn’t really a radical departure from Make, it is more of an adaptation of it to the (decidedly non-POSIX) Plan 9 shell, rc, with modest extensions. One design mistake in make that mk fixes is variables in recipes: those are now passed as environment variables, with no prior substitution, so no more writing $$$$ to get the current PID in a recipe (and it’s written $pid in rc anyway). Unfortunately, mk inherits from rc the principle of having the list of strings as the fundamental datatype (also used by Jam). That works, but it’s noticeably more limiting than Tcl’s route of having everything be strings but with robust quoting and unquoting procedures for putting lists inside them—at which point Tcl starts to look like a Lisp-2 with a slight propensity for stringiness and a mildly unusual syntax.
- mort96 3y ago
  Ohhh, passing variables as env vars is a good idea. That means things like "$foo" in a recipe just works even if 'foo' contains quotes.
  - throwawaaarrgh 3y ago
    Actually it doesn't really solve the $$foo problem. If your make target has an inline shell script with a variable reference, is it referring to a make variable or a shell variable? Is it set in make? If not, what will make do with that variable reference in the inline shell - nothing? Make it an empty string? Keep the literal '$foo' string? And what if both make and inline script have the variable set but they need to be different?
    - mananaysiempre 3y ago
      It does solve the $$foo problem in that mk just doesn’t do any variable substitution in recipes, at all. For each invocation of a recipe, it adds the values of mk variables it has computed to the parent environment, then execs a shell and passes it the entire recipe verbatim.
- rollcat 3y ago

TheLocehiliosan 3y ago

An alternative I quite like is “just”. https://github.com/casey/just https://github.com/casey/just

DanHulton 3y ago
Was going to mention this. I've recently started converting my Makefiles to justfiles, and it's just nicer. Even being able to inline scripts to clean up "loose" files is a big win.
packetlost 3y ago
Just isn't really a replacement for make outside of the "simple command/task runner". Make is really a lot more powerful and has things like files system driven dependency handling.
- zokier 3y ago
  Of course that is the whole point of just, to be simpler and more focused tool because a task runner is what people commonly want and the bloat/complexity of make is more of hindrance. This is clearly mentioned in the readme too > just has a ton of useful features, and many improvements over make: > • just is a command runner, not a build system, so it avoids much of make's complexity and idiosyncrasies. No need for .PHONY recipes!
dimator 3y ago
I've been using 'just' for running a set of commands that I can't be bothered to remember. It makes it easier to come back to a project after a few weeks, because you don't have to remember the N commands you were using to iterate/test, you only have to remember the 'just' invocation. For some reason, it feels like a more natural fit for this than 'make'

raddan 3y ago

Here’s another make alternative. You do not need to specify dependencies at all. https://www.usenix.org/conference/atc22/presentation/curtsinger https://www.usenix.org/conference/atc22/presentation/curtsin... (full disclosure: I am one of the authors)

imran-iq 3y ago
There's also tup: https://gittup.org/tup/ https://gittup.org/tup/
- nmz 3y ago
  From https://gittup.org/tup/getting_started.html https://gittup.org/tup/getting_started.html > Make sure your source files are backed up, like in source control, or something. Tup is able to delete old files automatically, though it tries to prevent you from doing silly things like overwrite your hand-written C files. Still, it would suck if you got boned because tup has a bug or something. Hey, your hard disk can go at anytime, too. This is unacceptable, no thanks, I build first (test) and then commit, not the other way around.
  - vimax 3y ago
    I agree with you about that statement, but why don’t you commit to an unstable branch? Why risk losing something or pass up having a better development history?
    - nemetroid 3y ago
      The way the warning is worded suggests that any invocation of tup has a risk of deleting your files, implying that to be entirely safe you should commit before every single build, which would be ludicrous.
      - rollcat 3y ago
        git commit --amend, rebase with squash/fixup, etc. I easily make a dozen commits per hour, it's like undo on steroids and you can always clean it up later.

ripe 3y ago

I like simple Makefiles. Keeping them POSIX compliant makes them portable, including on cygwin-type environments on Windows. I used this tutorial: https://nullprogram.com/blog/2017/08/20/ https://nullprogram.com/blog/2017/08/20/

throwawaaarrgh 3y ago

It seems to me that all build systems are just DAGs with syntactic sugar and functions. You could do away with build systems entirely if there were some Unix commands that manage an arbitrary DAG, where the state is kept out of band (in a file, in a database, etc) so you can operate on the DAG from any process. That way the shell (or any program, really) becomes your build system and you can compose any kind of logic that requires walking or manipulating a tree of dependencies. This could apply to anything where you need to execute arbitrary jobs with a DAG, not just builds.

chriswarbo 3y ago
Nix represents its builds in a way that's similar: each "derivation" is a text file like /nix/store/XXXXXXX-foo.drv, where XXXXXX is a hash of that file's content. This way each file can reference any others by their path, and there can never be cycles (unless we brute-forced SHA256 to find a pair of files which contain each others hashes!). This uses of hashing requires the files to be immutable, but that's good for caching/validation/reuse/etc. anyway. Note that Nix doesn't use a shell to execute things, it uses raw `exec` calls (each .drv file specifies the absolute path to an executable, a list of argument strings, and a set of environment variable strings). Though in practice, most .drv files specify a bash executable ;)
- klysm 3y ago
  I didn’t realize that content addressing girls like that prevented circular deps by construction (with very high probability). That’s a super cool property to get as a bonus
ur-whale 3y ago
> It seems to me that all build systems are just DAGs Yeah, well, it's a little bit more than that. Two things come to mind that don't neatly fit the DAG mental framework: - dynamically generated dependencies (e.g. when you compile a C++ file only to discover that it #includes something and therefore has a dependency on that thing, and therefore the DAG has to be updated on the fly). Creating them by hand is horribly tedious, and/or borderline impossible (#includes that #include other #include ad infinitum) - reproducible builds, where a build system is capable of rebuilding a binary from scratch down to having not a single different bit in the final output assuming the leaves of the DAG haven't changed. A desirable feature that is darn near impossible to do unless you pair the DAG with something else.

adamgordonbell 3y ago

Here is my silly alternative. You just write bash, with preconditions and annotations to describe what can run in parallel. https://github.com/adamgordonbell/job-runner/blob/main/tests/test1.job https://github.com/adamgordonbell/job-runner/blob/main/tests...

gigatexal 3y ago

I like the simplified syntax of this mk, anyone know why it didn't catch on?

nsajko 3y ago
Research Unix wasn't easily available, AFAIK Bell Labs were trying to sell it as a product, so it lost to more free alternatives. It only got open-sourced recently AFAIK, years after being abandoned.
- ggm 3y ago
  Also, and with no implied criticism, the people involved set the bar higher to capture their mindshare. I know, because I failed the bar (amicably I might add) Which meant they cared less about "winning" in some non-valuable game of "whose code is winning" and had less irritating conversations, "no thats not how it works, no I don't have to explain this idea called the plumber to you, yes rc is very different to other shells no thats not how it works"... ad infinitum.

vladxyz 3y ago

To add to the pile of task runners here, I made this one: https://code.ofvlad.xyz/v/lightning-runner https://code.ofvlad.xyz/v/lightning-runner . I like it, and currently use it for all personal projects. Mainly, posting here for some feedback!

deleted 3y ago
[deleted]

whiteinge 3y ago

I own a couple books on Make and it's a great tool, but I learned how to use Mk by just reading the manpage. It's a huge improvement and simplification at the same time. There's a solid, stand-alone implementation of mk in golang. No plan9 environment needed. https://github.com/henesy/mk https://github.com/henesy/mk