6 ms·

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of the

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological execution engine (did you know Make can coordinate job slots across recursive executions of Make? It's impressive! And shouldn't have to exist!) [1] All the alternative build systems wisely choose to tackle only some of these features. Redo is interesting in that it's a minimal system that leans on existing Unix tools to fill the gaps, but I can't say I'm impressed with using Shell as the scripting language, though it is arguably more useful than Make's home-grown language that too few people know. Edit: actually, redo's doesn't enforce Shell, it can be any executable. That's more interesting, maybe (flexibility also introduces complexity!) The most interesting development in this space is the paper "Build Systems A La Carte" [2] by Mokhov, Mitchell, & Peyton Jones, who did a great job of breaking down build systems to their fundamental concepts, and along the way found one particular point of the design space that wasn't fulfilled (Table 2, p79:16). Unfortunately, I fear it'll be some years before something production-ready emerges from that research, (and I'm sure I'll be grumpy about some of their implementation details anyhow! Build systems do not a happy developer make) [1] https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... [2] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

AceJohnny2 5y ago
I see GNU Make as sitting on the opposite end of a spectrum than Git. Git is fascinating as a sophisticated, featureful system emerging from a simple fundamental concept (Merkle tree of file hashes), whereas Make is a mish-mash of concepts cobbled together to expose a particular set of user features (describe and execute a graph of executions in a factorized way). Git emerged decades after other version control systems were first invented, and found a neat abstraction that covered the space. That's why I have such hope for the BSalC paper and what can emerge from it.
qbasic_forever 5y ago
Have you given bazel a serious look? It hits quite a few of the points you mention. There's a steep learning curve and you kind of have to go all in with it, but once you're there it's quite nice.
- dataflow 5y ago
  Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so I can distribute it. And with Python, etc. you hit more obstacles. Make degrades more "gradually" in a sense: it's easy to do arbitrary things in a 1-line recipe, but then it becomes harder to generalize and make it modular. Overall, Bazel definitely wins as the scale of your project grows, and it has tons of useful features, but it's got some ways to go before I'd regard it as "nice".
  - kylecordes 5y ago
    For the sake of readers: Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.
    - joshuamorton 5y ago
      (also depending on what you're doing, `bazel run <foo>` will invoke the binary directly, and `bazel build <foo>` will, if there's an output artifact, usually print its location)
    - dataflow 5y ago
      I meant random in the colloquial sense (same sense as which a SHA256 is "random"), not a random variable.
lolski 5y ago
I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive. What is steep is the technical know-hows: 1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python. 2. When you have to do something advanced such as building custom Bazel rule. You'll need to understand Bazel internals and unfortunately the documentation isn't very intuitive and also somewhat sparse.
- packetlost 5y ago
  Bazel is awful unless you're Google and have a team of devs supporting Bazel.
  - throwaway894345 5y ago
    I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it. If you’re just building Go binaries then I’m sure Bazel is great, but if you’re doing anything with Python 3 or if you’re doing something a little off the beaten path like code generation, it’s probably a frustrating experience.
    - xyzzy_plugh 5y ago
      > If you’re just building Go binaries then I’m sure Bazel is great Compared to other things in Bazel maybe, sure. But if you're just building Go binaries you don't need Bazel. The standard Go tooling is beyond sufficient: it's hard to beat.
      - throwaway894345 5y ago
        Yeah, I completely agree.
    - ameliaquining 5y ago
      Lots of things can be complicated in Bazel, but I would not at all consider code generation to be among them. You just write a genrule, and wrap it in a macro if you want to do the same thing more than once. (It can be more complicated than this, of course, but then code generation isn't the hard part of the problem.)
einpoklum 5y ago
> All the alternative build systems wisely choose to tackle only some of these features. But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene? More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make as superior to just-Make?
- AceJohnny2 5y ago
  That was exactly my point: perhaps it's best not to bundle all those features in a single piece of software. I haven't tried all the alternative combinations because, ultimately, I only have so much time and interest, and their learning curve is significant.
woodruffw 5y ago
Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.
- webstrand 5y ago
  I really like the declarative style of make, to the extent that I've been abusing it as an automation tool. Doing things like checking if a server is alive or setting up a serial port. But due to limitations like depending on mtime and issues with special characters in targets, I've switched some of my more intensive Makefiles to prolog using this: https://github.com/webstrand/robo https://github.com/webstrand/robo. When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.
- deleted 5y ago
  [deleted]
- ljm 5y ago
  It's funny you mention that because I've wondered for a while what it would be like to have a Prolog-based build system (or even just as a CI config target instead of abusing yaml). I just lack the technical nous (and time) to try and pull it off.
codedokode 5y ago
I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.
- fragmede 5y ago
  GNUMake allows for custom, non-tab leader characters. https://stackoverflow.com/questions/2131213/can-you-make-valid-makefiles-without-tab-characters/21920142#21920142 https://stackoverflow.com/questions/2131213/can-you-make-val...
- sigjuice 5y ago
  make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step. $ cat Makefile CFLAGS := -MD -MP hello: hello.o x.o y.o -include *.d $ echo '#include "x.h"' > x.c $ echo '#include "y.h"' > y.c $ touch x.h y.h $ echo 'int main(void) {}' > hello.c $ make cc -MD -MP -c -o hello.o hello.c cc -MD -MP -c -o x.o x.c cc -MD -MP -c -o y.o y.c cc hello.o x.o y.o -o hello $ touch x.h $ make cc -MD -MP -c -o x.o x.c cc hello.o x.o y.o -o hello
  - JdeBP 5y ago
    Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example. * http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDeficiencies http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...
    - glandium 5y ago
      Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.
thechao 5y ago
I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++): 1. I convert all paths to abspaths; 2. All intermediate products go into a temp dir; 3. All recipes are functions; 4. The set of objects are defined by a “find . -name ‘…’” for C sources; 5. Every C file depends on every header file in the repo; 6. Use ccache; and, 7. Deterministic build. My big work project is ~20mm loc, including ~65 FW images (embedded OSes). Incremental compilation is ~300ms. Clean rebuilds are 30-40s; scratch builds are 8m. (All of this on ~4 year old MBP.) Obviously, there are some design tricks to get raw compilation speed up — naive C++ only compiles at ~3kloc/s; it’s possible to get that up to 100kloc/s.
- Too 5y ago
  > Every C file depends on every header file in the repo. So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++? What's the difference between a clean rebuild and a scratch build?
  - thechao 5y ago
    Yes: touching any header file causes a full rebuild. Remember that ccache is "under the hood" doing memoization. So, a clean rebuild could still hit ccache, whereas a scratch build starts from an emoty ccache.
Arch-TK 5y ago
I think not adding an arbitrary new declarative language is good design feature of redo. I think redo has other issues but personally redo with bash as the underlying glue works a lot better than make (in terms of maintaining more complete DAGs). Now I'm not saying it wouldn't be nice to get a declarative language which works WITH redo, but it would be important to keep it separate. One thing redo is missing for me is support for building things in a different directory. It's a nice-to-have feature but it's difficult to envision how such a redo would work. Really overlays actually solve this problem, although a fuse based overlayfs would also be nice (so you can use it without needing to escalate privileges).
agumonkey 5y ago
I wonder if the reactive trend will hit make-like tools.
eddieh 5y ago
As I have been saying for years: make is the worst build system, except for all the others.
tn1 5y ago
Something a little better than make is makepp [1], although it can be more complex. We use it at $work and it works pretty well. [1] http://makepp.sourceforge.net/ http://makepp.sourceforge.net/
KarlKemp 5y ago
I’m quite happy with ruby’s rake for my needs. I mostly use it to define pipelines, often involving data downloads and transformations into datasets for ML. It works well when you work with files, preferably containing a single or few entities each. Have been meaning to wrap some tools to make it handle databases into an extension.
hv42 5y ago
Gradle is a good make replacement as well. The kotlin DSL is much more readable and easier to follow than makefiles and has a lot of additional features.
- grandinj 5y ago
  Except when stuff goes wrong. Then gradle is a right PITA because debuggability is essentially zero. And periodically a new version of gradle will break something non-obvious deep in your dependency tree.
1vuio0pswjnm7 5y ago
"As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement." Unless I am mistaken, the author of https://cr.yp.to/redo.html https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to https://nacl.cr.yp.to The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989. For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.

AceJohnny2 5y ago

I see GNU Make as sitting on the opposite end of a spectrum than Git. Git is fascinating as a sophisticated, featureful system emerging from a simple fundamental concept (Merkle tree of file hashes), whereas Make is a mish-mash of concepts cobbled together to expose a particular set of user features (describe and execute a graph of executions in a factorized way). Git emerged decades after other version control systems were first invented, and found a neat abstraction that covered the space. That's why I have such hope for the BSalC paper and what can emerge from it.

qbasic_forever 5y ago

Have you given bazel a serious look? It hits quite a few of the points you mention. There's a steep learning curve and you kind of have to go all in with it, but once you're there it's quite nice.

dataflow 5y ago
Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so I can distribute it. And with Python, etc. you hit more obstacles. Make degrades more "gradually" in a sense: it's easy to do arbitrary things in a 1-line recipe, but then it becomes harder to generalize and make it modular. Overall, Bazel definitely wins as the scale of your project grows, and it has tons of useful features, but it's got some ways to go before I'd regard it as "nice".
- kylecordes 5y ago
  For the sake of readers: Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.
  - joshuamorton 5y ago
    (also depending on what you're doing, `bazel run <foo>` will invoke the binary directly, and `bazel build <foo>` will, if there's an output artifact, usually print its location)
  - dataflow 5y ago
    I meant random in the colloquial sense (same sense as which a SHA256 is "random"), not a random variable.

lolski 5y ago

I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive. What is steep is the technical know-hows: 1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python. 2. When you have to do something advanced such as building custom Bazel rule. You'll need to understand Bazel internals and unfortunately the documentation isn't very intuitive and also somewhat sparse.

packetlost 5y ago
Bazel is awful unless you're Google and have a team of devs supporting Bazel.
- throwaway894345 5y ago
  I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it. If you’re just building Go binaries then I’m sure Bazel is great, but if you’re doing anything with Python 3 or if you’re doing something a little off the beaten path like code generation, it’s probably a frustrating experience.
  - xyzzy_plugh 5y ago
    > If you’re just building Go binaries then I’m sure Bazel is great Compared to other things in Bazel maybe, sure. But if you're just building Go binaries you don't need Bazel. The standard Go tooling is beyond sufficient: it's hard to beat.
    - throwaway894345 5y ago
      Yeah, I completely agree.
  - ameliaquining 5y ago
    Lots of things can be complicated in Bazel, but I would not at all consider code generation to be among them. You just write a genrule, and wrap it in a macro if you want to do the same thing more than once. (It can be more complicated than this, of course, but then code generation isn't the hard part of the problem.)

einpoklum 5y ago

> All the alternative build systems wisely choose to tackle only some of these features. But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene? More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make as superior to just-Make?

AceJohnny2 5y ago
That was exactly my point: perhaps it's best not to bundle all those features in a single piece of software. I haven't tried all the alternative combinations because, ultimately, I only have so much time and interest, and their learning curve is significant.

woodruffw 5y ago

Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.

webstrand 5y ago
I really like the declarative style of make, to the extent that I've been abusing it as an automation tool. Doing things like checking if a server is alive or setting up a serial port. But due to limitations like depending on mtime and issues with special characters in targets, I've switched some of my more intensive Makefiles to prolog using this: https://github.com/webstrand/robo https://github.com/webstrand/robo. When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.
deleted 5y ago
[deleted]
ljm 5y ago
It's funny you mention that because I've wondered for a while what it would be like to have a Prolog-based build system (or even just as a CI config target instead of abusing yaml). I just lack the technical nous (and time) to try and pull it off.

codedokode 5y ago

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

fragmede 5y ago
GNUMake allows for custom, non-tab leader characters. https://stackoverflow.com/questions/2131213/can-you-make-valid-makefiles-without-tab-characters/21920142#21920142 https://stackoverflow.com/questions/2131213/can-you-make-val...
sigjuice 5y ago
make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step. $ cat Makefile CFLAGS := -MD -MP hello: hello.o x.o y.o -include *.d $ echo '#include "x.h"' > x.c $ echo '#include "y.h"' > y.c $ touch x.h y.h $ echo 'int main(void) {}' > hello.c $ make cc -MD -MP -c -o hello.o hello.c cc -MD -MP -c -o x.o x.c cc -MD -MP -c -o y.o y.c cc hello.o x.o y.o -o hello $ touch x.h $ make cc -MD -MP -c -o x.o x.c cc hello.o x.o y.o -o hello
- JdeBP 5y ago
  Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example. * http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDeficiencies http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...
  - glandium 5y ago
    Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.

thechao 5y ago

I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++): 1. I convert all paths to abspaths; 2. All intermediate products go into a temp dir; 3. All recipes are functions; 4. The set of objects are defined by a “find . -name ‘…’” for C sources; 5. Every C file depends on every header file in the repo; 6. Use ccache; and, 7. Deterministic build. My big work project is ~20mm loc, including ~65 FW images (embedded OSes). Incremental compilation is ~300ms. Clean rebuilds are 30-40s; scratch builds are 8m. (All of this on ~4 year old MBP.) Obviously, there are some design tricks to get raw compilation speed up — naive C++ only compiles at ~3kloc/s; it’s possible to get that up to 100kloc/s.

Too 5y ago
> Every C file depends on every header file in the repo. So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++? What's the difference between a clean rebuild and a scratch build?
- thechao 5y ago
  Yes: touching any header file causes a full rebuild. Remember that ccache is "under the hood" doing memoization. So, a clean rebuild could still hit ccache, whereas a scratch build starts from an emoty ccache.

Arch-TK 5y ago

I think not adding an arbitrary new declarative language is good design feature of redo. I think redo has other issues but personally redo with bash as the underlying glue works a lot better than make (in terms of maintaining more complete DAGs). Now I'm not saying it wouldn't be nice to get a declarative language which works WITH redo, but it would be important to keep it separate. One thing redo is missing for me is support for building things in a different directory. It's a nice-to-have feature but it's difficult to envision how such a redo would work. Really overlays actually solve this problem, although a fuse based overlayfs would also be nice (so you can use it without needing to escalate privileges).

agumonkey 5y ago

I wonder if the reactive trend will hit make-like tools.

eddieh 5y ago

As I have been saying for years: make is the worst build system, except for all the others.

tn1 5y ago

Something a little better than make is makepp [1], although it can be more complex. We use it at $work and it works pretty well. [1] http://makepp.sourceforge.net/ http://makepp.sourceforge.net/

KarlKemp 5y ago

I’m quite happy with ruby’s rake for my needs. I mostly use it to define pipelines, often involving data downloads and transformations into datasets for ML. It works well when you work with files, preferably containing a single or few entities each. Have been meaning to wrap some tools to make it handle databases into an extension.

hv42 5y ago

Gradle is a good make replacement as well. The kotlin DSL is much more readable and easier to follow than makefiles and has a lot of additional features.

grandinj 5y ago
Except when stuff goes wrong. Then gradle is a right PITA because debuggability is essentially zero. And periodically a new version of gradle will break something non-obvious deep in your dependency tree.

1vuio0pswjnm7 5y ago

"As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement." Unless I am mistaken, the author of https://cr.yp.to/redo.html https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to https://nacl.cr.yp.to The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989. For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.