4 ms·

> Perhaps the best known example of this kind of tool is Make. At its core it is only a DAG solver and you can have arbitrary functionality just by writing shel

by jnxx 6y ago

> Perhaps the best known example of this kind of tool is Make. At its core it is only a DAG solver and you can have arbitrary functionality just by writing shell script fragments directly inside your Makefile. I am still fascinated by redo (https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/), which provides exactly the same purpose as Make, but turns its interface inside out: Make is a DSL for describing an acyclic dependency graph, with attached command lines to compile, link, install or run stuff, with a minimum amount operations. The fact that Make is not language-specific, and one just uses shell commands to build stuff, makes it versatile. But there are edge cases which have the effect that people often to prefer to do "make clean", because dependencies might not be captured completely. "redo" is internally more complex, but basically it is a set of just ten shell commands which are used declarative to capture dependencies, and are part of build scripts. For example, #!/bin/sh redo-ifchange /usr/local/include/stdio.h redo-ifchange helloworld.c gcc -I/usr/local/include helloworld.c -o $3 has the effect to rebuild a program if the file "/usr/local/include/stdio.h", or the source file "helloworld.c" was changed - be it by the programmer or by a system update. That's it. No "clean" command is needed. The result is, in terms of user interface and usage, fascinatingly simple (I tried it with a ~ 20,000 lines C++ project which generated a Python extension module using boost.) The lack of special syntax needed is just astonishing. But I wonder how well it would run with all the additional complexities like autoconf and autotools - knowing that the all complexity of these tools is usually there for a reason.

cjfd 6y ago
One thing that one usually does with make is that one also figures out on what header file helloworld.c depends during the make run. Is the above code fragment hand coded or generated? I also am wondering if this also works if one only wants to build part of the project. make can take a list of files to rebuild as arguments. This also provides the ability to only create some set on intermediate files.
- jnxx 6y ago
  You could read here: https://redo.readthedocs.io/en/latest/#whats-so-special-about-redo https://redo.readthedocs.io/en/latest/#whats-so-special-abou... (I think this explanation is much better than what I can do) And yes, it can build specific intermediate files or targets.
- jnxx 6y ago
  > Is the above code fragment hand coded or generated? It is a handwritten example, there is no code generation involved. However, standard rules can be defined by patterns for specific file extensions. Basically how redo works is that it runs the builds scripts to generate a target for the first time (these are shell scripts ending with *.do), and along the way it records every dependency for each target in an sqlite database, using a very small number of shell commands to define them in a declarative way. To re-built, it then uses this database of dependencies. Because it knows all the dependencies, and because the build product is always $3 and then mv'd to the target, it can build everything in parallel by default.
Nullabillity 6y ago
If you trust yourself to capture the dependencies totally then it doesn't really matter whether you use make or redo, you could add the stdio.h dependency to either. But the only way to actually achieve reliable clean-less builds is to run it in an environment where you either take away all access to non-dependencies (like Nix[0]) or automatically record all accessed files (like Tup[1]). GCC also supports emitting header lists in a format that Make understands, but that won't cover non-GCC targets, or be as comprehensive as doing it in the build system. [0]: https://nixos.org/ https://nixos.org/ [1]: http://gittup.org/tup/ http://gittup.org/tup/
- jnxx 6y ago
  > But the only way to actually achieve reliable clean-less builds is to run it in an environment where you either take away all access to non-dependencies (like Nix[0]) or automatically record all accessed files (like Tup[1]). You can also let the compiler record dependencies. Gcc, for example, does have an option for that. That's explained here: https://redo.readthedocs.io/en/latest/#whats-so-special-about-redo https://redo.readthedocs.io/en/latest/#whats-so-special-abou...
- jnxx 6y ago
  To mention, I think Nix or Guix are indeed good complements to things like redo! Guix System and NixOS are just geared to define the whole system, while make or redo build a single program or a set of artifacts.
- jlokier 6y ago
  GCC actually gets it wrong, as does almost every compiler and makefile combination. Consider this realistic example. Realistic in that I've encountered it in the real world and seen buggy demos shipped because of it, with non-reproducible bugs. We have a search path for headers, like so: -Imyinc -I../dep1/inc -Idep2/inc -I/usr/include We compile a source file which contains: #include "creative.h" GCC outputs a dependency which is read into the Makefile next time: object.o: ../dep2/inc/creative.h Looks good! Later that day we update from our upstream dependencies: git pull Which adds a file ../dep1/inc/creative.h Recompiling, the project works: make ... make test => 144,123 pass, 0 fail - well done, you earned a coffee! We captured automatic dependencies, this should be good to ship. But no: make clean ... make ... make test => FAIL! What happened? Turns out someone upstream moved (by copying) creative.h to ../dep1/inc and forgot to remove the old copy in ../dep2/inc. Then someone edited a data structure in that file. Automatic dependencies gave everyone false confidence in the build results seen by different people. That kind of automatic dependency is insufficient because it doesn't capture "negative file results" during path searches, where output depends on a file not existing, and changes if a new file is created. It is possible to use Makefiles with this handled accurately in the automatic dependency tracking, but I haven't seen it done often. An equivalent problem occurs in caches which automatically track data dependencies used to generate results. For example web pages generated from a combination of files, templates and data, build artifacts such as container/VM images, or at a much lower level, data dependencies inside calculations with conditional branches. To do caching right, they must validate negative tests as well as positive that arise in searches. I've seen a surprising number of cache implementations which don't try particularly hard to get this right.
aequitas 6y ago
I (ab)use Make for anything but compiling C sources to binaries. Eg. all my Python projects use Make to conditionally create virtualenv's, install packages, run code linting/tests. For which Make is just the most ubiquitous and simple tool available. But one thing that still annoys me with Make are the workarounds needed to work with stuff which state can not be derived from a file (or only from complex paths). Things like having a background process running (eg. a test db) or a external package that needs to be installed (which can be easily queries using apt, but is much harder to determine reliably using system files). I end up having to create intermediate hidden files which sometimes decouple from the real state. Then things really become messy with Make for me. Would redo be a better solution in this case as well?
- jnxx 6y ago
  > But one thing that still annoys me with Make are the workarounds needed to work with stuff which state can not be derived from a file I am not sure about that one. I think the reason is that make is not made for general scripting, but it is made for generating a deterministic build product, with minimum time, from a deterministic input (the source files). In that sense, it is something like a "pure function" in functional programming. If you build software, you do not want to depend the result today on your CPU fan's speed, and tomorrow on the moon's phase. In other words, you really try to make the result deterministic. What you certainly can do is to first generate a kind of task file from the current system state (e.g., needs to re-create a test db), and letting redo build these targets. For installing / updating stuff, I do not see problems, but I believe package managers such as guix or nix or pip have more specialized options for this.
dllthomas 6y ago
> No "clean" command is needed. I don't see how this follows from what you've said. IME, `clean` is for when I want things rebuilt despite "nothing" having changed. Maybe it's actually the case that nothing's changed, and I want a full build because I want to know how long my build takes. Or maybe I've changed implicit dependencies - compiler version or similar. Could you elaborate?
- jnxx 6y ago
  > IME, `clean` is for when I want things rebuilt despite "nothing" having changed. Of course, you can use clean in this case. You can do that also with redo, just delete all the intermediate files and start a normal build. However the point is that you do a "make clean" most often to be sure that all dependencies are refreshed, because you are actually not sure that your Makefile captures them all. A good example are builds in yocto. This is a system to build complete embedded Linux images based on make, with many subprojects. But if you change, for example, the kernel to support large files, or a 32 bit time_t, you can't be sure that a simple "make" delivers a correct system, because the dependencies on the system headers are not include in all the makefiles.
  - danaris 6y ago
    Please forgive the confusion, as I've only dabbled in Make, but... > You can do that also with redo, just delete all the intermediate files and start a normal build. ...isn't that pretty much what "make clean" generally does? I mean, in a project of any significant size, "all the intermediate files" aren't going to be trivial to delete without a pre-built "delete all the intermediate files" script...which is what "make clean" is (at least partly) for. Or are you just saying that "redo" (which I'd never heard of before, so, again, please forgive my ignorance) can also process a pre-built list of "intermediate files" to delete with some particular command or option...?
    - dllthomas 6y ago
      If I understand, the real point was that clean is less often needed. Having said that, you could certainly list system headers in a Makefile (... hopefully a generated one) if that's the behavior you want, so I am not sure what difference is actually being pointed to.
yjftsjthsd-h 6y ago
I don't see how that's any different. You've made your dependency on stdio.h explicit, so the build tool knows about it - and if you'd done the same thing for make, it would've done the same thing. helloworld: helloworld.c /usr/local/include/stdio.h gcc -I/usr/local/include helloworld.c -o helloworld Tada.

by jnxx 6y ago

cjfd 6y ago

One thing that one usually does with make is that one also figures out on what header file helloworld.c depends during the make run. Is the above code fragment hand coded or generated? I also am wondering if this also works if one only wants to build part of the project. make can take a list of files to rebuild as arguments. This also provides the ability to only create some set on intermediate files.

jnxx 6y ago
You could read here: https://redo.readthedocs.io/en/latest/#whats-so-special-about-redo https://redo.readthedocs.io/en/latest/#whats-so-special-abou... (I think this explanation is much better than what I can do) And yes, it can build specific intermediate files or targets.
jnxx 6y ago
> Is the above code fragment hand coded or generated? It is a handwritten example, there is no code generation involved. However, standard rules can be defined by patterns for specific file extensions. Basically how redo works is that it runs the builds scripts to generate a target for the first time (these are shell scripts ending with *.do), and along the way it records every dependency for each target in an sqlite database, using a very small number of shell commands to define them in a declarative way. To re-built, it then uses this database of dependencies. Because it knows all the dependencies, and because the build product is always $3 and then mv'd to the target, it can build everything in parallel by default.

Nullabillity 6y ago

If you trust yourself to capture the dependencies totally then it doesn't really matter whether you use make or redo, you could add the stdio.h dependency to either. But the only way to actually achieve reliable clean-less builds is to run it in an environment where you either take away all access to non-dependencies (like Nix[0]) or automatically record all accessed files (like Tup[1]). GCC also supports emitting header lists in a format that Make understands, but that won't cover non-GCC targets, or be as comprehensive as doing it in the build system. [0]: https://nixos.org/ https://nixos.org/ [1]: http://gittup.org/tup/ http://gittup.org/tup/

jnxx 6y ago
> But the only way to actually achieve reliable clean-less builds is to run it in an environment where you either take away all access to non-dependencies (like Nix[0]) or automatically record all accessed files (like Tup[1]). You can also let the compiler record dependencies. Gcc, for example, does have an option for that. That's explained here: https://redo.readthedocs.io/en/latest/#whats-so-special-about-redo https://redo.readthedocs.io/en/latest/#whats-so-special-abou...
jnxx 6y ago
To mention, I think Nix or Guix are indeed good complements to things like redo! Guix System and NixOS are just geared to define the whole system, while make or redo build a single program or a set of artifacts.
jlokier 6y ago
GCC actually gets it wrong, as does almost every compiler and makefile combination. Consider this realistic example. Realistic in that I've encountered it in the real world and seen buggy demos shipped because of it, with non-reproducible bugs. We have a search path for headers, like so: -Imyinc -I../dep1/inc -Idep2/inc -I/usr/include We compile a source file which contains: #include "creative.h" GCC outputs a dependency which is read into the Makefile next time: object.o: ../dep2/inc/creative.h Looks good! Later that day we update from our upstream dependencies: git pull Which adds a file ../dep1/inc/creative.h Recompiling, the project works: make ... make test => 144,123 pass, 0 fail - well done, you earned a coffee! We captured automatic dependencies, this should be good to ship. But no: make clean ... make ... make test => FAIL! What happened? Turns out someone upstream moved (by copying) creative.h to ../dep1/inc and forgot to remove the old copy in ../dep2/inc. Then someone edited a data structure in that file. Automatic dependencies gave everyone false confidence in the build results seen by different people. That kind of automatic dependency is insufficient because it doesn't capture "negative file results" during path searches, where output depends on a file not existing, and changes if a new file is created. It is possible to use Makefiles with this handled accurately in the automatic dependency tracking, but I haven't seen it done often. An equivalent problem occurs in caches which automatically track data dependencies used to generate results. For example web pages generated from a combination of files, templates and data, build artifacts such as container/VM images, or at a much lower level, data dependencies inside calculations with conditional branches. To do caching right, they must validate negative tests as well as positive that arise in searches. I've seen a surprising number of cache implementations which don't try particularly hard to get this right.

aequitas 6y ago

I (ab)use Make for anything but compiling C sources to binaries. Eg. all my Python projects use Make to conditionally create virtualenv's, install packages, run code linting/tests. For which Make is just the most ubiquitous and simple tool available. But one thing that still annoys me with Make are the workarounds needed to work with stuff which state can not be derived from a file (or only from complex paths). Things like having a background process running (eg. a test db) or a external package that needs to be installed (which can be easily queries using apt, but is much harder to determine reliably using system files). I end up having to create intermediate hidden files which sometimes decouple from the real state. Then things really become messy with Make for me. Would redo be a better solution in this case as well?

jnxx 6y ago
> But one thing that still annoys me with Make are the workarounds needed to work with stuff which state can not be derived from a file I am not sure about that one. I think the reason is that make is not made for general scripting, but it is made for generating a deterministic build product, with minimum time, from a deterministic input (the source files). In that sense, it is something like a "pure function" in functional programming. If you build software, you do not want to depend the result today on your CPU fan's speed, and tomorrow on the moon's phase. In other words, you really try to make the result deterministic. What you certainly can do is to first generate a kind of task file from the current system state (e.g., needs to re-create a test db), and letting redo build these targets. For installing / updating stuff, I do not see problems, but I believe package managers such as guix or nix or pip have more specialized options for this.

dllthomas 6y ago

> No "clean" command is needed. I don't see how this follows from what you've said. IME, `clean` is for when I want things rebuilt despite "nothing" having changed. Maybe it's actually the case that nothing's changed, and I want a full build because I want to know how long my build takes. Or maybe I've changed implicit dependencies - compiler version or similar. Could you elaborate?

jnxx 6y ago
> IME, `clean` is for when I want things rebuilt despite "nothing" having changed. Of course, you can use clean in this case. You can do that also with redo, just delete all the intermediate files and start a normal build. However the point is that you do a "make clean" most often to be sure that all dependencies are refreshed, because you are actually not sure that your Makefile captures them all. A good example are builds in yocto. This is a system to build complete embedded Linux images based on make, with many subprojects. But if you change, for example, the kernel to support large files, or a 32 bit time_t, you can't be sure that a simple "make" delivers a correct system, because the dependencies on the system headers are not include in all the makefiles.
- danaris 6y ago
  Please forgive the confusion, as I've only dabbled in Make, but... > You can do that also with redo, just delete all the intermediate files and start a normal build. ...isn't that pretty much what "make clean" generally does? I mean, in a project of any significant size, "all the intermediate files" aren't going to be trivial to delete without a pre-built "delete all the intermediate files" script...which is what "make clean" is (at least partly) for. Or are you just saying that "redo" (which I'd never heard of before, so, again, please forgive my ignorance) can also process a pre-built list of "intermediate files" to delete with some particular command or option...?
  - dllthomas 6y ago
    If I understand, the real point was that clean is less often needed. Having said that, you could certainly list system headers in a Makefile (... hopefully a generated one) if that's the behavior you want, so I am not sure what difference is actually being pointed to.

yjftsjthsd-h 6y ago

I don't see how that's any different. You've made your dependency on stdio.h explicit, so the build tool knows about it - and if you'd done the same thing for make, it would've done the same thing. helloworld: helloworld.c /usr/local/include/stdio.h gcc -I/usr/local/include helloworld.c -o helloworld Tada.