8 ms·

Build System Schism: The Curse of Meta Build Systems

81 points by gavinhoward 3y ago

jgeada 3y ago
Maybe I'm a dinosaur, but if the product you're building is so complex that it needs such a complex (and Turing complete!) description for how it needs to be built, maybe the problem is that product's architecture. Creating a meta build is fixing the symptom and not the problem. Yes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.
- gavinhoward 3y ago
  Author here. My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.
  - jgeada 3y ago
    I've designed and shipped more than one multi-million line of code product that had parts in C, C++, Fortran, Python, Tcl, and even some assembly, plus the usual grab bag of open source dependencies. For none of those was anything more sophisticated than (admittedly fairly complex) hierarchical Makefile description to make it come together. I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
    - gavinhoward 3y ago
      Recursive Make isn't the best. https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf That said, if Rig gives you nothing new, please don't even try it! I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
      - jgeada 3y ago
        There is something very fishy in that article: the order of traversal in a hierarchical make should be entirely expressed by dependency rules, not mere existence of a directory. Failure to describe dependencies is a flaw in the creation of the Makefile, not an inherent property of make itself. Specifically, this idiom: all: for dir in $(MODULES); do \ (cd $$dir; ${MAKE} all); \ done should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is: module1: (cd $module1_dir; make) module2: module1 (cd $module2_dir; make) etc etc etc Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
        jcranmer 3y ago
        > Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG. You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies). The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
        3y ago
- throwway120385 3y ago
  What if my product is not an executable, but a root filesystem?
- deleted 3y ago
  [deleted]
- bluGill 3y ago
  I want everything declarative, and mostly things are. However I keep running into something weird that the build system designers didn't think of and so I need something to make it work. Note that I'm the cmake expert here, so when I say I keep running into these situations understand that most of the other developers never hit these things, but when it happens they go to me and so I run into them often even though they are rare.
- rcxdude 3y ago
  I suspect that you can describe pretty muich all builds without turing completeness, but avoiding it is rarely particularly helpful. What you need is a language that has a reasonable amount of expressiveness to describe modules and how they fit together, including whatever decisions you need to allow for a) build configurations and b) dealing with platform variations. It's a lot easier to express this in a turing complete language than a non-turing complete language, and it's hard to make a non-turing-complete expressive enough without accidentally making it turing complete. The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
  - TheNewAndy 3y ago
    I have made a (internal only unfortunately) meta build system for my work. Instead of being Turing complete is uses Horn Clauses as the primary mode of logic. This means that the problem of determining "is this compiler flag set" or "is this file used in this build" is always solvable in polynomial time. It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects. I don't believe anyone has missed Turing completeness. Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
    - rcxdude 3y ago
- deleted 3y ago
  [deleted]
- jcranmer 3y ago
  Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic. Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled. Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache... I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result. Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz.build https://searchfox.org/mozilla-central/source/browser/app/moz...
  - pixl97 3y ago
    >so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
- trueismywork 3y ago
  Build systems do more than just build files. They find and run tests, analyze test results, analyze coverage. Conditionally generate code based on whether a feature is enabled or disable. Run "find dependency" logic. Read pkgconfig files.
taeric 3y ago
Wasn't a large part of the complexity introduced by autoconf from how loosely prescribed UNIX systems were in where things were installed and how they were implemented? Feels that we are much more homogenous now than we have ever been in the past. Such that it is not that surprising to find a lot of complexity in the past is not as necessary today.
- JohnFen 3y ago
  This is my experience. Autoconf used to be really important, but I haven't actually needed it for years now.
Darkstryder 3y ago
> Say you add a new source file. Wouldn’t it be great if your build system just picked it up? > Alas, it cannot; the list of stuff to build is passed from the meta build system to the build system, usually by fiat. It can, actually. At least for make. Just use a wildcard in uour rules: https://www.gnu.org/software/make/manual/make.html#Wildcards https://www.gnu.org/software/make/manual/make.html#Wildcards I highly recommend going through the make documentation at least once in your career. Per the lindy effect, as it has been around for 40 years, it has a decent chance of sticking around for another 40.
- gavinhoward 3y ago
  Author here. I mentioned suffix rules and pattern rules. Unfortunately, meta build systems don't like to generate rules like that.
- kjkjadksj 3y ago
  If the lindy effect were real we’d all be romans still
  - shrimp_emoji 3y ago
    It worked on these guys: https://en.wikipedia.org/wiki/Sultanate_of_Rum https://en.wikipedia.org/wiki/Sultanate_of_Rum
layer8 3y ago
Regarding dynamic dependencies, the author doesn’t seem to be familiar with the build systems used for Java (Maven and Gradle), which in my understanding use static dependencies (static build graph).
- gavinhoward 3y ago
  Author here. I have worked with them long enough ago I can't remember. Do they require the build files to list all dependencies?
  - layer8 3y ago
    Yes, that’s how they work. To be precise, the direct dependencies are listed, and each dependency (after retrieval from a package repository, or locally cached) provides a static list of its own direct dependencies.
    - gavinhoward 3y ago
      Okay. The point of dymanic dependencies is that Rig could run some code to figure out what the main function imports, transitively, and use that info to build the entire project. All without listing dependencies in the build file. A good API might look like this: build_java_package("com.yzena.rig"); So when you add a new package and use it, you do not have to change the build files; Rig would pick it up automatically.
      - layer8 3y ago
        In Java, there is no 1:1 relation between package imports and jars. A jar (dependency) usually contains many packages (namespaces), and of course the same packages can be contained in multiple (usually mutually incompatible) dependencies. (Java imports are often more fine-grained than, say, C/C++ header files.) Dependency selection therefore requires a conscious decision. IDEs provide tooling to suggest matching dependencies to be added to the build file based on the names of unresolved imports. However, the much more common direction is to first decide which libraries (dependencies) you want to use and add those to your build file, which then allows the IDE to suggest imports when you reference some unresolved type in your source code. So the dependency declarations in the build file generally come first, before you start writing your corresponding code. (This can be likened to specifying coarse-grained includes in your project build file, from which fine-grained sub-includes are then selected in each source file.) The conscious selection of dependencies is a good thing, IMO. Dependencies also don’t change frequently, so there isn’t much work to be saved by determining them automatically (except for transitive dependencies upon version bumps, which are determined automatically by the build system). Moreover, it’s not only the dependencies as such that are consciously selected, but also their respective version numbers. This prevents unexpected breaking changes.
somewhereoutth 3y ago
A => B You'd think that would be hard for even engineers to screw up, but here we are.
- barryrandall 3y ago
  The engineers wanted to write a thin encabulation module in K++, but management insisted that the engineering team use the industry standard Encabulate.moo library, and the Cows on Crack framework doesn't have a stable ABI.
  - somewhereoutth 3y ago
    Though it would be nice to blame this on Management (who are often the root cause of all sorts of screw ups) - managers generally have no idea of, let alone interest, in build systems. We did this to ourselves.
  - mdaniel 3y ago
    now all we need is a Markov Chain trained to produce this, publish YT shorts of them, and retire https://www.404media.co/inside-the-world-of-tiktok-spammers-and-the-ai-tools-that-enable-them/ https://www.404media.co/inside-the-world-of-tiktok-spammers-...
    - barryrandall 3y ago
      I'm pretty sure that party is over. "Make Money Online with ${x}" is what you pivot to when the return on ${x} drops below the going rate for "Make Money Online" courses.
- KWxIUElW8Xt0tD9 3y ago
  high logical intelligence doesn't imply common sense
nottorp 3y ago
How about a build system to manage the existing build systems? Perhap just one layer of meta isn't enough...
- mdaniel 3y ago
  CMake gotcha, fam: https://cmake.org/cmake/help/v3.29/manual/cmake-generator-expressions.7.html#introduction https://cmake.org/cmake/help/v3.29/manual/cmake-generator-ex...
  - nottorp 3y ago
    I been using it wrong all these years... <hides under carpet>
    - MathMonkeyMan 3y ago
      Don't feel bad, we all use it wrong.
choeger 3y ago
Good post. Good analysis. One problem: If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from? So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth. Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C. While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.
- gavinhoward 3y ago
  > If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? This is a great question that I will make sure to answer in my Turing-completeness post in a week. But to answer now, it's because of multi-project builds, where projects can be in multiple languages. If you have build systems that are not independent from their language, how do you get them to play nice? I designed Rig to be able to act as a babysitter and make them play nice. If I play my cards right, Rig will be the reason other build systems become irrelevant. :)
  - codethatwerks 3y ago
    What about a single API for builds, implemented in multiple general purpose languages. End users get to pick a familiar language to them. Similar to how Pulumi is available in multiple languages. Then code both builds and meta build concerns as you would code anything else.
  - aappleby 3y ago
    Not if Hancho beats you to it. ;) https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
jeffbee 3y ago
What can be done about libraries on which you want to depend, but that have too many opinions about how they should be built? For example I use bazel and one large library I use forced -Werror into the C compiler invocation, but with my C compiler it has warnings, so it can't actually be built unless I patch out that line of their bazel configs. Another library only builds correctly when using clang if the clang toolchain is named literally "clang" and not, for example, "clang-17". Consequently I have to adopt their convention or maintain a patch against their project. There are many, many unsolved rough edges in bazel along these lines.
- gavinhoward 3y ago
  Author here. Rig will have multi-project builds. It also has a concept called "context stacks". The idea would be that you would have a "compiler option" context stack, and you would push `-Wno-bad-warning`, then inside the context, build that library. Then the library would add `-Werror`, but it would also have `-Wno-bad-warning`, so no problem. But multi-project builds won't exist in the first release.
fanf2 3y ago
Before autoconf there was metaconfig, which was Larry Wall’s portability support for rn and perl.
- KWxIUElW8Xt0tD9 3y ago
  Bell Labs had a very nice portability framework also -- there was a Cygwin-like product called U/WIN that made extensive use of it -- might find it associated with ksh93 maybe. Speaking of Bell Labs, there are some great ideas for a build tool in the Bell Labs "nmake" tool - I believe it was originally created to speed up ESS5 switch builds, which took 3 days before make. It has programmable header file scanning for example, a shell coprocess to avoid all the fork/exec of standard make, build state files, lots of other things I have forgotten.
bluGill 3y ago
This is missing something: what does my ide do? I want a key combo or menu select for all the things I do. Build and run tests in address sanitizer. Build, deploy to my embedded target then run in the debugger with these options.
- gavinhoward 3y ago
  Author here. Every IDE is different, unfortunately. Rig will have to adapt to each IDE, but it will generate files with enough info to call into Rig, which will do the job.
max-privatevoid 3y ago
Nix with dynamic derivations (RFC92) could potentially beat this curse. https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...
- ghthor 3y ago
  Seems like a good idea. You know if there’s been any progress since 2019 on that RFC? I’m going to look into it more deeply tomorrow.
aappleby 3y ago
I recently wrote Hancho (https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho) that is designed to be fully dynamic; I thiiiink it would fit most of the desired criteria here (aside from cloud caching) but I'm not positive. I'll have to do a bit more investigating into what this article means by "dynamic dependencies".
tom_ 3y ago
> Say you add a new source file. Wouldn’t it be great if your build system just picked it up? I have never enjoyed the experience of working with tools that did this. How does it know which files I actually want, and which files are there because I (or somebody else) got something wrong? The answer, of course: it doesn't. I moan a lot about having to do the computer's work for it, but I think in this case I am happy to put up with some minor drudgery (adding a file name to a list, once) in exchange for the computer's boundless ability to then patiently follow my instructions without trying to second-guess me on every single build.
- gavinhoward 3y ago
  Author here. Rig will do this too. You can statically list dependencies.
  - scrubs 3y ago
    Look at bloomberg's bde library on github. There is a convention for naming so every file can only be in one place.
  - scrubs 3y ago
    Looking fwd to reading about rig. Consider pushing a standard for builds to the c++ committee. With modules on the table it's criminally bad c++ has no eta to get substantially closer to golang like builds.
- bsder 3y ago
  Even worse, I have yet to see a build system that screams at you when there is a file where it shouldn't be. Yes, I want my build fast. However, give me a way to kick off a scan to look for extraneous stuff.
  - tom_ 3y ago
    That's a good idea too. For my smaller-scale personal projects I rely on the git status, but if you use some other version control system then you don't always get early warning of this sort of thing. (My work stuff typically uses Perforce, and it's annoyingly blasé about this stuff.)
soulbadguy 3y ago
> So there are five features of build systems that you want, and the harsh truth is that meta build systems cannot have some of them! Some of these things are only available on end-to-end build systems. Did I miss something or did the authors actually didn't explain what prevents meta build systems to implement any of those features. Looking at the current state of cmake it seems to have everything mentioned here. Including dependencies and header parsing in c++
bjkchoy 3y ago
SCons is listed as a meta build system here, but it is clearly an end-to-end build system. It seems to me SCons even fits all the requirements listed by the author. Plus, you program your SCons build using Python. He also describes the dynamic discovery of header dependencies of Ninja as a hack. It is true that it is a special case to handle a specific type of dynamic dependencies. But the basic idea behind it (using the compiler's diagnostic as the source of truth) is sane and efficient. The alternative is to scan the sources using a custom parser, which is slower and more fragile.
- jason_s 3y ago
  > The alternative is to scan the sources using a custom parser, which is slower and more fragile. More fragile, yes, but I dispute the "slower" aspect. waf does this very well. In C you don't need to parse the entire program structure, you just need to parse what the preprocessor does. Also there's a difference between the complete set of header dependencies (notably #include statements within an #if or #ifdef) and a more sane but incomplete methodology for #include statements (don't do it within #if/#ifdef). Not sure whether this is a big enough difference to try to matter.
- gavinhoward 3y ago
  You can definitely use the actual compiler with a more general build system with dynamic dependencies.
MarkSweep 3y ago
Do you have an RSS or Atom feed so I can subscribe and be notified when Rig is available. Anyways, I’m looking forward to more options in this space. The current pattern of rebuilding everything from scratch every time in a non deterministic way is the gross reality of GitHub Actions and Dockerfiles. It would be great to have more people working on a more cacheable build system. That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
- gavinhoward 3y ago
  https://yzena.com/index.xml https://yzena.com/index.xml But I can email you directly if you wish. > It would be great to have more people working on a more cacheable build system. Rig is not that yet, unfortunately. I don't want to get your hopes up. It can't even do incremental builds yet. And while it has multi-threading, I haven't tested it. This is very alpha software. > That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system. Generating those files was my plan, but friction is not good. Can you describe the friction and its source? I'll do some rethinking and redesign. Thank you for your interest!
  - MarkSweep 3y ago
    > Rig is not that yet, unfortunately. I don't want to get your hopes up. No worries, this space has so much inefficiency that I'm sure you will eventually start chipping away at it. > Can you describe the friction and its source? I'll do some rethinking and redesign. Some software ecosystems already have their own build tools. Often these build systems are intertwined with IDE support. So care must be taken to ensure the IDE still functions as developers expect. If you ignore the functioning of the IDE, developers will balk at adopting the new build tool. There is some prior art for generating files that an IDE can understand: https://buck.build/command/project.html https://buck.build/command/project.html https://github.com/bazelbuild/tulsi https://github.com/bazelbuild/tulsi Interfacing existing build tools into a meta build tool can be difficult to do efficiently, but I’ve run out of time for talking about that tonight.
boris 3y ago
The article lists some specific limitations of the meta build systems but doesn't really get to the underlying, fundamental issue with the approach. Which leaves the possibility of these specific limitations being addressed somehow and thus refuting the claim that meta build systems are fundamentally broken. I think the fundamental issue boils down to aggregation. Normally, aggregation is a good thing, a variant of the divide and conquer technique which we were all taught in kindergarten you can never go wrong with. But in build systems aggregation leads to the loss of precision (rebuilding more than necessary) or accuracy (not rebuilding when necessary, rebuilding things in the wrong order), and usually both. This is the fundamental reason why "recursive make is considered harmful": we split one big graph that has the total, accurate view of all the dependencies into a number of aggregate sub-graphs. And the same happens with meta build systems where we do some part of the build (configuration, pre-build steps where we generate source code, etc) in the meta build system and the rest in the underlying build system, essentially splitting the graph into two. P.S. And, yes, `build2` supports both dynamic prerequisites/dependencies[1] and dynamic targets[2]. [1] https://build2.org/release/0.15.0.xhtml#dyndep https://build2.org/release/0.15.0.xhtml#dyndep [2] https://build2.org/release/0.16.0.xhtml#dyn-target https://build2.org/release/0.16.0.xhtml#dyn-target
- gavinhoward 3y ago
  I may not have said it as well as you did, but that is what I was trying to get across with the barrier analogy.

81 points by gavinhoward 3y ago

Maybe I'm a dinosaur, but if the product you're building is so complex that it needs such a complex (and Turing complete!) description for how it needs to be built, maybe the problem is that product's architecture. Creating a meta build is fixing the symptom and not the problem. Yes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.

gavinhoward 3y ago
Author here. My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.
- jgeada 3y ago
  I've designed and shipped more than one multi-million line of code product that had parts in C, C++, Fortran, Python, Tcl, and even some assembly, plus the usual grab bag of open source dependencies. For none of those was anything more sophisticated than (admittedly fairly complex) hierarchical Makefile description to make it come together. I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
  - gavinhoward 3y ago
    Recursive Make isn't the best. https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf That said, if Rig gives you nothing new, please don't even try it! I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
    - jgeada 3y ago
      There is something very fishy in that article: the order of traversal in a hierarchical make should be entirely expressed by dependency rules, not mere existence of a directory. Failure to describe dependencies is a flaw in the creation of the Makefile, not an inherent property of make itself. Specifically, this idiom: all: for dir in $(MODULES); do \ (cd $$dir; ${MAKE} all); \ done should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is: module1: (cd $module1_dir; make) module2: module1 (cd $module2_dir; make) etc etc etc Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
      - jcranmer 3y ago
        > Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG. You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies). The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
        3y ago
throwway120385 3y ago
What if my product is not an executable, but a root filesystem?
deleted 3y ago
[deleted]
bluGill 3y ago
I want everything declarative, and mostly things are. However I keep running into something weird that the build system designers didn't think of and so I need something to make it work. Note that I'm the cmake expert here, so when I say I keep running into these situations understand that most of the other developers never hit these things, but when it happens they go to me and so I run into them often even though they are rare.
rcxdude 3y ago
I suspect that you can describe pretty muich all builds without turing completeness, but avoiding it is rarely particularly helpful. What you need is a language that has a reasonable amount of expressiveness to describe modules and how they fit together, including whatever decisions you need to allow for a) build configurations and b) dealing with platform variations. It's a lot easier to express this in a turing complete language than a non-turing complete language, and it's hard to make a non-turing-complete expressive enough without accidentally making it turing complete. The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
- TheNewAndy 3y ago
  I have made a (internal only unfortunately) meta build system for my work. Instead of being Turing complete is uses Horn Clauses as the primary mode of logic. This means that the problem of determining "is this compiler flag set" or "is this file used in this build" is always solvable in polynomial time. It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects. I don't believe anyone has missed Turing completeness. Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
  - rcxdude 3y ago
deleted 3y ago
[deleted]
jcranmer 3y ago
Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic. Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled. Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache... I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result. Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz.build https://searchfox.org/mozilla-central/source/browser/app/moz...
- pixl97 3y ago
  >so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
trueismywork 3y ago
Build systems do more than just build files. They find and run tests, analyze test results, analyze coverage. Conditionally generate code based on whether a feature is enabled or disable. Run "find dependency" logic. Read pkgconfig files.

taeric 3y ago

Wasn't a large part of the complexity introduced by autoconf from how loosely prescribed UNIX systems were in where things were installed and how they were implemented? Feels that we are much more homogenous now than we have ever been in the past. Such that it is not that surprising to find a lot of complexity in the past is not as necessary today.

JohnFen 3y ago
This is my experience. Autoconf used to be really important, but I haven't actually needed it for years now.

Darkstryder 3y ago

> Say you add a new source file. Wouldn’t it be great if your build system just picked it up? > Alas, it cannot; the list of stuff to build is passed from the meta build system to the build system, usually by fiat. It can, actually. At least for make. Just use a wildcard in uour rules: https://www.gnu.org/software/make/manual/make.html#Wildcards https://www.gnu.org/software/make/manual/make.html#Wildcards I highly recommend going through the make documentation at least once in your career. Per the lindy effect, as it has been around for 40 years, it has a decent chance of sticking around for another 40.

gavinhoward 3y ago
Author here. I mentioned suffix rules and pattern rules. Unfortunately, meta build systems don't like to generate rules like that.
kjkjadksj 3y ago
If the lindy effect were real we’d all be romans still
- shrimp_emoji 3y ago
  It worked on these guys: https://en.wikipedia.org/wiki/Sultanate_of_Rum https://en.wikipedia.org/wiki/Sultanate_of_Rum

layer8 3y ago

Regarding dynamic dependencies, the author doesn’t seem to be familiar with the build systems used for Java (Maven and Gradle), which in my understanding use static dependencies (static build graph).

gavinhoward 3y ago
Author here. I have worked with them long enough ago I can't remember. Do they require the build files to list all dependencies?
- layer8 3y ago
  Yes, that’s how they work. To be precise, the direct dependencies are listed, and each dependency (after retrieval from a package repository, or locally cached) provides a static list of its own direct dependencies.
  - gavinhoward 3y ago
    Okay. The point of dymanic dependencies is that Rig could run some code to figure out what the main function imports, transitively, and use that info to build the entire project. All without listing dependencies in the build file. A good API might look like this: build_java_package("com.yzena.rig"); So when you add a new package and use it, you do not have to change the build files; Rig would pick it up automatically.
    - layer8 3y ago
      In Java, there is no 1:1 relation between package imports and jars. A jar (dependency) usually contains many packages (namespaces), and of course the same packages can be contained in multiple (usually mutually incompatible) dependencies. (Java imports are often more fine-grained than, say, C/C++ header files.) Dependency selection therefore requires a conscious decision. IDEs provide tooling to suggest matching dependencies to be added to the build file based on the names of unresolved imports. However, the much more common direction is to first decide which libraries (dependencies) you want to use and add those to your build file, which then allows the IDE to suggest imports when you reference some unresolved type in your source code. So the dependency declarations in the build file generally come first, before you start writing your corresponding code. (This can be likened to specifying coarse-grained includes in your project build file, from which fine-grained sub-includes are then selected in each source file.) The conscious selection of dependencies is a good thing, IMO. Dependencies also don’t change frequently, so there isn’t much work to be saved by determining them automatically (except for transitive dependencies upon version bumps, which are determined automatically by the build system). Moreover, it’s not only the dependencies as such that are consciously selected, but also their respective version numbers. This prevents unexpected breaking changes.

somewhereoutth 3y ago

A => B You'd think that would be hard for even engineers to screw up, but here we are.

barryrandall 3y ago
The engineers wanted to write a thin encabulation module in K++, but management insisted that the engineering team use the industry standard Encabulate.moo library, and the Cows on Crack framework doesn't have a stable ABI.
- somewhereoutth 3y ago
  Though it would be nice to blame this on Management (who are often the root cause of all sorts of screw ups) - managers generally have no idea of, let alone interest, in build systems. We did this to ourselves.
- mdaniel 3y ago
  now all we need is a Markov Chain trained to produce this, publish YT shorts of them, and retire https://www.404media.co/inside-the-world-of-tiktok-spammers-and-the-ai-tools-that-enable-them/ https://www.404media.co/inside-the-world-of-tiktok-spammers-...
  - barryrandall 3y ago
    I'm pretty sure that party is over. "Make Money Online with ${x}" is what you pivot to when the return on ${x} drops below the going rate for "Make Money Online" courses.
KWxIUElW8Xt0tD9 3y ago
high logical intelligence doesn't imply common sense

nottorp 3y ago

How about a build system to manage the existing build systems? Perhap just one layer of meta isn't enough...

mdaniel 3y ago
CMake gotcha, fam: https://cmake.org/cmake/help/v3.29/manual/cmake-generator-expressions.7.html#introduction https://cmake.org/cmake/help/v3.29/manual/cmake-generator-ex...
- nottorp 3y ago
  I been using it wrong all these years... <hides under carpet>
  - MathMonkeyMan 3y ago
    Don't feel bad, we all use it wrong.

choeger 3y ago

Good post. Good analysis. One problem: If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from? So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth. Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C. While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.

gavinhoward 3y ago
> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now? This is a great question that I will make sure to answer in my Turing-completeness post in a week. But to answer now, it's because of multi-project builds, where projects can be in multiple languages. If you have build systems that are not independent from their language, how do you get them to play nice? I designed Rig to be able to act as a babysitter and make them play nice. If I play my cards right, Rig will be the reason other build systems become irrelevant. :)
- codethatwerks 3y ago
  What about a single API for builds, implemented in multiple general purpose languages. End users get to pick a familiar language to them. Similar to how Pulumi is available in multiple languages. Then code both builds and meta build concerns as you would code anything else.
- aappleby 3y ago
  Not if Hancho beats you to it. ;) https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.

jeffbee 3y ago

What can be done about libraries on which you want to depend, but that have too many opinions about how they should be built? For example I use bazel and one large library I use forced -Werror into the C compiler invocation, but with my C compiler it has warnings, so it can't actually be built unless I patch out that line of their bazel configs. Another library only builds correctly when using clang if the clang toolchain is named literally "clang" and not, for example, "clang-17". Consequently I have to adopt their convention or maintain a patch against their project. There are many, many unsolved rough edges in bazel along these lines.

gavinhoward 3y ago
Author here. Rig will have multi-project builds. It also has a concept called "context stacks". The idea would be that you would have a "compiler option" context stack, and you would push `-Wno-bad-warning`, then inside the context, build that library. Then the library would add `-Werror`, but it would also have `-Wno-bad-warning`, so no problem. But multi-project builds won't exist in the first release.

fanf2 3y ago

Before autoconf there was metaconfig, which was Larry Wall’s portability support for rn and perl.

KWxIUElW8Xt0tD9 3y ago
Bell Labs had a very nice portability framework also -- there was a Cygwin-like product called U/WIN that made extensive use of it -- might find it associated with ksh93 maybe. Speaking of Bell Labs, there are some great ideas for a build tool in the Bell Labs "nmake" tool - I believe it was originally created to speed up ESS5 switch builds, which took 3 days before make. It has programmable header file scanning for example, a shell coprocess to avoid all the fork/exec of standard make, build state files, lots of other things I have forgotten.

bluGill 3y ago

This is missing something: what does my ide do? I want a key combo or menu select for all the things I do. Build and run tests in address sanitizer. Build, deploy to my embedded target then run in the debugger with these options.

gavinhoward 3y ago
Author here. Every IDE is different, unfortunately. Rig will have to adapt to each IDE, but it will generate files with enough info to call into Rig, which will do the job.

max-privatevoid 3y ago

Nix with dynamic derivations (RFC92) could potentially beat this curse. https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...

ghthor 3y ago
Seems like a good idea. You know if there’s been any progress since 2019 on that RFC? I’m going to look into it more deeply tomorrow.

aappleby 3y ago

I recently wrote Hancho (https://github.com/aappleby/Hancho https://github.com/aappleby/Hancho) that is designed to be fully dynamic; I thiiiink it would fit most of the desired criteria here (aside from cloud caching) but I'm not positive. I'll have to do a bit more investigating into what this article means by "dynamic dependencies".

tom_ 3y ago

> Say you add a new source file. Wouldn’t it be great if your build system just picked it up? I have never enjoyed the experience of working with tools that did this. How does it know which files I actually want, and which files are there because I (or somebody else) got something wrong? The answer, of course: it doesn't. I moan a lot about having to do the computer's work for it, but I think in this case I am happy to put up with some minor drudgery (adding a file name to a list, once) in exchange for the computer's boundless ability to then patiently follow my instructions without trying to second-guess me on every single build.

gavinhoward 3y ago
Author here. Rig will do this too. You can statically list dependencies.
- scrubs 3y ago
  Look at bloomberg's bde library on github. There is a convention for naming so every file can only be in one place.
- scrubs 3y ago
  Looking fwd to reading about rig. Consider pushing a standard for builds to the c++ committee. With modules on the table it's criminally bad c++ has no eta to get substantially closer to golang like builds.
bsder 3y ago
Even worse, I have yet to see a build system that screams at you when there is a file where it shouldn't be. Yes, I want my build fast. However, give me a way to kick off a scan to look for extraneous stuff.
- tom_ 3y ago
  That's a good idea too. For my smaller-scale personal projects I rely on the git status, but if you use some other version control system then you don't always get early warning of this sort of thing. (My work stuff typically uses Perforce, and it's annoyingly blasé about this stuff.)

soulbadguy 3y ago

> So there are five features of build systems that you want, and the harsh truth is that meta build systems cannot have some of them! Some of these things are only available on end-to-end build systems. Did I miss something or did the authors actually didn't explain what prevents meta build systems to implement any of those features. Looking at the current state of cmake it seems to have everything mentioned here. Including dependencies and header parsing in c++

bjkchoy 3y ago

SCons is listed as a meta build system here, but it is clearly an end-to-end build system. It seems to me SCons even fits all the requirements listed by the author. Plus, you program your SCons build using Python. He also describes the dynamic discovery of header dependencies of Ninja as a hack. It is true that it is a special case to handle a specific type of dynamic dependencies. But the basic idea behind it (using the compiler's diagnostic as the source of truth) is sane and efficient. The alternative is to scan the sources using a custom parser, which is slower and more fragile.

jason_s 3y ago
> The alternative is to scan the sources using a custom parser, which is slower and more fragile. More fragile, yes, but I dispute the "slower" aspect. waf does this very well. In C you don't need to parse the entire program structure, you just need to parse what the preprocessor does. Also there's a difference between the complete set of header dependencies (notably #include statements within an #if or #ifdef) and a more sane but incomplete methodology for #include statements (don't do it within #if/#ifdef). Not sure whether this is a big enough difference to try to matter.
gavinhoward 3y ago
You can definitely use the actual compiler with a more general build system with dynamic dependencies.

MarkSweep 3y ago

Do you have an RSS or Atom feed so I can subscribe and be notified when Rig is available. Anyways, I’m looking forward to more options in this space. The current pattern of rebuilding everything from scratch every time in a non deterministic way is the gross reality of GitHub Actions and Dockerfiles. It would be great to have more people working on a more cacheable build system. That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.

gavinhoward 3y ago
https://yzena.com/index.xml https://yzena.com/index.xml But I can email you directly if you wish. > It would be great to have more people working on a more cacheable build system. Rig is not that yet, unfortunately. I don't want to get your hopes up. It can't even do incremental builds yet. And while it has multi-threading, I haven't tested it. This is very alpha software. > That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system. Generating those files was my plan, but friction is not good. Can you describe the friction and its source? I'll do some rethinking and redesign. Thank you for your interest!
- MarkSweep 3y ago
  > Rig is not that yet, unfortunately. I don't want to get your hopes up. No worries, this space has so much inefficiency that I'm sure you will eventually start chipping away at it. > Can you describe the friction and its source? I'll do some rethinking and redesign. Some software ecosystems already have their own build tools. Often these build systems are intertwined with IDE support. So care must be taken to ensure the IDE still functions as developers expect. If you ignore the functioning of the IDE, developers will balk at adopting the new build tool. There is some prior art for generating files that an IDE can understand: https://buck.build/command/project.html https://buck.build/command/project.html https://github.com/bazelbuild/tulsi https://github.com/bazelbuild/tulsi Interfacing existing build tools into a meta build tool can be difficult to do efficiently, but I’ve run out of time for talking about that tonight.

boris 3y ago

The article lists some specific limitations of the meta build systems but doesn't really get to the underlying, fundamental issue with the approach. Which leaves the possibility of these specific limitations being addressed somehow and thus refuting the claim that meta build systems are fundamentally broken. I think the fundamental issue boils down to aggregation. Normally, aggregation is a good thing, a variant of the divide and conquer technique which we were all taught in kindergarten you can never go wrong with. But in build systems aggregation leads to the loss of precision (rebuilding more than necessary) or accuracy (not rebuilding when necessary, rebuilding things in the wrong order), and usually both. This is the fundamental reason why "recursive make is considered harmful": we split one big graph that has the total, accurate view of all the dependencies into a number of aggregate sub-graphs. And the same happens with meta build systems where we do some part of the build (configuration, pre-build steps where we generate source code, etc) in the meta build system and the rest in the underlying build system, essentially splitting the graph into two. P.S. And, yes, `build2` supports both dynamic prerequisites/dependencies[1] and dynamic targets[2]. [1] https://build2.org/release/0.15.0.xhtml#dyndep https://build2.org/release/0.15.0.xhtml#dyndep [2] https://build2.org/release/0.16.0.xhtml#dyn-target https://build2.org/release/0.16.0.xhtml#dyn-target

gavinhoward 3y ago
I may not have said it as well as you did, but that is what I was trying to get across with the barrier analogy.