6 ms·

Maybe I'm a dinosaur, but if the product you're building is so complex that it needs such a complex (and Turing complete!) description for how it needs to be bu

by jgeada 3y ago

Maybe I'm a dinosaur, but if the product you're building is so complex that it needs such a complex (and Turing complete!) description for how it needs to be built, maybe the problem is that product's architecture. Creating a meta build is fixing the symptom and not the problem. Yes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.

gavinhoward 3y ago
Author here. My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.
- jgeada 3y ago
  I've designed and shipped more than one multi-million line of code product that had parts in C, C++, Fortran, Python, Tcl, and even some assembly, plus the usual grab bag of open source dependencies. For none of those was anything more sophisticated than (admittedly fairly complex) hierarchical Makefile description to make it come together. I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
  - gavinhoward 3y ago
    Recursive Make isn't the best. https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf That said, if Rig gives you nothing new, please don't even try it! I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
    - jgeada 3y ago
      There is something very fishy in that article: the order of traversal in a hierarchical make should be entirely expressed by dependency rules, not mere existence of a directory. Failure to describe dependencies is a flaw in the creation of the Makefile, not an inherent property of make itself. Specifically, this idiom: all: for dir in $(MODULES); do \ (cd $$dir; ${MAKE} all); \ done should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is: module1: (cd $module1_dir; make) module2: module1 (cd $module2_dir; make) etc etc etc Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
      - jcranmer 3y ago
        > Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG. You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies). The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
        3y ago
throwway120385 3y ago
What if my product is not an executable, but a root filesystem?
deleted 3y ago
[deleted]
bluGill 3y ago
I want everything declarative, and mostly things are. However I keep running into something weird that the build system designers didn't think of and so I need something to make it work. Note that I'm the cmake expert here, so when I say I keep running into these situations understand that most of the other developers never hit these things, but when it happens they go to me and so I run into them often even though they are rare.
rcxdude 3y ago
I suspect that you can describe pretty muich all builds without turing completeness, but avoiding it is rarely particularly helpful. What you need is a language that has a reasonable amount of expressiveness to describe modules and how they fit together, including whatever decisions you need to allow for a) build configurations and b) dealing with platform variations. It's a lot easier to express this in a turing complete language than a non-turing complete language, and it's hard to make a non-turing-complete expressive enough without accidentally making it turing complete. The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
- TheNewAndy 3y ago
  I have made a (internal only unfortunately) meta build system for my work. Instead of being Turing complete is uses Horn Clauses as the primary mode of logic. This means that the problem of determining "is this compiler flag set" or "is this file used in this build" is always solvable in polynomial time. It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects. I don't believe anyone has missed Turing completeness. Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
  - rcxdude 3y ago
deleted 3y ago
[deleted]
jcranmer 3y ago
Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic. Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled. Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache... I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result. Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz.build https://searchfox.org/mozilla-central/source/browser/app/moz...
- pixl97 3y ago
  >so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
trueismywork 3y ago
Build systems do more than just build files. They find and run tests, analyze test results, analyze coverage. Conditionally generate code based on whether a feature is enabled or disable. Run "find dependency" logic. Read pkgconfig files.

by jgeada 3y ago

gavinhoward 3y ago

Author here. My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.

jgeada 3y ago
I've designed and shipped more than one multi-million line of code product that had parts in C, C++, Fortran, Python, Tcl, and even some assembly, plus the usual grab bag of open source dependencies. For none of those was anything more sophisticated than (admittedly fairly complex) hierarchical Makefile description to make it come together. I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
- gavinhoward 3y ago
  Recursive Make isn't the best. https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf That said, if Rig gives you nothing new, please don't even try it! I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
  - jgeada 3y ago
    There is something very fishy in that article: the order of traversal in a hierarchical make should be entirely expressed by dependency rules, not mere existence of a directory. Failure to describe dependencies is a flaw in the creation of the Makefile, not an inherent property of make itself. Specifically, this idiom: all: for dir in $(MODULES); do \ (cd $$dir; ${MAKE} all); \ done should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is: module1: (cd $module1_dir; make) module2: module1 (cd $module2_dir; make) etc etc etc Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
    - jcranmer 3y ago
      > Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG. You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies). The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
      - 3y ago

throwway120385 3y ago

What if my product is not an executable, but a root filesystem?

deleted 3y ago

[deleted]

bluGill 3y ago

I want everything declarative, and mostly things are. However I keep running into something weird that the build system designers didn't think of and so I need something to make it work. Note that I'm the cmake expert here, so when I say I keep running into these situations understand that most of the other developers never hit these things, but when it happens they go to me and so I run into them often even though they are rare.

rcxdude 3y ago

I suspect that you can describe pretty muich all builds without turing completeness, but avoiding it is rarely particularly helpful. What you need is a language that has a reasonable amount of expressiveness to describe modules and how they fit together, including whatever decisions you need to allow for a) build configurations and b) dealing with platform variations. It's a lot easier to express this in a turing complete language than a non-turing complete language, and it's hard to make a non-turing-complete expressive enough without accidentally making it turing complete. The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.

TheNewAndy 3y ago
I have made a (internal only unfortunately) meta build system for my work. Instead of being Turing complete is uses Horn Clauses as the primary mode of logic. This means that the problem of determining "is this compiler flag set" or "is this file used in this build" is always solvable in polynomial time. It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects. I don't believe anyone has missed Turing completeness. Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
- rcxdude 3y ago

deleted 3y ago

[deleted]

jcranmer 3y ago

Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic. Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled. Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache... I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result. Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz.build https://searchfox.org/mozilla-central/source/browser/app/moz...

pixl97 3y ago
>so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.

trueismywork 3y ago

Build systems do more than just build files. They find and run tests, analyze test results, analyze coverage. Conditionally generate code based on whether a feature is enabled or disable. Run "find dependency" logic. Read pkgconfig files.