3 ms·

Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative

Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic. Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled. Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache... I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result. Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz.build https://searchfox.org/mozilla-central/source/browser/app/moz...

pixl97 3y ago
>so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
joshmarinacci 3y ago
Yup. This is the difference between computer science and software engineering. Build systems in the real world have to be resilient in all sorts of crazy scenarios.
AtlasBarfed 3y ago
Simple declarative models are great until they ... Aren't. Essentially you have a DSL. A DSL can be great if you use it frequently enough to know the insurance and outs. Like SQL. But a build DSL is something you muck with like ... Once every six months? And most build files I one off rather than write from scratch. I'm not going to o learn your DSL. I'm going to stack overflow it. And build systems no matter how genius prescient the authors are, do weird things sometimes. Integrate with so e proprietary library? Have a required code generation step? Pull a dependency from a weird location? Auto annotate code? Alas then you have to either be able to decompile the DSL to the base lang.in your head, or inject some "real code". And ... By the standards of this article when dealing with the problem of building code (which, come on, evolves to a workflow engine eventually, and not a simple one) becomes meta.

pixl97 3y ago

>so many other possible complexities This right here. When working with large enterprise clients I'm seeing insanely complex build chains. Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.

joshmarinacci 3y ago

Yup. This is the difference between computer science and software engineering. Build systems in the real world have to be resilient in all sorts of crazy scenarios.

AtlasBarfed 3y ago

Simple declarative models are great until they ... Aren't. Essentially you have a DSL. A DSL can be great if you use it frequently enough to know the insurance and outs. Like SQL. But a build DSL is something you muck with like ... Once every six months? And most build files I one off rather than write from scratch. I'm not going to o learn your DSL. I'm going to stack overflow it. And build systems no matter how genius prescient the authors are, do weird things sometimes. Integrate with so e proprietary library? Have a required code generation step? Pull a dependency from a weird location? Auto annotate code? Alas then you have to either be able to decompile the DSL to the base lang.in your head, or inject some "real code". And ... By the standards of this article when dealing with the problem of building code (which, come on, evolves to a workflow engine eventually, and not a simple one) becomes meta.