3 ms·

> the right conceptual basis for build systems: ”build is the first stage of execution” I have long been thinking the same. And also: "Running tests is the fir

> the right conceptual basis for build systems: ”build is the first stage of execution” I have long been thinking the same. And also: "Running tests is the first stage of deploying in production" etc. In other words: There is often a whole dependency graph between various stages (install toolchain, install 3rd-party dependencies, do code generation, compile, link/bundle, test, run, …) and each of those stages should ideally be a pure function mapping inputs to outputs. In my experience, we often don't do a good job making this graph explicit, nor do we usually implement build steps as pure functions or think of build systems as another piece of software which needs to be subject to the same quality criteria that we apply to any other line of code we write. As a result, developer experience ends up suffering. Large JavaScript projects are particularly bad in this regard. Dependencies, auto-generated code and build output live right alongside source code, essentially making them global state from the point of view of the build system. The package.json contains dozens of "run" commands which more often than not are an arcane mix of bash scripts invoking JS code. Even worse, those commands typically need to be run in juuust the right order because they all operate on the same global state. There is no notion of one command depending on another. No effort put into isolating build tasks from each other and making them pure. No caching of intermediate results. Pipelines take ages even though they wouldn't have to. Developers get confused because a coworker introduced a new npm command yesterday which now needs be to run before anything else. Ugghhh.

haberman 2y ago
"Build is the first stage of execution" resonates with me too. It follows that build systems should be written in terms of real, encapsulated APIs that separate interface from implementation. Make-like build systems are like programming in assembly language: there is no structure that isolates one build step from another. Make offers no way of creating a "local variable" -- an intermediate file that is hidden from any other build step. The filesystem is like memory: as long as you can figure out the filename (address) somehow, you can read and write whatever you want. Even if you can declare that one build step depends on another, there's usually nothing that prevents a step from reading (or even writing!) files from other build steps that it did not declare a dependency on. What you want is a build system that lets you define real APIs, with truly private state. For example, cc_library() in Bazel (cxx_library() in Buck2) lets you define a C++ library in terms of the srcs/hdrs/etc, but you don't know how it will be implemented. Other rules and targets can consume the outputs of cc_library(), but only the files that are publicly exposed in the returned providers. This model brings true encapsulation to build systems, which makes it much more feasible to change implementation details without breaking users.
- necovek 2y ago
  It's basically impossible to program with no side effects in any general purpose language, and providing a build system language likely requires side-effect rich APIs too (to write files, call external tools, etc). Which is to say that you can perfectly well implement a "clean" build setup with make, and provide an "interface" for parts of the system to tie into. Files and directories are as good an input/output interface as any other, and you can do some simple safeguards with permissions on directories and files. To create a local, intermediate file with make, you create a file in TMPDIR (eg with mktemp or tempfile) and remove it at the end, setting the permissions the way you want for any nested steps not to be able to access it.
kaba0 2y ago
I think one build tool that has the fundamental abstraction well thought out is mill (written in Scala). Every task is just an ordinary function written in plain Scala, outputting something. Other functions can use this output by simply calling that function. In many cases the output will be simply a target directory, that is cached when none of the dependencies have changed.
- codethief 2y ago
  Too bad that mill is JVM-specific. :\

haberman 2y ago

"Build is the first stage of execution" resonates with me too. It follows that build systems should be written in terms of real, encapsulated APIs that separate interface from implementation. Make-like build systems are like programming in assembly language: there is no structure that isolates one build step from another. Make offers no way of creating a "local variable" -- an intermediate file that is hidden from any other build step. The filesystem is like memory: as long as you can figure out the filename (address) somehow, you can read and write whatever you want. Even if you can declare that one build step depends on another, there's usually nothing that prevents a step from reading (or even writing!) files from other build steps that it did not declare a dependency on. What you want is a build system that lets you define real APIs, with truly private state. For example, cc_library() in Bazel (cxx_library() in Buck2) lets you define a C++ library in terms of the srcs/hdrs/etc, but you don't know how it will be implemented. Other rules and targets can consume the outputs of cc_library(), but only the files that are publicly exposed in the returned providers. This model brings true encapsulation to build systems, which makes it much more feasible to change implementation details without breaking users.

necovek 2y ago
It's basically impossible to program with no side effects in any general purpose language, and providing a build system language likely requires side-effect rich APIs too (to write files, call external tools, etc). Which is to say that you can perfectly well implement a "clean" build setup with make, and provide an "interface" for parts of the system to tie into. Files and directories are as good an input/output interface as any other, and you can do some simple safeguards with permissions on directories and files. To create a local, intermediate file with make, you create a file in TMPDIR (eg with mktemp or tempfile) and remove it at the end, setting the permissions the way you want for any nested steps not to be able to access it.

kaba0 2y ago

I think one build tool that has the fundamental abstraction well thought out is mill (written in Scala). Every task is just an ordinary function written in plain Scala, outputting something. Other functions can use this output by simply calling that function. In many cases the output will be simply a target directory, that is cached when none of the dependencies have changed.

codethief 2y ago
Too bad that mill is JVM-specific. :\