5 ms·

why are there so many build systems? it's understandable why programmers have opinions about aesthetics regarding their IDE, language, or framework of choice,

why are there so many build systems? it's understandable why programmers have opinions about aesthetics regarding their IDE, language, or framework of choice, but what is there to be opinionated about with a build system?

woadwarrior01 8y ago
Because Google didn't open source Blaze soon enough, and Xooglers who moved to other companies wanted something similar and ended up building clones (Buck and Pants come to mind). Eventually, Google did release an open source version of Blaze called Bazel, but by then the eco-system was already rife with many other similar but slightly different build systems.
amelius 8y ago
I think it boils down to taste, just as there are many languages which have basically the same expressive power. Similarly, there are many shells like bash, tcsh, ...
pytester 8y ago
It's a hard thing to do right despite looking deceptively simple.
- blattimwind 8y ago
  Well many build systems don't even try to work properly, many new ones included. E.g. anything using timestamps is unequivocally an incorrect build system.
  - ek750 8y ago
    Can you explain why? I’d be interested to learn more.
    - lhorie 8y ago
      In principle, you want the output of a build to always be the same if the inputs are the same. In terms of correctness, you want "sameness" to be defined in terms of the contents of the input, not their timestamp, since timestamps can be easily be changed inadvertently by things like `touch`, 3rd party tools, etc. Also, relying on timestamps could pose problems for caching/checksums/etc if they are printed into any transitive dependency of a build pipeline.
      - ithkuil 8y ago
        Furthermore, timestamp precision is still problematic. Some filesystems still have one second granularity, but even if you use a filesystems capable of storing sub-millisecond timrstamps, that doesn't mean commands that get invoked as part of the build process (e.g. cp -p) invoke the right APIs that make use of such precision. e.g. see the doc for the .LOW_RESOLUTION_TIME directive of GNU make
    - ithkuil 8y ago
      Apart from issues pertaining timestamp granularity and clock skew, there is a problem that is technically not about timestamps but is related since it's often a consequence of the dependency data model employed by that class of build systems. Consider this makefile rule: foo.a: $(patsubst %.c,%.o,$(wildcard *.c)) Now, the foo.a target will be considered as stale if any of timestamps of it's dependencies is newer than foo.a But what if you remove a .c file? As build systems like make don't capture a fingerprint of the names of the inputs of a built (let alone the hash of their contents) it's very hard for them to deal with that case correctly. Correctness is important otherwise your trust in the incremental builds quicky erodes and you start doing clean builds all the times you get an error, just in case.
      - qznc 8y ago
        According to Apenwarr (redo implementer) claims [0] that checksums are not necessary, if you extend the mtime with size, inode number, file mode, owner uid, owner gid, and (targets only) the sequence number of the last time it was built. The blog post [0] also contains arguments against checksums: Sometimes building a target has side effects. Checksumming every output after building it is somewhat slow. Checksumming every input file before building is very slow. [0] https://apenwarr.ca/log/20181113 https://apenwarr.ca/log/20181113
gbrown_ 8y ago
https://xkcd.com/927/ https://xkcd.com/927/
- entelechy 8y ago
  The C++ community started a comparison of various buildsystem a while ago: https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWwbeg-HCpLkNiFJPO4Rdw/edit# https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWw...
falsedan 8y ago
Because it’s easy to implement something that does 80% of the features, and also it’s easier than learning all the gnarly features of make Last year I learned about .PRECIOUS and it was awful
detaro 8y ago
- configuration format/language - system requirements - convention vs configuration - flexibility vs predefined paths - tradeoff features vs their complexity (parallelism, caching, ...) - level of integration with specific other tooling (VCS, languages, package managers, ...)
jasode 8y ago
>, but what is there to be opinionated about with a build system? I think you're saying this because your idea of a "build system" is a very simple set of sequential steps from known source code input files to an output binary. The differing opinions come in when the "build system" includes various contradictory philosophies of how to configure and specify the building of complex software. Different opinions on: - syntax : should build config be XML, or JSON, or YAML, or custom syntax? E.g. Ant and Maven used XML but Gradle does not - dependencies search: should build system implicitly find and add relevant dependencies? Or should programmer explicitly specify each one? - reproducible builds (version pinning) vs auto-updated dependencies. - should the build system be "smart" about cross-platform differences? If yes, you end up with complicated build tools like GNU Autoconf "configure" bash script, or CMake's complex syntax. - should build system do optimization and bundling tasks that's not strictly limited to "compile" steps? Some Javascript build tools try to eliminate redundant or unused js code to make downloads smaller. - etc, etc. I also recommend reading the blog post "So you want to write a package manager"[1] to get an idea of how complicated a build system can get. The title says "package manager" but much of the material is also about build systems. The bottom line is that reasonable people can disagree on the priorities and therefore, you can't create The One & Only Build System to End All Other Build Systems. [1] https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
- Aeolun 8y ago
  I feel like there is a ‘best’ answer to most if not all of these questions, and the build system should standardize on whatever that best practice is. It’s great that Timmy thinks an obscure extension of YAML is the best way to define build configs, but all the rest of the engineers use JSON, so that is what we use.
abathologist 8y ago
Building and deploying software is an unsolved problem and this bottleneck becomes increasingly costly and painful as the digital revolution accelerates.
foobarbazetc 8y ago
Because it’s fun to build them and “do it right”... :)
geezerjay 8y ago
> why are there so many build systems? Perhaps because the incumbent/popular build systems are far from being the best solution for the job, particularly when compared with build systems available for other technologies. For instance, in C and C++ land the incumbents are still hand-written Makefiles, autotools, or cmake, which leave much to be desired.

woadwarrior01 8y ago

Because Google didn't open source Blaze soon enough, and Xooglers who moved to other companies wanted something similar and ended up building clones (Buck and Pants come to mind). Eventually, Google did release an open source version of Blaze called Bazel, but by then the eco-system was already rife with many other similar but slightly different build systems.

amelius 8y ago

I think it boils down to taste, just as there are many languages which have basically the same expressive power. Similarly, there are many shells like bash, tcsh, ...

pytester 8y ago

It's a hard thing to do right despite looking deceptively simple.

blattimwind 8y ago
Well many build systems don't even try to work properly, many new ones included. E.g. anything using timestamps is unequivocally an incorrect build system.
- ek750 8y ago
  Can you explain why? I’d be interested to learn more.
  - lhorie 8y ago
    In principle, you want the output of a build to always be the same if the inputs are the same. In terms of correctness, you want "sameness" to be defined in terms of the contents of the input, not their timestamp, since timestamps can be easily be changed inadvertently by things like `touch`, 3rd party tools, etc. Also, relying on timestamps could pose problems for caching/checksums/etc if they are printed into any transitive dependency of a build pipeline.
    - ithkuil 8y ago
      Furthermore, timestamp precision is still problematic. Some filesystems still have one second granularity, but even if you use a filesystems capable of storing sub-millisecond timrstamps, that doesn't mean commands that get invoked as part of the build process (e.g. cp -p) invoke the right APIs that make use of such precision. e.g. see the doc for the .LOW_RESOLUTION_TIME directive of GNU make
  - ithkuil 8y ago
    Apart from issues pertaining timestamp granularity and clock skew, there is a problem that is technically not about timestamps but is related since it's often a consequence of the dependency data model employed by that class of build systems. Consider this makefile rule: foo.a: $(patsubst %.c,%.o,$(wildcard *.c)) Now, the foo.a target will be considered as stale if any of timestamps of it's dependencies is newer than foo.a But what if you remove a .c file? As build systems like make don't capture a fingerprint of the names of the inputs of a built (let alone the hash of their contents) it's very hard for them to deal with that case correctly. Correctness is important otherwise your trust in the incremental builds quicky erodes and you start doing clean builds all the times you get an error, just in case.
    - qznc 8y ago
      According to Apenwarr (redo implementer) claims [0] that checksums are not necessary, if you extend the mtime with size, inode number, file mode, owner uid, owner gid, and (targets only) the sequence number of the last time it was built. The blog post [0] also contains arguments against checksums: Sometimes building a target has side effects. Checksumming every output after building it is somewhat slow. Checksumming every input file before building is very slow. [0] https://apenwarr.ca/log/20181113 https://apenwarr.ca/log/20181113

gbrown_ 8y ago

https://xkcd.com/927/ https://xkcd.com/927/

entelechy 8y ago
The C++ community started a comparison of various buildsystem a while ago: https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWwbeg-HCpLkNiFJPO4Rdw/edit# https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWw...

falsedan 8y ago

Because it’s easy to implement something that does 80% of the features, and also it’s easier than learning all the gnarly features of make Last year I learned about .PRECIOUS and it was awful

detaro 8y ago

- configuration format/language - system requirements - convention vs configuration - flexibility vs predefined paths - tradeoff features vs their complexity (parallelism, caching, ...) - level of integration with specific other tooling (VCS, languages, package managers, ...)

jasode 8y ago

>, but what is there to be opinionated about with a build system? I think you're saying this because your idea of a "build system" is a very simple set of sequential steps from known source code input files to an output binary. The differing opinions come in when the "build system" includes various contradictory philosophies of how to configure and specify the building of complex software. Different opinions on: - syntax : should build config be XML, or JSON, or YAML, or custom syntax? E.g. Ant and Maven used XML but Gradle does not - dependencies search: should build system implicitly find and add relevant dependencies? Or should programmer explicitly specify each one? - reproducible builds (version pinning) vs auto-updated dependencies. - should the build system be "smart" about cross-platform differences? If yes, you end up with complicated build tools like GNU Autoconf "configure" bash script, or CMake's complex syntax. - should build system do optimization and bundling tasks that's not strictly limited to "compile" steps? Some Javascript build tools try to eliminate redundant or unused js code to make downloads smaller. - etc, etc. I also recommend reading the blog post "So you want to write a package manager"[1] to get an idea of how complicated a build system can get. The title says "package manager" but much of the material is also about build systems. The bottom line is that reasonable people can disagree on the priorities and therefore, you can't create The One & Only Build System to End All Other Build Systems. [1] https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...

Aeolun 8y ago
I feel like there is a ‘best’ answer to most if not all of these questions, and the build system should standardize on whatever that best practice is. It’s great that Timmy thinks an obscure extension of YAML is the best way to define build configs, but all the rest of the engineers use JSON, so that is what we use.

abathologist 8y ago

Building and deploying software is an unsolved problem and this bottleneck becomes increasingly costly and painful as the digital revolution accelerates.

foobarbazetc 8y ago

Because it’s fun to build them and “do it right”... :)

geezerjay 8y ago

> why are there so many build systems? Perhaps because the incumbent/popular build systems are far from being the best solution for the job, particularly when compared with build systems available for other technologies. For instance, in C and C++ land the incumbents are still hand-written Makefiles, autotools, or cmake, which leave much to be desired.