3 ms·

No, in most build systems it often can't be expressed precisely. It can be done in a way that will mostly work or be over approximated, though. So having some a

by aseipp 4y ago

No, in most build systems it often can't be expressed precisely. It can be done in a way that will mostly work or be over approximated, though. So having some amount of abstraction capability is really important. Especially when you have a build pipeline involving something like "compile program G -> run G -> compile output of G", the output of G can often have many dependencies that are not statically known, or you have to repeat the logic of the generator program G inside the build system to declare them correctly, which is often error prone. So you normally just either have some incorrect logic where the build will fail -- because G started outputting code that needed "foobar.h" and someone forgot to update the dependency list to include "foobar.h", so now dependencies aren't tracked -- or some wildly conservative dependency graph that rebuilds things way more often than necessary, "the output of G depends on every header file possible." Now imagine that there are basically infinite variations on this theme and the cycle can continue infinitely, and it gets to be a problem. Most people either work on programs small enough where having a conservative approximation of the dependency graph is "fine" and excessive rebuilds are OK, they are just compiling a simple static DAG of known source files, and if they encounter these issues, just structure the project with a little repetition between the between the build time dependencies and the code. Or, they just let the module system in their language of language choice figure it out. These are all completely OK. But it's very expensive to do conservative recompilation and very error prone to do manual repetition in larger projects, and gets very complex in multi-language setups. You can sometimes structure programs to avoid these sorts of problems with enough foresight, but when you can't, or you need these things, you really need them, and the lack of it will really really hurt.

inkyoto 4y ago
> […] and excessive rebuilds are OK. Where does the presupposition of just some build tools, but not the others (if I am reading it correctly), resulting in excessive rebuilds come from? Apart from building a dependencies DAG for a project, the next fundamental responsibility of a build tool is to watch for / detect changed graph nodes and apply relevant build rules (expressed as graph edges) to rebuild only the necessary parts of the graph by traversing the path upward (if the apex node is up at the top). Naturally, if it is a dependency at the very bottom of a very leafy graph that has changed and upstream nodes have a dependency on it, it will trigger a large (of a very large) rebuild of the entire dependency path. But that is true of all build systems, not just select ones, and that is what we need buid tools for. The problem is that many people either can't or don't express dependency rules correctly or dependencies generate build byproducts that are to be dynamically injected into the DAG, and they not properly accounted for in the build rules, and that results in an incorrectly built DAG at the build system run time resulting in excessive rebuilds. In the ideal world, dependency management is relegated to a module system of a programming language. But since long serving incumbents cough cough C, C++ and Java (not until Java 9 anyway) do not have a module system, people have created a kludge (make) and later multiple kludges (autoconf/automake, cmake, maven/ivy etc) to bandaid the problem. However a generic build tool can't account for a permutation of all possible scenarios, all possible programming languages, linkers, pre- and post-processors and whatever else needs to be done to produce a build artefact and excel at it in every possible use case. Some do it better than the others in very specific scenarios but do poorly in some other or fringe use cases. Then there are personal preferences. Therefore, the discourse over which build system is better than the other(s) will never end.
- charcircuit 4y ago
  Modules don't expose file level dependencies which is what you want as a build system.
  - inkyoto 4y ago
    Yes, they do, and I can't think of an operating system (other than OS/400) where a module is not a single file or a collection of files. Could you please reference an implementation where it is done otherwise? The module system is tasked with figuring out dependencies (i.e. files comprising a module), which is true for all programming languages starting from Standard ML through to Rust and anything in between.
palata 4y ago
I did not mean to sound like CMake is perfect for all projects. In some specific situations (typically huge projects), I can totally understand that it becomes a pain. That's why big companies like Google tend to have their own tool. Now it really feels that most people complaining about CMake and using Meson instead are not in this situation. And if they were, it would seem more pragmatic to me to say "CMake is fine for small projects, but at some size we needed something else" than "Meson is obviously, definitely orders of magnitude better because it took me less time learning how to compile my 20-files project".
- aseipp 4y ago
  Okay, except your whole reply to OP started by replying someone who is in this situation and does have this exact problem. Then you go and ask why their solution can't be simple, and when it's explained why, you say, "well, most people don't need this?" ?????? You're talking to people who do need this! And this isn't something that ultra-massive billion line projects singularly suffer from. You can easily suffer from all these issues and more in small and big software projects alike. I've suffered from these problems plenty in FOSS and commercial software that were small and large. "Computer program that generates another computer program" is actually a pretty normal and fine way to build software. But when compilation is expensive, these things matter, and you can't keep making excuses about it. A build system should accurately track dependencies and rebuild things. It's that simple. You can say that running two commands is OK or whatever, but don't try to imply that people who want that criteria -- accurate deps and rebuilding -- aren't trying to solve problems they actually have, and then ditch the example when it becomes inconvenient to you that they are.
  - palata 4y ago
    Well my issue is that I have used CMake quite a lot, with generators, in non-trivial projects (both open source and proprietary), and I have always been fine with autotools, cmake and meson. I don't try to have cmake do too much, e.g. I don't run the generation (or python scripts or custom functions) from cmake, because it quickly becomes a mess and running the steps manually / in a script is usually easy. I haven't seen a project yet where the cmake config is not a mess (because let's be honest, most CMakeLists out there are pretty bad) and where this is clearly a limitation of cmake (and not of meson). Happy to get an actual open source example where that's clearly the case, though.

by aseipp 4y ago

inkyoto 4y ago

> […] and excessive rebuilds are OK. Where does the presupposition of just some build tools, but not the others (if I am reading it correctly), resulting in excessive rebuilds come from? Apart from building a dependencies DAG for a project, the next fundamental responsibility of a build tool is to watch for / detect changed graph nodes and apply relevant build rules (expressed as graph edges) to rebuild only the necessary parts of the graph by traversing the path upward (if the apex node is up at the top). Naturally, if it is a dependency at the very bottom of a very leafy graph that has changed and upstream nodes have a dependency on it, it will trigger a large (of a very large) rebuild of the entire dependency path. But that is true of all build systems, not just select ones, and that is what we need buid tools for. The problem is that many people either can't or don't express dependency rules correctly or dependencies generate build byproducts that are to be dynamically injected into the DAG, and they not properly accounted for in the build rules, and that results in an incorrectly built DAG at the build system run time resulting in excessive rebuilds. In the ideal world, dependency management is relegated to a module system of a programming language. But since long serving incumbents cough cough C, C++ and Java (not until Java 9 anyway) do not have a module system, people have created a kludge (make) and later multiple kludges (autoconf/automake, cmake, maven/ivy etc) to bandaid the problem. However a generic build tool can't account for a permutation of all possible scenarios, all possible programming languages, linkers, pre- and post-processors and whatever else needs to be done to produce a build artefact and excel at it in every possible use case. Some do it better than the others in very specific scenarios but do poorly in some other or fringe use cases. Then there are personal preferences. Therefore, the discourse over which build system is better than the other(s) will never end.

charcircuit 4y ago
Modules don't expose file level dependencies which is what you want as a build system.
- inkyoto 4y ago
  Yes, they do, and I can't think of an operating system (other than OS/400) where a module is not a single file or a collection of files. Could you please reference an implementation where it is done otherwise? The module system is tasked with figuring out dependencies (i.e. files comprising a module), which is true for all programming languages starting from Standard ML through to Rust and anything in between.

palata 4y ago

I did not mean to sound like CMake is perfect for all projects. In some specific situations (typically huge projects), I can totally understand that it becomes a pain. That's why big companies like Google tend to have their own tool. Now it really feels that most people complaining about CMake and using Meson instead are not in this situation. And if they were, it would seem more pragmatic to me to say "CMake is fine for small projects, but at some size we needed something else" than "Meson is obviously, definitely orders of magnitude better because it took me less time learning how to compile my 20-files project".

aseipp 4y ago
Okay, except your whole reply to OP started by replying someone who is in this situation and does have this exact problem. Then you go and ask why their solution can't be simple, and when it's explained why, you say, "well, most people don't need this?" ?????? You're talking to people who do need this! And this isn't something that ultra-massive billion line projects singularly suffer from. You can easily suffer from all these issues and more in small and big software projects alike. I've suffered from these problems plenty in FOSS and commercial software that were small and large. "Computer program that generates another computer program" is actually a pretty normal and fine way to build software. But when compilation is expensive, these things matter, and you can't keep making excuses about it. A build system should accurately track dependencies and rebuild things. It's that simple. You can say that running two commands is OK or whatever, but don't try to imply that people who want that criteria -- accurate deps and rebuilding -- aren't trying to solve problems they actually have, and then ditch the example when it becomes inconvenient to you that they are.
- palata 4y ago
  Well my issue is that I have used CMake quite a lot, with generators, in non-trivial projects (both open source and proprietary), and I have always been fine with autotools, cmake and meson. I don't try to have cmake do too much, e.g. I don't run the generation (or python scripts or custom functions) from cmake, because it quickly becomes a mess and running the steps manually / in a script is usually easy. I haven't seen a project yet where the cmake config is not a mess (because let's be honest, most CMakeLists out there are pretty bad) and where this is clearly a limitation of cmake (and not of meson). Happy to get an actual open source example where that's clearly the case, though.