11 ms·

Make your own build system

107 points by jstimpfle 9y ago

RcouF1uZ4gsC 9y ago
> Although it adds to the code that must be maintained as part of the project, I think that having a custom build infrastructure that can grow with the project is a good idea for medium-sized and larger projects. Please don't do this for open source projects. Other people may want to use your code, but they generally don't want yet another build system. There are already a lot of build systems that are building very complicated projects. Surely, one of these existing, widely used build systems can be made to work for your project.
- jstimpfle 9y ago
  TBH I don't have many fond memories of open-source build systems. Any widely used build system (that probably <10% of the developers know very well, anyway) is not going to help if the build realized with it is broken (too flexible, too confusing, too optimistic about others' environments). And frankly most uses of those build systems in the wild are "give up control and pray". If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. And the build script just needs to be run. It takes no arguments, so I don't think there is a problem with that.
  - rout39574 9y ago
    I think that your build would be much more readable as make. Building object code and libraries from C is.. kind of well traveled ground? Making an idiosyncratic system is satisfying, but the usefulness of your work is decreased if it's decorated with locally invented barriers to comprehension. Rock on; scratch your itches.
    - jstimpfle 9y ago
      I don't entirely disagree. If you only compile C files this is not for you. This is about exploring what building blocks it takes to also support building all the custom things you need besides that in a larger project.
  - Ace17 9y ago
    > If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. This misses the point IMHO. In practice, whatever complex the build system is, it's generally easy to add a new source file: simply grep for an existing source file name, find the build files that refers to it, and insert a similar line for your new file. > And the build script just needs to be run. It takes no arguments, so I don't think there is a problem with that. What about: cross compilation? ccache injection? verbose builds? parallelism control? installation prefix? Try being a package maintainer for a GNU/Linux distribution, where all of the above needs to be kept under control. You will despise custom build systems. Established build systems (like GNU Autoconf (e.g gcc), and, to some extent, custom Autoconf (ffmpeg, libvpx)) handle all these cases pretty well, and provide fixed commande-line options (--prefix, --host) to control them. This means the project author doesn't get to decide how the option will be named and how it will work. This make a huge difference if you're trying to packages for your distrib dozens of projects from dozens of various authors. With a custom build system, the best case scenario is you have to re-learn all these options for each project. The worst case scenario is: one project doesn't support some use case (e.g cross-compiling), so you have to maintain a patch that hacks into the build files to add the behavior your need.
    - lolikoisuru 9y ago
      >What about: cross compilation? ccache injection? verbose builds? parallelism control? installation prefix? Try being a package maintainer for a GNU/Linux distribution, where all of the above needs to be kept under control. You will despise custom build systems. None of those are features that developers care about. They are only of interest to package maintainers, systems administrators and other lunatics.
      - pleasecalllater 9y ago
        Yea... how come that "I don't care" became a virtue.
        oblio 9y ago
  - lolikoisuru 9y ago
    >If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. If you write a generic makefile the uninitiated collaborator doesn't have to do anything to add new source files.
    - jstimpfle 9y ago
      You can setup any build that way (no need to use Make) but I don't think that's a good idea.
      - MereInterest 9y ago
        I'd be interested to hear what you think that way. In my mind, if a file exists within the src directory but is not compiled as part of the project, its existence is lying to the developer. Keeping a separate list of files to be included makes it possible to lie to the developer. Therefore, wildcard builds should be used, so the list of files in the src directory and the list of files being compiled are one and the same.
        jstimpfle 9y ago
        I like to have an official place where the build rules and artifacts are clearly laid out. I don't think file system directories are a good place to put the truth. For one thing, parsing the information from the directory contents means relying on out-of-band data. For another, directories often contain files that "should not be there". Or maybe you accidentally delete an important file. The build couldn't detect the mistake.
    - imtringued 9y ago
      That's the problem with make and why there are so many build systems built on top of it. Make alone isn't enough. It actually doesn't handle this case at all unless you use a compiler (-MMD) or any other external tool to basically generate the makefile. Otherwise you have to manage the source files and header dependencies yourself.
sushisource 9y ago
So he re-built Gradle, or Buck, or Bazel, or any number of roughly similar it-runs-a-dag build tools? Not sure I buy that his problem was real. Gradle's the only one I'm meaningfully familiar with and making custom commands over a dag is a breeze.
- jstimpfle 9y ago
  The "run the DAG" part is very small and simple (as you can verify). I don't think there is a good argument against "rebuilding" that. As I explained my motivation was primarily keeping my build description in my own datastructures (see the text database). IMO that's a very worthwhile goal. I then went on to see what it takes to NIH everything, and it turns out that detecting what files need to be rebuilt is hairy, especially considering transitive C includes (which requires a dependency cache). But there is no one solution to these problems, and I do think there is some value in controlling that logic. And if you get the concepts right, it can be done. I've divided the code into loosely coupled building blocks. That hopefully makes it easier to understand. It should also be possible to recombine these blocks to make different build systems.
  - kimikimi 9y ago
    FWIW there's a scope threshold that many projects don't always have to cross that justifies using a "real" build tool - and it has to do with those "hairy" parts of file detection and automatically finding dependencies in the shared environment. If the project allows it, though, your dependencies can be shrunk down until there's no issue. This is far more the case with target languages that already understand modules (e.g. Rust or Python versus C or JS), or "one big framework" projects where you are relying on linking in the one framework and having it do all the heavy lifting on the build side of things. Actually customizing a build substantially beyond that, tends to be the case more often when you have something like a compiler within your own pipeline, like automatically transforming and clipping source art assets into optimized, UI-ready form. With those systems there is a lot to be said for a simple script that you can debug.
  - sushisource 9y ago
    I suspect you could (continuing the gradle example) write gradle plugins that would accomplish what you want to do as far as using your own data structures goes. Seems to me though the majority of what's in that textual database is standard C compile/link stuff that'd be handled fine by https://guides.gradle.org/building-c-executables/ https://guides.gradle.org/building-c-executables/
boris 9y ago
Writing your own, general-purpose build system is a hard, multi-year project (spoken from experience[1]). [1] https://build2.org https://build2.org
- adrianratnapala 9y ago
  Ahh, but you don't need a general-purpose one do you?
- Awtem 9y ago
  If it takes you too long, maybe you're doing it wrong then? Are you sure you are solving a real problem with it? I don't mean to insult you, so please don't take this the wrong way... The Point I want to make is this: Is it even a reasonable project to build a build system that does everything for everybody? It would appear -- from plenty of evidence around in most open-source projects -- that mostly each of them had to work around at least a couple of restrictions in their off-the-shelf build-system, and I haven't seen a reasonable sized project that got away without additional scripting. So if the developer already has the need for custom scripting, why not take python? And if it Takes less Time+LOC to roll your own project-customized build-system than it would take you to integrate a "general-purpose" build-system... There is a very strong case for a small python script like the one presented here.
Awtem 9y ago
This is awesome! Instead of getting all mixed up in the discussion about which of the existing build-systems is best, and instead of trying to figure out how to work around all of the (mostly arbitrary from user perspective) restrictions, it is a very rational and pragmatic choice to just get it done and make it do what you want.
- jstimpfle 9y ago
  Let me be honest, the last step of doing all myself was quite a time sink over the last 3 weeks. But in the end I've learned something that will be of use in the future. I don't think this applies equally to the time I've put in off-the-shelf systems.
  - Awtem 9y ago
    I completely agree! All the hours (more like days/weeks) I already spent trying to make CMake do what I want... It only strengthens my dependency on CMake in the end :-/
ndh2 9y ago
The biggest problem I see with the article is that you didn't say exactly what you're doing with this build system. And this gets people's imagination going. What is the scope? Is this an open source project? How many people are working with it? To me it looks like this is a personal project that only you are working on. I don't see any problem using this approach. I mean, I use a batch file as my build "system" for my personal projects. But for anything else it would be worth discussing the trade-offs versus using one of the available build systems. I didn't understand how this works: scanCmd = 'scanincludes-%s' %(oPath). What is this scanincludes thing?
- jstimpfle 9y ago
  It's just supporting my own project, and I'm the only person working on the build and the project. The listed build description is all I build with it currently. > scanCmd = 'scanincludes-%s' %(oPath) I have each node in the DAG identified by a unique string, including command nodes. This one is a name I make up for the node that is responsible for rescanning the dependencies of a particular compilation (the compilation is identified by the .o path)
evincarofautumn 9y ago
I once used Shake[1], a Haskell library/EDSL for expressing and running build rules, to make a build system for work. I highly recommend it over rolling your own, if you need custom build tooling at all. Our use case was a developer tool that would take in project files, sources, and assets, and bundle them for various platforms, push code to devices, or launch simulators for you. Notably, Shake made it fairly easy to ensure that we never needed to implement a “clean” command—all dependencies were correctly tracked. Having strong static types and the Haskell library ecosystem available was a huge win for convenience and correctness over something like Make. Shake makes it easy to do concurrent builds with full or limited parallelism, minimise incremental build time, write multi-stage rules where e.g. you need to run the compiler to calculate the dependencies of a file, and properly handle commands that generate multiple output files. Nowadays it even supports progress reporting, profiling, and some linting of build rules. [1]: https://shakebuild.com/ https://shakebuild.com/
MatzeBraun 9y ago
Nice to see a minimal version of a system. In my experience the biggest (or most annoying?) problem though is not setting up the dependencies and executing the steps along the DAG but doing the configuration part of a build: What libraries are installed on the system, giving the user ways to opt in/out of build-time features, figuring out how to build shared libraries on the target OS, how to crosscompile, there’s tons of of these that may need solving depending on your project. ninja does a fantastic job about handling the core and leaving all the other annoying things for someone else to solve (like cmake with the ninja backend :)
evmar 9y ago
This is neat! If you're interested, Ninja (https://ninja-build.org/ https://ninja-build.org/) is designed exactly to provide the dependency-graph execution underneath a program like yours that has a project-local higher-level description of the build goals.
- AstralStorm 9y ago
  Ninja is a very specific tool that does only one thing and that only somewhat well enough, though it is fast. Even cmake has trouble employing it for certain generated file cases.
HaoZeke 9y ago
I think for most projects, contributions are hard enough to come by without having to factor in a new build system. Honestly even bazel isn't built with bazel. (it can be, but they do that as a test) It's like rolling a whole new project into your existing ones... For no clear reason. Perhaps it's a learning experience, however there's no reason to hobble a project with it. Thankfully, I doubt most established projects would suddenly switch to a custom build system. Also there are plenty of auxiliary tools developed for build systems which are at best, difficult to reproduce. If one needs to improve a build system... Try a pull request?
- ndh2 9y ago
  Yeah, about that last sentence. Are you actually a contributor to an existing build system? CMake? Make? Bazel? Anything?
  - maccard 9y ago
    I agree with the sentence. I've contributed to both cmake and gradle to fix very specific pain points that I've experienced. It's definitely been less work than writing an entire build system because I don't like how big and complex the existing systems are.
- lucideer 9y ago
  > having to factor in a new build system. > there's no reason to hobble a project with it. I think this underestimates the complexity and overhead of using and configuring some* existing build systems, and also overestimates the complexity of running (no need to edit the build config for most contributions to the project, provided the build system works well) a custom build system. Established build systems are often complex due to being heavily generalised and configurable. I've seen projects with build configs that are longer and more complex than a custom build system for that project could be. > there are plenty of auxiliary tools developed for build systems This is the key comment. You can create your own custom build system while using lower level build-related libraries. There's no need to reinvent the wheel entirely. * I said "some" existing build systems: I'm sure some are simpler than others, so this may not apply universally.
reacweb 9y ago
I work on a project where a huge build system has been developed. It uses the same source to build a dynamic library, the same library in static and many test programs. It handles well dependencies whatever the directory structure. It is developed in gnumake. I think you will be less criticized if you use gnumake instead of python. It has all the features needed for dynamic generation of dependency rules. And if someone want to improve the build system, a good knowledge of gnumake seems a reasonable prerequisite. The main defect is that it needs 10s before starting any compilation.
- jstimpfle 9y ago
  However, a few disadvantages to Make: Only files and timestamps means that the granularity of your project's modules needs to be at least as fine as required by the build (I think that's one reason why C builds are slow: Too many files written). Pattern matches are not enough IMO. The scripting language is bad for advanced things. It's like implementing a large project in shell code. Having the full power of Python really is something else. So I was considering generating simple repetitive Make rules. But I wanted to explore what it takes to do the rest myself, as well. Setting up dependencies for Make is not nice, either. Having the code under control means being able to adapt to any requirement. It would be easy for example to convert this into a build server kind of thing. Having a 10s delay is not acceptable to me. I didn't experience that myself though.
  - reacweb 9y ago
    The 10s delay is caused by the correct handling of dependency in a full source tree (around 5000 source files). It is the price of perfectionism. There is no simple repetitive make rules in the build system of the project I am on: all the rules are generated using macros. gnumake is a full scripting language that is dedicated to timestamp dependencies. It can manipulate easily lists of string (as long as they do not contain spaces). What are the advanced things that requires a better language ?
    - jstimpfle 9y ago
      > The 10s delay is caused by the correct handling of dependency in a full source tree (around 5000 source files) There is something awfully wrong if processing 5000 files and cached dependendencies takes 10s. It should take a few ms at most, even if written in a scripting language like Python. I would not be surprised if this is because that build system heavily misuses GNU Make's variable substitution features, instead of being written in a language that features proper data structures (e.g. set, map)
archi42 9y ago
Seems nice from a "been there, done that" point of view. Usually rolling your own solution helps in better understanding the underlying problem. But, as many people pointed out, having a custom build system calls for trouble (eg cross compilation, prefixes, feature control) and often produces a lot of work to maintain or cater to new needs. Source: We roll a package based, distributed build and test system. Most build packages wrap around cmake. A core dev created it, and while he works on the build system, he can't work on the product. And if he ever decides to switch company or has a severe accident, well... (The code is mostly well written, but still the code base is huge since it was adapted to changing needs in the past decade; and while we're not tiny, we're small enough to lack the resources of having him train a second person in the guts of the system - I know probably 70% of it and am regularly lost). If it was up to me, we'd take a look at bazel.
- nrclark 9y ago
  Our company is in the same boat - we have this weird, custom package management system that's a franken-hybrid of shell scripts, vanilla Python, Scons, and CMake. Every time I have to do anything even slightly outside of the norm with it, it breaks badly. And there's this huge sunk-cost that makes it functionally impossible to suggest replacing with anything else. It mostly works but it's getting crustier by the day. So I feel your pain.
zaarn 9y ago
I've implemented build/task systems in two different ways in the past. The first is a bash script that ends in $1 ${2:} or something in that region. Then I can define functions and call them directly from the command line. It's rather protable (if you have bash or I bothered to write POSIX sh compatible) The other was a simple DAG similar to how the author describes it. Each task was a root node and subtasks were nodes connected to it. To figure out which tasks to run I'd simply scan the entire task graph for what has no subtasks and spawn a thread/process for each runnable task. Once a task finished, it's removed from the DAG. Successfully used to run database upgrade tasks (update multiple tables at once if they queries don't depend on eachother) The only thing I ask of a custom build system is that it A) includes the classic "all" and "clean" options B) it doesn't have any dependencies beyond what I can expect from any random distro, C) autodetect libraries in common locations, not the locations on the original dev machine and D) don't require root permissions. ever. (outside "install" and maybe we can insert a rant in here how a lot of packages don't have a "uninstall" or "remove" option, making me track down the files it shat onto my disk)
lolikoisuru 9y ago
>its configuration syntax is awkward This argument and others similar to it are often repeated but I've yet to see any of those people describe what makes it awkward. >and it is restricted to filesystem files and shell commands It'd be pretty easy to make a makefile that depends on say a specific row on a SQL database using dummy files. Also changing the language for recipes is trivial and painfully obvious. If you can't do something as simple as `SHELL := /your/own/interpreter' then why do you think you are qualified to make a build system?
- jstimpfle 9y ago
  It's very easy to explain and there are many examples of bad Makefiles on the internet. Magic variables, unintelligible string replacement routines, unportable extensions. Almost nobody understands the finer points like = vs := and execution order. Using a simple scripting language with well known semantics can be relieving. Maybe try this? http://nibblestew.blogspot.de/2017/12/a-simple-makefile-is-unicorn.html http://nibblestew.blogspot.de/2017/12/a-simple-makefile-is-u... Creating dummy files is inefficient, inelegant, and unmaintainable. Changing SHELL does not solve the inconvenience of having start a new process and having to go through a command-line interface each time. Btw. watch your language, please.
  - lolikoisuru 9y ago
    > Magic variables, unintelligible string replacement routines, unportable extensions. Almost nobody understands the finer points like = vs := and execution order. There is this old saying that is perfect for these kinds of arguments that stem from mere laziness: Read The Fucking Manual. Here is a link for you: https://gnu.org/s/make/manual/ https://gnu.org/s/make/manual/ > Creating dummy files is inefficient, inelegant, and unmaintainable. Sure, if you overuse them. If you end up having more than a few of them you might want to consider simplifying the build a bit. > Changing SHELL does not solve the inconvenience of having start a new process and having to go through a command-line interface each time. I asked a few people to help me understand what you're trying to say here but we weren't able to decipher it. Is your issue that each line is by separate shell by default? You can easily change this with .ONESHELL. Also what do you mean by "command-line interface" that one has to go through? Do you mean the recipes? You don't have to use any traditional unix shell, you could use javascript, python, whatever you fucking want so where exactly does your "command-line interface" come to play? > Btw. watch your language, please. What a pathetic ad hominem. There was no problem with the language in the comment.
daemin 9y ago
Actually the harder thing is building your own data/content build system, where not only the assets change, so does the compiler and tool chain used to build them.
loriverkutya 9y ago
Build your own build system sounds like reinventing the wheel. As a former build build engineer and a nowadays devops, a custom build system would be enough for me not to take a job. Because investing time to learn a custom build system is just wasted time, because you never going to use it again, on the next project or next job or whatever. Also for a project, a custom build system means, that you cannot hire anybody, who knows the build system, however if you are using any established build system, you can. And also, software developers tend to forget, that they are not the only users of a build system, and usually it needs to be integrated into a build&release pipeline.
Annatar 9y ago
From these articles, one can always tell who just hit the “need to really master make” stage in their project.
- gmueckl 9y ago
  You mean the one build system that doesn't know the first thing about header files? The one build system that fails to detect that a header file changed and thus a certain set of source files must now be rebuilt? If there's a case for an existing build system here, the system in question is not make. There are much better systems out there (scons, cmake...). Just my two cents.
  - AstralStorm 9y ago
    Nah, it can spot this if you happen to add those header files to the rules and/or the rule to extract the dependencies using the compiler like autotools and cmake do.
    - gmueckl 9y ago
      So I need an extra preprocessor run to output make rules into temporary files and then I have to hack make rules to make it work. That's what we had 20 years ago. Any decent C/C++ build system these days does not need extra rules for this. And that's how it should be.
      - Annatar 9y ago
        There is no such thing as a “decent C/C++ build system”; that’s a fallacy. Master the compiler front-end’s -M option.
        gmueckl 9y ago
        Your world must be quite small then if all you have are conpilers that unterstand that flag. There are perfectly workable compilers out there without this feature. And with a good build system (thus thread has mentioned a few) you do not have to care. I can show you build acripts for C++ programs that know nothing about the compiler. They work with GCC on Linux, MSVC on Windows and clang on OS X without a single change. Hiw is that not decent?
nrclark 9y ago
I'll probably get some flack for this, but I'd like to put in a plug for autotools (if you don't need to target Windows). Autotools has really come a long way since the bad old days back in the early 2000s. It's got some crusty corner cases, but I like it more and more as I use it. With CMake, everything seems like it's easy to bury yourself in custom macros. And before you know it, you have a build system that's complex, hard to debug, and impossible to re-target. Scons is similar (although at least you can step through the code in Python). I'd encourage anybody to try it out. At least, if your codebase is C or C++ and you don't need to target Windows.
ndh2 9y ago
Here's Casey Muratori on saying "reinvent the wheel". This is in the context of game development, but it sure applies to build systems for C/C++. https://youtu.be/fQeqsn7JJWA?t=140 https://youtu.be/fQeqsn7JJWA?t=140
jstimpfle 9y ago
Oopsie. Must have todo with me enabling ipv6 at request from another guy. Will fix this later when i come back. You can get the descriptio from https://github.com/jstimpfle/learn-opengl/blob/master/build.wsl https://github.com/jstimpfle/learn-opengl/blob/master/build.... as well

107 points by jstimpfle 9y ago

> Although it adds to the code that must be maintained as part of the project, I think that having a custom build infrastructure that can grow with the project is a good idea for medium-sized and larger projects. Please don't do this for open source projects. Other people may want to use your code, but they generally don't want yet another build system. There are already a lot of build systems that are building very complicated projects. Surely, one of these existing, widely used build systems can be made to work for your project.

jstimpfle 9y ago
TBH I don't have many fond memories of open-source build systems. Any widely used build system (that probably <10% of the developers know very well, anyway) is not going to help if the build realized with it is broken (too flexible, too confusing, too optimistic about others' environments). And frankly most uses of those build systems in the wild are "give up control and pray". If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. And the build script just needs to be run. It takes no arguments, so I don't think there is a problem with that.
- rout39574 9y ago
  I think that your build would be much more readable as make. Building object code and libraries from C is.. kind of well traveled ground? Making an idiosyncratic system is satisfying, but the usefulness of your work is decreased if it's decorated with locally invented barriers to comprehension. Rock on; scratch your itches.
  - jstimpfle 9y ago
    I don't entirely disagree. If you only compile C files this is not for you. This is about exploring what building blocks it takes to also support building all the custom things you need besides that in a larger project.
- Ace17 9y ago
  > If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. This misses the point IMHO. In practice, whatever complex the build system is, it's generally easy to add a new source file: simply grep for an existing source file name, find the build files that refers to it, and insert a similar line for your new file. > And the build script just needs to be run. It takes no arguments, so I don't think there is a problem with that. What about: cross compilation? ccache injection? verbose builds? parallelism control? installation prefix? Try being a package maintainer for a GNU/Linux distribution, where all of the above needs to be kept under control. You will despise custom build systems. Established build systems (like GNU Autoconf (e.g gcc), and, to some extent, custom Autoconf (ffmpeg, libvpx)) handle all these cases pretty well, and provide fixed commande-line options (--prefix, --host) to control them. This means the project author doesn't get to decide how the option will be named and how it will work. This make a huge difference if you're trying to packages for your distrib dozens of projects from dozens of various authors. With a custom build system, the best case scenario is you have to re-learn all these options for each project. The worst case scenario is: one project doesn't support some use case (e.g cross-compiling), so you have to maintain a patch that hacks into the build files to add the behavior your need.
  - lolikoisuru 9y ago
    >What about: cross compilation? ccache injection? verbose builds? parallelism control? installation prefix? Try being a package maintainer for a GNU/Linux distribution, where all of the above needs to be kept under control. You will despise custom build systems. None of those are features that developers care about. They are only of interest to package maintainers, systems administrators and other lunatics.
    - pleasecalllater 9y ago
      Yea... how come that "I don't care" became a virtue.
      - oblio 9y ago
- lolikoisuru 9y ago
  >If it's about making it easy for uninitiated collaborators to add new source files, then the listed build description format is pretty self-explanatory to use. If you write a generic makefile the uninitiated collaborator doesn't have to do anything to add new source files.
  - jstimpfle 9y ago
    You can setup any build that way (no need to use Make) but I don't think that's a good idea.
    - MereInterest 9y ago
      I'd be interested to hear what you think that way. In my mind, if a file exists within the src directory but is not compiled as part of the project, its existence is lying to the developer. Keeping a separate list of files to be included makes it possible to lie to the developer. Therefore, wildcard builds should be used, so the list of files in the src directory and the list of files being compiled are one and the same.
      - jstimpfle 9y ago
        I like to have an official place where the build rules and artifacts are clearly laid out. I don't think file system directories are a good place to put the truth. For one thing, parsing the information from the directory contents means relying on out-of-band data. For another, directories often contain files that "should not be there". Or maybe you accidentally delete an important file. The build couldn't detect the mistake.
  - imtringued 9y ago
    That's the problem with make and why there are so many build systems built on top of it. Make alone isn't enough. It actually doesn't handle this case at all unless you use a compiler (-MMD) or any other external tool to basically generate the makefile. Otherwise you have to manage the source files and header dependencies yourself.

sushisource 9y ago

So he re-built Gradle, or Buck, or Bazel, or any number of roughly similar it-runs-a-dag build tools? Not sure I buy that his problem was real. Gradle's the only one I'm meaningfully familiar with and making custom commands over a dag is a breeze.

jstimpfle 9y ago
The "run the DAG" part is very small and simple (as you can verify). I don't think there is a good argument against "rebuilding" that. As I explained my motivation was primarily keeping my build description in my own datastructures (see the text database). IMO that's a very worthwhile goal. I then went on to see what it takes to NIH everything, and it turns out that detecting what files need to be rebuilt is hairy, especially considering transitive C includes (which requires a dependency cache). But there is no one solution to these problems, and I do think there is some value in controlling that logic. And if you get the concepts right, it can be done. I've divided the code into loosely coupled building blocks. That hopefully makes it easier to understand. It should also be possible to recombine these blocks to make different build systems.
- kimikimi 9y ago
  FWIW there's a scope threshold that many projects don't always have to cross that justifies using a "real" build tool - and it has to do with those "hairy" parts of file detection and automatically finding dependencies in the shared environment. If the project allows it, though, your dependencies can be shrunk down until there's no issue. This is far more the case with target languages that already understand modules (e.g. Rust or Python versus C or JS), or "one big framework" projects where you are relying on linking in the one framework and having it do all the heavy lifting on the build side of things. Actually customizing a build substantially beyond that, tends to be the case more often when you have something like a compiler within your own pipeline, like automatically transforming and clipping source art assets into optimized, UI-ready form. With those systems there is a lot to be said for a simple script that you can debug.
- sushisource 9y ago
  I suspect you could (continuing the gradle example) write gradle plugins that would accomplish what you want to do as far as using your own data structures goes. Seems to me though the majority of what's in that textual database is standard C compile/link stuff that'd be handled fine by https://guides.gradle.org/building-c-executables/ https://guides.gradle.org/building-c-executables/

boris 9y ago

Writing your own, general-purpose build system is a hard, multi-year project (spoken from experience[1]). [1] https://build2.org https://build2.org

adrianratnapala 9y ago
Ahh, but you don't need a general-purpose one do you?
Awtem 9y ago
If it takes you too long, maybe you're doing it wrong then? Are you sure you are solving a real problem with it? I don't mean to insult you, so please don't take this the wrong way... The Point I want to make is this: Is it even a reasonable project to build a build system that does everything for everybody? It would appear -- from plenty of evidence around in most open-source projects -- that mostly each of them had to work around at least a couple of restrictions in their off-the-shelf build-system, and I haven't seen a reasonable sized project that got away without additional scripting. So if the developer already has the need for custom scripting, why not take python? And if it Takes less Time+LOC to roll your own project-customized build-system than it would take you to integrate a "general-purpose" build-system... There is a very strong case for a small python script like the one presented here.

Awtem 9y ago

This is awesome! Instead of getting all mixed up in the discussion about which of the existing build-systems is best, and instead of trying to figure out how to work around all of the (mostly arbitrary from user perspective) restrictions, it is a very rational and pragmatic choice to just get it done and make it do what you want.

jstimpfle 9y ago
Let me be honest, the last step of doing all myself was quite a time sink over the last 3 weeks. But in the end I've learned something that will be of use in the future. I don't think this applies equally to the time I've put in off-the-shelf systems.
- Awtem 9y ago
  I completely agree! All the hours (more like days/weeks) I already spent trying to make CMake do what I want... It only strengthens my dependency on CMake in the end :-/

ndh2 9y ago

The biggest problem I see with the article is that you didn't say exactly what you're doing with this build system. And this gets people's imagination going. What is the scope? Is this an open source project? How many people are working with it? To me it looks like this is a personal project that only you are working on. I don't see any problem using this approach. I mean, I use a batch file as my build "system" for my personal projects. But for anything else it would be worth discussing the trade-offs versus using one of the available build systems. I didn't understand how this works: scanCmd = 'scanincludes-%s' %(oPath). What is this scanincludes thing?

jstimpfle 9y ago
It's just supporting my own project, and I'm the only person working on the build and the project. The listed build description is all I build with it currently. > scanCmd = 'scanincludes-%s' %(oPath) I have each node in the DAG identified by a unique string, including command nodes. This one is a name I make up for the node that is responsible for rescanning the dependencies of a particular compilation (the compilation is identified by the .o path)

evincarofautumn 9y ago

I once used Shake[1], a Haskell library/EDSL for expressing and running build rules, to make a build system for work. I highly recommend it over rolling your own, if you need custom build tooling at all. Our use case was a developer tool that would take in project files, sources, and assets, and bundle them for various platforms, push code to devices, or launch simulators for you. Notably, Shake made it fairly easy to ensure that we never needed to implement a “clean” command—all dependencies were correctly tracked. Having strong static types and the Haskell library ecosystem available was a huge win for convenience and correctness over something like Make. Shake makes it easy to do concurrent builds with full or limited parallelism, minimise incremental build time, write multi-stage rules where e.g. you need to run the compiler to calculate the dependencies of a file, and properly handle commands that generate multiple output files. Nowadays it even supports progress reporting, profiling, and some linting of build rules. [1]: https://shakebuild.com/ https://shakebuild.com/

MatzeBraun 9y ago

Nice to see a minimal version of a system. In my experience the biggest (or most annoying?) problem though is not setting up the dependencies and executing the steps along the DAG but doing the configuration part of a build: What libraries are installed on the system, giving the user ways to opt in/out of build-time features, figuring out how to build shared libraries on the target OS, how to crosscompile, there’s tons of of these that may need solving depending on your project. ninja does a fantastic job about handling the core and leaving all the other annoying things for someone else to solve (like cmake with the ninja backend :)

evmar 9y ago

This is neat! If you're interested, Ninja (https://ninja-build.org/ https://ninja-build.org/) is designed exactly to provide the dependency-graph execution underneath a program like yours that has a project-local higher-level description of the build goals.

AstralStorm 9y ago
Ninja is a very specific tool that does only one thing and that only somewhat well enough, though it is fast. Even cmake has trouble employing it for certain generated file cases.

HaoZeke 9y ago

I think for most projects, contributions are hard enough to come by without having to factor in a new build system. Honestly even bazel isn't built with bazel. (it can be, but they do that as a test) It's like rolling a whole new project into your existing ones... For no clear reason. Perhaps it's a learning experience, however there's no reason to hobble a project with it. Thankfully, I doubt most established projects would suddenly switch to a custom build system. Also there are plenty of auxiliary tools developed for build systems which are at best, difficult to reproduce. If one needs to improve a build system... Try a pull request?

ndh2 9y ago
Yeah, about that last sentence. Are you actually a contributor to an existing build system? CMake? Make? Bazel? Anything?
- maccard 9y ago
  I agree with the sentence. I've contributed to both cmake and gradle to fix very specific pain points that I've experienced. It's definitely been less work than writing an entire build system because I don't like how big and complex the existing systems are.
lucideer 9y ago
> having to factor in a new build system. > there's no reason to hobble a project with it. I think this underestimates the complexity and overhead of using and configuring some* existing build systems, and also overestimates the complexity of running (no need to edit the build config for most contributions to the project, provided the build system works well) a custom build system. Established build systems are often complex due to being heavily generalised and configurable. I've seen projects with build configs that are longer and more complex than a custom build system for that project could be. > there are plenty of auxiliary tools developed for build systems This is the key comment. You can create your own custom build system while using lower level build-related libraries. There's no need to reinvent the wheel entirely. * I said "some" existing build systems: I'm sure some are simpler than others, so this may not apply universally.

reacweb 9y ago

I work on a project where a huge build system has been developed. It uses the same source to build a dynamic library, the same library in static and many test programs. It handles well dependencies whatever the directory structure. It is developed in gnumake. I think you will be less criticized if you use gnumake instead of python. It has all the features needed for dynamic generation of dependency rules. And if someone want to improve the build system, a good knowledge of gnumake seems a reasonable prerequisite. The main defect is that it needs 10s before starting any compilation.

jstimpfle 9y ago
However, a few disadvantages to Make: Only files and timestamps means that the granularity of your project's modules needs to be at least as fine as required by the build (I think that's one reason why C builds are slow: Too many files written). Pattern matches are not enough IMO. The scripting language is bad for advanced things. It's like implementing a large project in shell code. Having the full power of Python really is something else. So I was considering generating simple repetitive Make rules. But I wanted to explore what it takes to do the rest myself, as well. Setting up dependencies for Make is not nice, either. Having the code under control means being able to adapt to any requirement. It would be easy for example to convert this into a build server kind of thing. Having a 10s delay is not acceptable to me. I didn't experience that myself though.
- reacweb 9y ago
  The 10s delay is caused by the correct handling of dependency in a full source tree (around 5000 source files). It is the price of perfectionism. There is no simple repetitive make rules in the build system of the project I am on: all the rules are generated using macros. gnumake is a full scripting language that is dedicated to timestamp dependencies. It can manipulate easily lists of string (as long as they do not contain spaces). What are the advanced things that requires a better language ?
  - jstimpfle 9y ago
    > The 10s delay is caused by the correct handling of dependency in a full source tree (around 5000 source files) There is something awfully wrong if processing 5000 files and cached dependendencies takes 10s. It should take a few ms at most, even if written in a scripting language like Python. I would not be surprised if this is because that build system heavily misuses GNU Make's variable substitution features, instead of being written in a language that features proper data structures (e.g. set, map)

archi42 9y ago

Seems nice from a "been there, done that" point of view. Usually rolling your own solution helps in better understanding the underlying problem. But, as many people pointed out, having a custom build system calls for trouble (eg cross compilation, prefixes, feature control) and often produces a lot of work to maintain or cater to new needs. Source: We roll a package based, distributed build and test system. Most build packages wrap around cmake. A core dev created it, and while he works on the build system, he can't work on the product. And if he ever decides to switch company or has a severe accident, well... (The code is mostly well written, but still the code base is huge since it was adapted to changing needs in the past decade; and while we're not tiny, we're small enough to lack the resources of having him train a second person in the guts of the system - I know probably 70% of it and am regularly lost). If it was up to me, we'd take a look at bazel.

nrclark 9y ago
Our company is in the same boat - we have this weird, custom package management system that's a franken-hybrid of shell scripts, vanilla Python, Scons, and CMake. Every time I have to do anything even slightly outside of the norm with it, it breaks badly. And there's this huge sunk-cost that makes it functionally impossible to suggest replacing with anything else. It mostly works but it's getting crustier by the day. So I feel your pain.

zaarn 9y ago

I've implemented build/task systems in two different ways in the past. The first is a bash script that ends in $1 ${2:} or something in that region. Then I can define functions and call them directly from the command line. It's rather protable (if you have bash or I bothered to write POSIX sh compatible) The other was a simple DAG similar to how the author describes it. Each task was a root node and subtasks were nodes connected to it. To figure out which tasks to run I'd simply scan the entire task graph for what has no subtasks and spawn a thread/process for each runnable task. Once a task finished, it's removed from the DAG. Successfully used to run database upgrade tasks (update multiple tables at once if they queries don't depend on eachother) The only thing I ask of a custom build system is that it A) includes the classic "all" and "clean" options B) it doesn't have any dependencies beyond what I can expect from any random distro, C) autodetect libraries in common locations, not the locations on the original dev machine and D) don't require root permissions. ever. (outside "install" and maybe we can insert a rant in here how a lot of packages don't have a "uninstall" or "remove" option, making me track down the files it shat onto my disk)

lolikoisuru 9y ago

>its configuration syntax is awkward This argument and others similar to it are often repeated but I've yet to see any of those people describe what makes it awkward. >and it is restricted to filesystem files and shell commands It'd be pretty easy to make a makefile that depends on say a specific row on a SQL database using dummy files. Also changing the language for recipes is trivial and painfully obvious. If you can't do something as simple as `SHELL := /your/own/interpreter' then why do you think you are qualified to make a build system?

jstimpfle 9y ago
It's very easy to explain and there are many examples of bad Makefiles on the internet. Magic variables, unintelligible string replacement routines, unportable extensions. Almost nobody understands the finer points like = vs := and execution order. Using a simple scripting language with well known semantics can be relieving. Maybe try this? http://nibblestew.blogspot.de/2017/12/a-simple-makefile-is-unicorn.html http://nibblestew.blogspot.de/2017/12/a-simple-makefile-is-u... Creating dummy files is inefficient, inelegant, and unmaintainable. Changing SHELL does not solve the inconvenience of having start a new process and having to go through a command-line interface each time. Btw. watch your language, please.
- lolikoisuru 9y ago
  > Magic variables, unintelligible string replacement routines, unportable extensions. Almost nobody understands the finer points like = vs := and execution order. There is this old saying that is perfect for these kinds of arguments that stem from mere laziness: Read The Fucking Manual. Here is a link for you: https://gnu.org/s/make/manual/ https://gnu.org/s/make/manual/ > Creating dummy files is inefficient, inelegant, and unmaintainable. Sure, if you overuse them. If you end up having more than a few of them you might want to consider simplifying the build a bit. > Changing SHELL does not solve the inconvenience of having start a new process and having to go through a command-line interface each time. I asked a few people to help me understand what you're trying to say here but we weren't able to decipher it. Is your issue that each line is by separate shell by default? You can easily change this with .ONESHELL. Also what do you mean by "command-line interface" that one has to go through? Do you mean the recipes? You don't have to use any traditional unix shell, you could use javascript, python, whatever you fucking want so where exactly does your "command-line interface" come to play? > Btw. watch your language, please. What a pathetic ad hominem. There was no problem with the language in the comment.

daemin 9y ago

Actually the harder thing is building your own data/content build system, where not only the assets change, so does the compiler and tool chain used to build them.

loriverkutya 9y ago

Build your own build system sounds like reinventing the wheel. As a former build build engineer and a nowadays devops, a custom build system would be enough for me not to take a job. Because investing time to learn a custom build system is just wasted time, because you never going to use it again, on the next project or next job or whatever. Also for a project, a custom build system means, that you cannot hire anybody, who knows the build system, however if you are using any established build system, you can. And also, software developers tend to forget, that they are not the only users of a build system, and usually it needs to be integrated into a build&release pipeline.

Annatar 9y ago

From these articles, one can always tell who just hit the “need to really master make” stage in their project.

gmueckl 9y ago
You mean the one build system that doesn't know the first thing about header files? The one build system that fails to detect that a header file changed and thus a certain set of source files must now be rebuilt? If there's a case for an existing build system here, the system in question is not make. There are much better systems out there (scons, cmake...). Just my two cents.
- AstralStorm 9y ago
  Nah, it can spot this if you happen to add those header files to the rules and/or the rule to extract the dependencies using the compiler like autotools and cmake do.
  - gmueckl 9y ago
    So I need an extra preprocessor run to output make rules into temporary files and then I have to hack make rules to make it work. That's what we had 20 years ago. Any decent C/C++ build system these days does not need extra rules for this. And that's how it should be.
    - Annatar 9y ago
      There is no such thing as a “decent C/C++ build system”; that’s a fallacy. Master the compiler front-end’s -M option.
      - gmueckl 9y ago
        Your world must be quite small then if all you have are conpilers that unterstand that flag. There are perfectly workable compilers out there without this feature. And with a good build system (thus thread has mentioned a few) you do not have to care. I can show you build acripts for C++ programs that know nothing about the compiler. They work with GCC on Linux, MSVC on Windows and clang on OS X without a single change. Hiw is that not decent?

nrclark 9y ago

I'll probably get some flack for this, but I'd like to put in a plug for autotools (if you don't need to target Windows). Autotools has really come a long way since the bad old days back in the early 2000s. It's got some crusty corner cases, but I like it more and more as I use it. With CMake, everything seems like it's easy to bury yourself in custom macros. And before you know it, you have a build system that's complex, hard to debug, and impossible to re-target. Scons is similar (although at least you can step through the code in Python). I'd encourage anybody to try it out. At least, if your codebase is C or C++ and you don't need to target Windows.

ndh2 9y ago

Here's Casey Muratori on saying "reinvent the wheel". This is in the context of game development, but it sure applies to build systems for C/C++. https://youtu.be/fQeqsn7JJWA?t=140 https://youtu.be/fQeqsn7JJWA?t=140

jstimpfle 9y ago

Oopsie. Must have todo with me enabling ipv6 at request from another guy. Will fix this later when i come back. You can get the descriptio from https://github.com/jstimpfle/learn-opengl/blob/master/build.wsl https://github.com/jstimpfle/learn-opengl/blob/master/build.... as well