6 ms·

Recovering an ancient project sent me back down the "which build tool should I be using?" rabbit hole. Getting a script to grave dig 57 inconsistently styled K&

Recovering an ancient project sent me back down the "which build tool should I be using?" rabbit hole. Getting a script to grave dig 57 inconsistently styled K&R C files (!) really stresses the build cycle, more than writing new code. First approximation: No one's happy with any build tool, with good reason. Then I discovered Ninja. It's an assembly language for builds, advertised as a back end for higher level tools, but many people script Ninja themselves. For me, 15 lines of Ruby code plus some heredoc boilerplate spawned a 138 line build.ninja file. Back in the day, this computer algebra system used to take an hour to build. Make still needs a frustrating number of seconds. Ninja on all my cores is a correct fraction of a second. Ninja is exceptionally fast, and handles parallelism extraordinarily well. It scales; they use it for Google Chrome. Various tools such as CMake can target Ninja. It is unique for its design goals. Most importantly, looking at Ninja code is not a "Just kill me now!" experience. https://ninja-build.org/ https://ninja-build.org/

grlass 4y ago
+1 for ninja. I first encountered is using Apache TVM, which uses CMake. I was first of all using the default make backend, and it was taking a while. Then the docs said to try adding the `-GNinja` flag to Cmake to build with ninja. I was blown away by how much faster the compilation was, and now try to use it whenever possible.
- faho 4y ago
  One issue is that make will default to only using one job (i.e. one cpu), and you need to pass `-j NUMBER` to make it use more, while Ninja is parallel by default. For my uses, I've not found `ninja` to be much faster than `make -j8` on an 8-core machine. Not that the defaults don't matter, of course.
  - Syzygies 4y ago
    I'm surely also in that range. It took us years to write this project, but 57 C files is a small project. Others find themselves in the range where the build decision-making itself can take seconds, with tools other than ninja. Ninja is pared down for speed; having to declare keywords before I used them made me feel young again. I had to write these build files over again, and I just couldn't stomach the idea of writing another makefile.
- jcelerier 4y ago
  > Then the docs said to try adding the `-GNinja` flag to Cmake to build with ninja. note that nowadays you can just pass --parallel to cmake and it will do "the right thing" whenever possible, c.f. https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-a-project https://cmake.org/cmake/help/latest/manual/cmake.1.html#buil...
jonpalmisc 4y ago
Any chance you can share that Ruby script? (It can be just the script if the source isn’t shareable.) I’ve wanted to explore something like this for a while and having a reference would be nice.
- Syzygies 4y ago
  https://gist.github.com/Syzygies/fe8cf2096fcb91dd11b5c8cf763f9fc7 https://gist.github.com/Syzygies/fe8cf2096fcb91dd11b5c8cf763... The source will be shareable once we get it running again. Other than historical curiosity, people are most likely to use it for assistance in converting old work to the successor program Macaulay2. The script bin/ninja.rb creates two flavors of build.rb, one that keeps going no matter what, writing error files for each exception, and another for use once everything works. One might be able to craft a single file that does both, but I'll want to ditch these complications once we're done with the conversion, and end users won't want to see them. It would have helped me to see more examples like this, learning ninja.
kitd 4y ago
Meson + ninja is the sweet spot for me https://mesonbuild.com/index.html https://mesonbuild.com/index.html
- daptaq 4y ago
  I frequently see meson (and other tools like cmake) using build directories, which is something I don't like. Is there a reason to this that I am not thinking of (besides being able to "rm -rf" everything instead of doing something like "make clean")?
  - dagw 4y ago
    Other than it being cleaner in many ways (all build artefacts in one place, your base directory structure never getting touched, no chance of accidentally checking in build artefacts into your repo), you can have several different build configurations 'active' in parallel (build.release, build.debug, build.cuda etc.) and then activate the right build by just going to the relevant folder and running make.
    - daptaq 4y ago
      > you can have several different build configurations 'active' in parallel (build.release, build.debug, build.cuda etc.) and then activate the right build by just going to the relevant folder and running make. That is an interesting use-case I hadn't heard of before. But I guess if that is really needed, you could also just have multiple checkouts?
  - trasz 4y ago
    FWIW, you can easily have that (separate build directories) with make(1), it’s just that’s it’s not the default.
    - daptaq 4y ago
      Of course, I use make most of the time, but I'm used to having a .o file right next to my .c file.
chubot 4y ago
That's funny, I wrote basically the same thing here a few hours ago. I "replaced" GNU Make with Python + Ninja and it's way better. https://news.ycombinator.com/item?id=32303096 https://news.ycombinator.com/item?id=32303096 And my build is a lot bigger, with ~1000 lines of Python generating ~9000 lines of Ninja. I will probably write a blog post about it at some point, but it supports - build variants - dbg, opt, asan, ubsan - code coverage with HTML output - uftrace instrumentation - malloc and tracing - clang and gcc (because clang coverage is better, ubsan is different, etc.) - test and benchmark binaries
- Syzygies 4y ago
  I'd love to see your ninja files, to see how you handled such things. Rather than complicating my build.ninja files, I handle build variants in my script. Easy enough to output a new variant, probably faster to use, and certainly easier for others to understand. People are more likely to adopt ninja as we did if they see it first as the easiest way to manage a small project. Why bother with CMake or other tools, if one can script in one's favorite language? I'm sick of working with bad build description languages; ninja is going straight to the bare metal.
  - chubot 4y ago
    Sure you can check it out here: https://github.com/oilshell/oil https://github.com/oilshell/oil $ wc -l NINJA* */NINJA* 64 NINJA_config.py ... 260 cpp/NINJA-steps.sh 361 cpp/NINJA_subgraph.py 305 mycpp/NINJA-steps.sh 605 mycpp/NINJA_subgraph.py 1615 total I borrowed the little Python library that Ninja itself uses to bootstrap its Ninja files. There are a few missing things like some textual code generation, but overall it should make sense. It's structured a bit like a lightweight Bazel -- each directory has a NINJA_subgraph.py file that specifies what to build. And then the "actions" are shell scripts in NINJA-steps.sh. At the top of one file it shows the directory structure for variants, which is basically _bin/$compiler-$variant/foo_test and _build/obj/$compiler-$variant/foo_test.o The variants control the compile and link flags: https://github.com/oilshell/oil/blob/master/cpp/NINJA-steps.sh https://github.com/oilshell/oil/blob/master/cpp/NINJA-steps.... Overall this is the best build system I've used :) My pet peeve is having to 'make clean' when changing the variant, and this avoids that, making everything available at once. I'm sure it's reinventing some of the 120K lines of CMakeLists.txt that comes with CMake, but this way I don't have to learn and debug a wonky shell-ish language! One downside of the pattern is that you probably don't want to invoke shell on Windows (which doesn't affect my project). I find that very useful though, so I'm not sure what would replace it for Windows projects. (I guess batch files, but I don't think they're powerful enough)

grlass 4y ago

+1 for ninja. I first encountered is using Apache TVM, which uses CMake. I was first of all using the default make backend, and it was taking a while. Then the docs said to try adding the `-GNinja` flag to Cmake to build with ninja. I was blown away by how much faster the compilation was, and now try to use it whenever possible.

faho 4y ago
One issue is that make will default to only using one job (i.e. one cpu), and you need to pass `-j NUMBER` to make it use more, while Ninja is parallel by default. For my uses, I've not found `ninja` to be much faster than `make -j8` on an 8-core machine. Not that the defaults don't matter, of course.
- Syzygies 4y ago
  I'm surely also in that range. It took us years to write this project, but 57 C files is a small project. Others find themselves in the range where the build decision-making itself can take seconds, with tools other than ninja. Ninja is pared down for speed; having to declare keywords before I used them made me feel young again. I had to write these build files over again, and I just couldn't stomach the idea of writing another makefile.
jcelerier 4y ago
> Then the docs said to try adding the `-GNinja` flag to Cmake to build with ninja. note that nowadays you can just pass --parallel to cmake and it will do "the right thing" whenever possible, c.f. https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-a-project https://cmake.org/cmake/help/latest/manual/cmake.1.html#buil...

jonpalmisc 4y ago

Any chance you can share that Ruby script? (It can be just the script if the source isn’t shareable.) I’ve wanted to explore something like this for a while and having a reference would be nice.

Syzygies 4y ago
https://gist.github.com/Syzygies/fe8cf2096fcb91dd11b5c8cf763f9fc7 https://gist.github.com/Syzygies/fe8cf2096fcb91dd11b5c8cf763... The source will be shareable once we get it running again. Other than historical curiosity, people are most likely to use it for assistance in converting old work to the successor program Macaulay2. The script bin/ninja.rb creates two flavors of build.rb, one that keeps going no matter what, writing error files for each exception, and another for use once everything works. One might be able to craft a single file that does both, but I'll want to ditch these complications once we're done with the conversion, and end users won't want to see them. It would have helped me to see more examples like this, learning ninja.

kitd 4y ago

Meson + ninja is the sweet spot for me https://mesonbuild.com/index.html https://mesonbuild.com/index.html

daptaq 4y ago
I frequently see meson (and other tools like cmake) using build directories, which is something I don't like. Is there a reason to this that I am not thinking of (besides being able to "rm -rf" everything instead of doing something like "make clean")?
- dagw 4y ago
  Other than it being cleaner in many ways (all build artefacts in one place, your base directory structure never getting touched, no chance of accidentally checking in build artefacts into your repo), you can have several different build configurations 'active' in parallel (build.release, build.debug, build.cuda etc.) and then activate the right build by just going to the relevant folder and running make.
  - daptaq 4y ago
    > you can have several different build configurations 'active' in parallel (build.release, build.debug, build.cuda etc.) and then activate the right build by just going to the relevant folder and running make. That is an interesting use-case I hadn't heard of before. But I guess if that is really needed, you could also just have multiple checkouts?
- trasz 4y ago
  FWIW, you can easily have that (separate build directories) with make(1), it’s just that’s it’s not the default.
  - daptaq 4y ago
    Of course, I use make most of the time, but I'm used to having a .o file right next to my .c file.

chubot 4y ago

That's funny, I wrote basically the same thing here a few hours ago. I "replaced" GNU Make with Python + Ninja and it's way better. https://news.ycombinator.com/item?id=32303096 https://news.ycombinator.com/item?id=32303096 And my build is a lot bigger, with ~1000 lines of Python generating ~9000 lines of Ninja. I will probably write a blog post about it at some point, but it supports - build variants - dbg, opt, asan, ubsan - code coverage with HTML output - uftrace instrumentation - malloc and tracing - clang and gcc (because clang coverage is better, ubsan is different, etc.) - test and benchmark binaries

Syzygies 4y ago
I'd love to see your ninja files, to see how you handled such things. Rather than complicating my build.ninja files, I handle build variants in my script. Easy enough to output a new variant, probably faster to use, and certainly easier for others to understand. People are more likely to adopt ninja as we did if they see it first as the easiest way to manage a small project. Why bother with CMake or other tools, if one can script in one's favorite language? I'm sick of working with bad build description languages; ninja is going straight to the bare metal.
- chubot 4y ago
  Sure you can check it out here: https://github.com/oilshell/oil https://github.com/oilshell/oil $ wc -l NINJA* */NINJA* 64 NINJA_config.py ... 260 cpp/NINJA-steps.sh 361 cpp/NINJA_subgraph.py 305 mycpp/NINJA-steps.sh 605 mycpp/NINJA_subgraph.py 1615 total I borrowed the little Python library that Ninja itself uses to bootstrap its Ninja files. There are a few missing things like some textual code generation, but overall it should make sense. It's structured a bit like a lightweight Bazel -- each directory has a NINJA_subgraph.py file that specifies what to build. And then the "actions" are shell scripts in NINJA-steps.sh. At the top of one file it shows the directory structure for variants, which is basically _bin/$compiler-$variant/foo_test and _build/obj/$compiler-$variant/foo_test.o The variants control the compile and link flags: https://github.com/oilshell/oil/blob/master/cpp/NINJA-steps.sh https://github.com/oilshell/oil/blob/master/cpp/NINJA-steps.... Overall this is the best build system I've used :) My pet peeve is having to 'make clean' when changing the variant, and this avoids that, making everything available at once. I'm sure it's reinventing some of the 120K lines of CMakeLists.txt that comes with CMake, but this way I don't have to learn and debug a wonky shell-ish language! One downside of the pattern is that you probably don't want to invoke shell on Windows (which doesn't affect my project). I find that very useful though, so I'm not sure what would replace it for Windows projects. (I guess batch files, but I don't think they're powerful enough)