6 ms·
N2: Alternative Ninja Implementation
- secure 5y agoCorresponding blog post: http://neugierig.org/software/blog/2022/03/n2.html http://neugierig.org/software/blog/2022/03/n2.html
- einpoklum 5y agoNutshell summary of the post: Instead of relying on modification-time comparisons between dependency and target output, use hashes of the relevant "ingredients" of building a target (cmdline, arguments, meta-data about input files etc.).
- boardwaalk 5y agoI’m a little confused by some of this (mtime vs manifest) because Ninja does rebuild things if things like the command line changes, doesn’t it? Or have I misunderstood how this tool works forever…
- jhasse 5y agoYes, Ninja hashes the command line and compares that hash in addition to comparing the mtimes.
- sylware 5y agoand we should not forget about the simple C (C99) re-implementation of ninja from M. Forney: https://github.com/michaelforney/samurai https://github.com/michaelforney/samurai
- einpoklum 5y agoWhat are the benefits of the C implementation? I noticed the adherence to POSIX command-line conventions, but that doesn't sound like a good enough reason for a reimplementation in a different language.
- floatboth 5y agoI guess the ability to work in really restricted/minimal environments where you only have a simple C compiler like tcc, and nothing like LLVM. Or paranoid environments where you can only use CompCert. Something like that.
- einpoklum 5y agoBut can't you just cross-compile?
- floatboth 5y agoI'm not talking about embedded, I'm talking about e.g. intentionally minimalist systems that have to be self-building, or systems that have to be built with a formally verified compiler (which does not exist for C++), or anything else like that.
- cmrdporcupine 5y agoWell, HN commenters will like this: Something rewritten in Rust without advertising that as its primary feature.
- gkbrk 5y agoShameless plug, I also wrote my own Ninja implementation [1]. I tried to make it easy to understand, while supporting all the Ninja features I used. In the end, it was able to compile my own kernel and a few other Ninja projects. [1]: https://github.com/gkbrk/scripts/blob/master/ninja.py https://github.com/gkbrk/scripts/blob/master/ninja.py
- einpoklum 5y agoTo improve your shameless plug, you should tell us: * What your motivation for writing it was. * Why people might like to use it instead of ninja or n2.
- AceJohnny2 5y agoThe author mentions it, and I highly recommend anybody interested in build systems read the seminal paper Build Systems A La Carte [1], which breaks down various build systems (including Excel!) into their component concepts, and along the way finds a new feature-space that hadn't been explored. [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... (an understanding of Haskell is recommended to understand the implementations, but not required to understand the concepts)
- NateEag 5y agoOnly tangentially related, I'm personally intrigued by redo, but have never had a good justification for trying it out. It looks eminently reasonable to me, and Avery Pennarun plus DJB sounds like a majorly winning team to me. https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/
- ncmncm 5y agoRedo has a conceptual cleanliness that other systems lack. But maybe it should emit a ninja script instead of building directly.
- wscott 5y agoI love build systems and have tried many of them. Redo is fun to play with. It is really easy to understand. There is some messy reality that causes redo the struggle. For instance it doesn't natively support a single command that produces multiple build targets. You end up needing workarounds. I also find that redo starts to feel like Rails where your logic is spread around is many little files. Each file is easy to understand, but it takes a while to build the full picture.
- NateEag 5y agoThat's interesting. Thanks for the info!
- humanrebar 5y agoIf someone wanted to reimplement ninja and add support for the GNU Make job server, I'd be sold. https://github.com/ninja-build/ninja/issues/1139 https://github.com/ninja-build/ninja/issues/1139 At a certain scale (i.e. package management and not a mono build system), having a common job server is essential to scaling in a healthy way.
- mastax 5y agoAside: the make job server is very simple and UNIXy. There's a pipe shared between `make` and its subprocesses. `make` writes `N-1` bytes to the pipe. When a job wants to do parallel work, it reads a byte from the pipe. When it's done, it writes a byte back to the pipe. (Both rustc and cargo support the make job server)
- stormbrew 5y agoIt's really an elegant design, which is nice, other than the way that information is passed into the child process (roughly: MAKEFLAGS=" -j --job-server=3,4"), which requires you to argparse out the fd numbers. It would be nice if a new, less cumbersome, environment protocol could be adopted.
- ajconway 5y agoWhat happens when the job crashes?
- stormbrew 5y agoI dunno if they'll accept it but I might actually submit a PR to add this to n2, because I also would very much like "ninja with jobserver" and it being written in rust it should be fairly easy to add since the code cargo uses for jobserver support is broken out into a very simple library.
- jhasse 5y agoDo you mean jobserver client or sever? As for "ninja with jobserver client" there's already https://github.com/Kitware/ninja https://github.com/Kitware/ninja
- ccxx 5y agoNone of these have a historical ETA estimator, which has been requested in ninja maybe a decade ago. I hacked a few lines to show something, but would prefer the correct solution.
- evmar 5y ago(n2 author here) ETA is pretty hard in the presence of parallelization unless you make big assumptions like "all pending build tasks will be able to run in parallel". If you had any good ideas here I'd love to hear them.
- jhasse 5y agoI think perfect is the enemy of good regarding ETA. This Ninja PR comes pretty close to the best possible solution I can think of: https://github.com/ninja-build/ninja/pull/1963 https://github.com/ninja-build/ninja/pull/1963
- stormbrew 5y agoIt's not explicitly stated anywhere, but this appears to be by the original author and one of the later maintainers of the original ninja, which probably puts it ahead of some of the other rewrites out there in terms of its likelihood of adoption.
- nsm 5y agoI also created a minimal Ninja implementation in Rust some time ago. My goals were to implement it in terms of the Build Systems a la carte paper. Of course, hard to compete with the original ninja authors who obviously understand it much better. For example I used a separate lexer and environments, which got a little annoying and is something explicitly called out in their design. https://github.com/nikhilm/ninja-rs https://github.com/nikhilm/ninja-rs
- evmar 5y ago(n2 author here) This is really cool! I am sorry if weird ninja decisions (like not being lexer-friendly or phony being underspecified) caused you grief, the whole thing was really built as a quick hack before I understood the problem space.