9 ms·

Ninja, a small build system with a focus on speed

36 points by vgnet 14y ago

daphoenix 14y ago
How is this better than http://industriousone.com/premake http://industriousone.com/premake ?
- i80and 14y ago
  Wrong question; they're working on different levels of abstraction. Ninja is designed to be a very fast and lightweight tool for doing work based on a dependency graph: actually knowing how to build anything on its own is an explicit non-goal. Premake, on the other hand is a full-blown build system that can make decisions and knows about how to perform specific code-related tasks, and can then generate rules for other tools to follow. Unless I'm mistaken, Premake could hypothetically output a build.ninja file.
- vgnet 14y ago
  It seems to be faster (starts building faster) and gives progress information make lacks. See http://neugierig.org/software/chromium/notes/2011/02/ninja.html http://neugierig.org/software/chromium/notes/2011/02/ninja.h... (for motivation) and https://plus.google.com/108996039294665965197/posts/SfhrFAhRyyd https://plus.google.com/108996039294665965197/posts/SfhrFAhR... (for benchmarks). TLDR: Firefox takes 14m to fully build with make and a nop build takes 57.9s, Chrome takes 30m with ninja, but a nop build takes only 0.74s.
- drothlis 14y ago
  I don't know the design goals of premake but ninja is very very fast at deciding what to build (a no-op build of a large project that might take a traditional make system several seconds, is almost instantaneous with ninja, allegedly). For those already using CMake--ninja will slot right in. Are there any large projects using premake? Ninja+gyp is used by Chromium.
Meai 14y ago
I use tup: http://gittup.org/tup/ http://gittup.org/tup/ but I'd like to hear from the author of Ninja what his opinion is on tup.
- malkia 14y ago
  This always comes in three, someone mentions ninja, then comes redo, and lastly tup :) (Saw in two places, reading mailing lists). LOL, And I'm brewing my own solution for brewing other people's solutions :)
- Ralith 14y ago
  I also use and enjoy tup, which has the significant features of automatically constructing most of the dependency graph, and proven optimality and correctness for incremental rebuilds. Not to mention that it's highly usable standalone. Does Ninja offer any of that? It doesn't seem to, judging from a skim of the docs.
  - evmar 14y ago
    I (the author of Ninja) think tup is a fine choice for your project. Ninja was designed to work within a specific pragmatic context: a very large project (Chrome) that had existing constraints on how the build works. (This design also makes Ninja suitable for use from CMake, which means you can use Ninja to build e.g. LLVM in a manner faster than the existing alternatives.) Here is a longer post on the subject. https://groups.google.com/group/ninja-build/msg/b52e7d3b77bb2e2c https://groups.google.com/group/ninja-build/msg/b52e7d3b77bb... Most projects should probably not use Ninja. A previous iteration of the home page tried to scare users away. I could probably improve that.
ajross 14y ago
Bravo! Granted, it's yet another make replacement the world really doesn't need. But that said, and unlike all the other attemps, this actually seems to be better than make. Almost always, these things are junk (Ant, I'm looking at you) which at best implement a subset of make's features in a "pure" way (and thus look good to people who don't understand make but like Java-or-whatever). This one actually seems to understand what make needs (cleaner variable semantics, multiple outputs for a target) and what parts need to be tossed (all the default rules). I'm impressed. But I still won't likely use it. The world doesn't need it.
- pestaa 14y ago
  yet another make replacement the world really doesn't need Didn't it start exactly like that with CMake? Turns out, the world needed it.
  - ajross 14y ago
    cmake is a replacement for autotools, not make. It generates makefiles as output.
    - pestaa 14y ago
      Good point, although if autotools was invented to replace makefiles by generating them, I'd argue CMake was too, even if indirectly.
      - gnok 14y ago
        I don't quite think autotools were invented to replace Makefiles. Autoconf and friends _configure_ the build environment by discovering locations of dependencies, feature sets, available compilers and build tools and other build conditions. This information is then used to generate the appropriate Makefile. Makefiles solve a different problem; which is of local build dependencies, looking for changes, packaging and suchlike. CMake, as it happens, was written because Autotools were particularly ineffecient at the configuration problem. Make happens to be very efficient.
julian37 14y ago
Ninja is mostly encoding agnostic, as long as the bytes Ninja cares about (like slashes in paths) are ASCII. This means e.g. UTF-8 or ISO-8859-1 input files ought to work. I was going to say that this is brittle because UTF-8 multi-byte sequences might contain bytes such as 0x2F (forward slash) without actually encoding a slash... but it turns out that's wrong. All bytes in multi-byte sequences always have the high bit set, so you can look for ASCII-7 characters in UTF-8 strings without having to worry about getting false positives. That's a very useful property of UTF-8 I wasn't (consciously) aware of before. http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description
- groby_b 14y ago
  It was actually one of the design goals of UTF-8. As much of a pain as dealing with it is, it certainly had a lot of forethought put in.
- grogs 14y ago
  UTF-8 is just... really good. The only disadvantage I can really think of is that the implementations must be fairly complex. That and determining character count can no longer be calculated from filesize.
beagle3 14y ago
Anyone has an idea of how this compares to apenwarr's implementation of djb's "redo" concept? Compared to make, redo is extremely simple, yet more versatile, more robust - and potentially very efficient. djb only released the spec (not working code). apenwarr implemented it in Python, which means it's a lot slower than it could be (which you'd mostly feel on nop builds).
- apenwarr 14y ago
  The short version is that ninja is declarative while redo is imperative. And I'm not sure that battle will be resolved in my lifetime :) Personally, I prefer imperative stuff most of the time, but lots of smart people disagree. ninja config files, as I understand it, are designed to be produced by some other tool, because purely-declarative languages are typically a pain for humans to write by hand. So it's one layer in a multi-layer system, hence the integration with cmake. redo removes layers; you can quite easily write your whole build system in redo, without first translating your configuration from one file type to another. The down side of that design is it's hard to guarantee your build system is "hygienic"; since every .do script is a program, the program might go do things it shouldn't be doing or which might be insecure. In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. It can be very powerful to manipulate declarative structures like ninja's. (Not that you couldn't do those things with redo, but it would be trickier.) For similar reasons, ninja is probably more portable to Windows than redo is. (redo can run on Windows, but you need a Unix-compatible sh to do it with, which is obviously rather un-Windowsy.)
  - beagle3 14y ago
    > In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. But apparently, these are not the goals for ninja. The goals for ninja appear to be speed, speed and more speed, especially for a no-op or one-file-change build. I wonder if anyone converted the build project of a project e.g. the size of chrome to redo and can compare build speeds to the ninja version. Furthermore, if speed is your major optimization point, it seems the approach taken by http://gittup.org/tup http://gittup.org/tup seems impossible to beat, and as a bonus you get perfect dependency information with no additional work (and see http://gittup.org/gittup http://gittup.org/gittup - they ported quite a few projects to it)
zserge 14y ago
Ninja is a build system of my dream - minimalistic, clean and simple. I've tried it on some small projects about a year ago, and it seemed very fast and stable. That days I generated .ninja with shell scripts, so now I'm happy to hear of CMake support. Still sorry that ninja build system is not included in most Linux distros, I believe that would significantly increase its popularity among developers.
bla2 14y ago
More ninja benchmark numbers: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC https://plus.google.com/101038813433650812235/posts/irc26fhR...
nicholassmith 14y ago
I've been looking to replace a projects build system from ant to something else, and this might just be it. Now to see if I can fiddle with Qt to see if I can get it to generate .ninja's rather than Makefiles.

36 points by vgnet 14y ago

daphoenix 14y ago

How is this better than http://industriousone.com/premake http://industriousone.com/premake ?

i80and 14y ago
Wrong question; they're working on different levels of abstraction. Ninja is designed to be a very fast and lightweight tool for doing work based on a dependency graph: actually knowing how to build anything on its own is an explicit non-goal. Premake, on the other hand is a full-blown build system that can make decisions and knows about how to perform specific code-related tasks, and can then generate rules for other tools to follow. Unless I'm mistaken, Premake could hypothetically output a build.ninja file.
vgnet 14y ago
It seems to be faster (starts building faster) and gives progress information make lacks. See http://neugierig.org/software/chromium/notes/2011/02/ninja.html http://neugierig.org/software/chromium/notes/2011/02/ninja.h... (for motivation) and https://plus.google.com/108996039294665965197/posts/SfhrFAhRyyd https://plus.google.com/108996039294665965197/posts/SfhrFAhR... (for benchmarks). TLDR: Firefox takes 14m to fully build with make and a nop build takes 57.9s, Chrome takes 30m with ninja, but a nop build takes only 0.74s.
drothlis 14y ago
I don't know the design goals of premake but ninja is very very fast at deciding what to build (a no-op build of a large project that might take a traditional make system several seconds, is almost instantaneous with ninja, allegedly). For those already using CMake--ninja will slot right in. Are there any large projects using premake? Ninja+gyp is used by Chromium.

Meai 14y ago

I use tup: http://gittup.org/tup/ http://gittup.org/tup/ but I'd like to hear from the author of Ninja what his opinion is on tup.

malkia 14y ago
This always comes in three, someone mentions ninja, then comes redo, and lastly tup :) (Saw in two places, reading mailing lists). LOL, And I'm brewing my own solution for brewing other people's solutions :)
Ralith 14y ago
I also use and enjoy tup, which has the significant features of automatically constructing most of the dependency graph, and proven optimality and correctness for incremental rebuilds. Not to mention that it's highly usable standalone. Does Ninja offer any of that? It doesn't seem to, judging from a skim of the docs.
- evmar 14y ago
  I (the author of Ninja) think tup is a fine choice for your project. Ninja was designed to work within a specific pragmatic context: a very large project (Chrome) that had existing constraints on how the build works. (This design also makes Ninja suitable for use from CMake, which means you can use Ninja to build e.g. LLVM in a manner faster than the existing alternatives.) Here is a longer post on the subject. https://groups.google.com/group/ninja-build/msg/b52e7d3b77bb2e2c https://groups.google.com/group/ninja-build/msg/b52e7d3b77bb... Most projects should probably not use Ninja. A previous iteration of the home page tried to scare users away. I could probably improve that.

ajross 14y ago

Bravo! Granted, it's yet another make replacement the world really doesn't need. But that said, and unlike all the other attemps, this actually seems to be better than make. Almost always, these things are junk (Ant, I'm looking at you) which at best implement a subset of make's features in a "pure" way (and thus look good to people who don't understand make but like Java-or-whatever). This one actually seems to understand what make needs (cleaner variable semantics, multiple outputs for a target) and what parts need to be tossed (all the default rules). I'm impressed. But I still won't likely use it. The world doesn't need it.

pestaa 14y ago
yet another make replacement the world really doesn't need Didn't it start exactly like that with CMake? Turns out, the world needed it.
- ajross 14y ago
  cmake is a replacement for autotools, not make. It generates makefiles as output.
  - pestaa 14y ago
    Good point, although if autotools was invented to replace makefiles by generating them, I'd argue CMake was too, even if indirectly.
    - gnok 14y ago
      I don't quite think autotools were invented to replace Makefiles. Autoconf and friends _configure_ the build environment by discovering locations of dependencies, feature sets, available compilers and build tools and other build conditions. This information is then used to generate the appropriate Makefile. Makefiles solve a different problem; which is of local build dependencies, looking for changes, packaging and suchlike. CMake, as it happens, was written because Autotools were particularly ineffecient at the configuration problem. Make happens to be very efficient.

julian37 14y ago

Ninja is mostly encoding agnostic, as long as the bytes Ninja cares about (like slashes in paths) are ASCII. This means e.g. UTF-8 or ISO-8859-1 input files ought to work. I was going to say that this is brittle because UTF-8 multi-byte sequences might contain bytes such as 0x2F (forward slash) without actually encoding a slash... but it turns out that's wrong. All bytes in multi-byte sequences always have the high bit set, so you can look for ASCII-7 characters in UTF-8 strings without having to worry about getting false positives. That's a very useful property of UTF-8 I wasn't (consciously) aware of before. http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description

groby_b 14y ago
It was actually one of the design goals of UTF-8. As much of a pain as dealing with it is, it certainly had a lot of forethought put in.
grogs 14y ago
UTF-8 is just... really good. The only disadvantage I can really think of is that the implementations must be fairly complex. That and determining character count can no longer be calculated from filesize.

beagle3 14y ago

Anyone has an idea of how this compares to apenwarr's implementation of djb's "redo" concept? Compared to make, redo is extremely simple, yet more versatile, more robust - and potentially very efficient. djb only released the spec (not working code). apenwarr implemented it in Python, which means it's a lot slower than it could be (which you'd mostly feel on nop builds).

apenwarr 14y ago
The short version is that ninja is declarative while redo is imperative. And I'm not sure that battle will be resolved in my lifetime :) Personally, I prefer imperative stuff most of the time, but lots of smart people disagree. ninja config files, as I understand it, are designed to be produced by some other tool, because purely-declarative languages are typically a pain for humans to write by hand. So it's one layer in a multi-layer system, hence the integration with cmake. redo removes layers; you can quite easily write your whole build system in redo, without first translating your configuration from one file type to another. The down side of that design is it's hard to guarantee your build system is "hygienic"; since every .do script is a program, the program might go do things it shouldn't be doing or which might be insecure. In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. It can be very powerful to manipulate declarative structures like ninja's. (Not that you couldn't do those things with redo, but it would be trickier.) For similar reasons, ninja is probably more portable to Windows than redo is. (redo can run on Windows, but you need a Unix-compatible sh to do it with, which is obviously rather un-Windowsy.)
- beagle3 14y ago
  > In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. But apparently, these are not the goals for ninja. The goals for ninja appear to be speed, speed and more speed, especially for a no-op or one-file-change build. I wonder if anyone converted the build project of a project e.g. the size of chrome to redo and can compare build speeds to the ninja version. Furthermore, if speed is your major optimization point, it seems the approach taken by http://gittup.org/tup http://gittup.org/tup seems impossible to beat, and as a bonus you get perfect dependency information with no additional work (and see http://gittup.org/gittup http://gittup.org/gittup - they ported quite a few projects to it)

zserge 14y ago

Ninja is a build system of my dream - minimalistic, clean and simple. I've tried it on some small projects about a year ago, and it seemed very fast and stable. That days I generated .ninja with shell scripts, so now I'm happy to hear of CMake support. Still sorry that ninja build system is not included in most Linux distros, I believe that would significantly increase its popularity among developers.

bla2 14y ago

More ninja benchmark numbers: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC https://plus.google.com/101038813433650812235/posts/irc26fhR...

nicholassmith 14y ago

I've been looking to replace a projects build system from ant to something else, and this might just be it. Now to see if I can fiddle with Qt to see if I can get it to generate .ninja's rather than Makefiles.