4 ms·

FWIW, my tool Ninja (was just on here again this week here: https://news.ycombinator.com/item?id=9282539 https://news.ycombinator.com/item?id=9282539 ) was desi

by evmar 12y ago

FWIW, my tool Ninja (was just on here again this week here: https://news.ycombinator.com/item?id=9282539 https://news.ycombinator.com/item?id=9282539 ) was designed to be more or less the same idea as make with some semantic fixes, and that includes supporting multiple outputs. You can see the brief list of improvements here: https://martine.github.io/ninja/manual.html#_comparison_to_make https://martine.github.io/ninja/manual.html#_comparison_to_m... (Within Ninja this manifests concretely as needing to represent the build graph as a bipartite graph between files and commands. Though I don't know make's internal representation, I imagine the straightforward implementation is as a graph between files and I can imagine why it'd be difficult to make multiple outputs work in that case.)

ploxiln 12y ago
I really like the minimal and solid design of ninja; I'd agree that if make isn't cutting it, ninja + (your own script to generate all the correct rules) is probably the best way to go.
JoshTriplett 12y ago
Is there some fundamental reason why the approach used in Ninja couldn't be used to accelerate a substantial subset of Makefiles? If you dropped Make's built-in rules, more complex functions, VPATH, conditionals, and similar items the Ninja manual describes as slowing down Make (all of which are non-portable features of GNU Make), while mostly keeping compatibility with the portable subset of Make, is there some fundamental reason that couldn't be accelerated to the same degree as Ninja?
- evmar 12y ago
  No, they should be equivalent. The Ninja manual goes so far as to encourage you to continue using Make if Make is fast enough for you. However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all. Because we cared more about performance than other things we maybe were a little crazier about shaving off milliseconds than other systems, so it might be hard to make make fast enough. But (as the Ninja manual suggests) this may only matter for very large (Chrome-sized) projects. You can read a chapter about some of the performance work done on Ninja here: http://www.aosabook.org/en/posa/ninja.html http://www.aosabook.org/en/posa/ninja.html
  - JoshTriplett 12y ago
    > However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all. Not exactly. I'm asking if an implementation of the portable subset of Make people actually use (pointedly excluding GNU Make extensions) could be made nearly as fast as Ninja. That would have the advantage of running existing makefiles, as long as those makefiles weren't written specifically for GNU Make.
    - evmar 12y ago
      I think we are maybe talking in circles. :) When you write "could be made nearly as fast as Ninja", I thought you meant you would change the code of Make or write a new tool. If you don't write a new tool and just use a subset of Make's functionality for your own Makefiles I'd expect those to be faster than your standard Makefiles, sure. But "existing makefiles" typically use a mixture of GNU make-isms and other things that are not GNU-specific but still slow (e.g. from a skim of the BSD make manual I see lazy variable expansions and control structures). So if you're talking about implementing a subset of Make you're talking about something that's unlikely to be compatible with Makefiles seen in the wild. And then if you're not compatible with Makefiles seen in the wild you've effectively written an incompatible faster subset of Make, which brings us back to what Ninja is. (In case it's not clear, in Chrome's case the same code was used generate Makefiles and Ninja files with some relatively minor output differences -- the tools are that close.) Perhaps there's some "common" subset of Make that covers some high percentage of build files seen in the wild that could be made faster. That could be valuable to organizations who want faster builds without changing anything. I vaguely recall seeing some commercial software that did this even -- as I recall, their value add was that they had tooling that would figure out how to run your Makefiles in parallel; though Make supports parallel execution, apparently this business found enough customers by just targeting those with underspecified Makefiles that weren't parallel safe and then fixing them! Anyway, all of this is kinda moot because for 99% of projects Make is plenty fast. I even use Make myself for all my personal projects! In Chrome's case (what we wrote Ninja for) there's so much code that the build files themselves are over ten megabytes of text, so to parse that quickly you're at the level of worrying about avoiding memory allocations in the lexer, which is beyond what most people would care about.
rspeer 12y ago
Pretty cool. The fact that I'd have to write my own tool to generate the list of dependencies, instead of using pattern rules, is kind of a downer, but-- Okay, now I'm wondering if I could get Make to generate a Ninja file.
- evmar 12y ago
  Part of the reason Ninja doesn't support pattern rules is that it would require matching patterns while choosing what to compile. In projects like Chrome (with hundreds of thousands of source files) that can take a while.

by evmar 12y ago

ploxiln 12y ago

I really like the minimal and solid design of ninja; I'd agree that if make isn't cutting it, ninja + (your own script to generate all the correct rules) is probably the best way to go.

JoshTriplett 12y ago

Is there some fundamental reason why the approach used in Ninja couldn't be used to accelerate a substantial subset of Makefiles? If you dropped Make's built-in rules, more complex functions, VPATH, conditionals, and similar items the Ninja manual describes as slowing down Make (all of which are non-portable features of GNU Make), while mostly keeping compatibility with the portable subset of Make, is there some fundamental reason that couldn't be accelerated to the same degree as Ninja?

evmar 12y ago
No, they should be equivalent. The Ninja manual goes so far as to encourage you to continue using Make if Make is fast enough for you. However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all. Because we cared more about performance than other things we maybe were a little crazier about shaving off milliseconds than other systems, so it might be hard to make make fast enough. But (as the Ninja manual suggests) this may only matter for very large (Chrome-sized) projects. You can read a chapter about some of the performance work done on Ninja here: http://www.aosabook.org/en/posa/ninja.html http://www.aosabook.org/en/posa/ninja.html
- JoshTriplett 12y ago
  > However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all. Not exactly. I'm asking if an implementation of the portable subset of Make people actually use (pointedly excluding GNU Make extensions) could be made nearly as fast as Ninja. That would have the advantage of running existing makefiles, as long as those makefiles weren't written specifically for GNU Make.
  - evmar 12y ago
    I think we are maybe talking in circles. :) When you write "could be made nearly as fast as Ninja", I thought you meant you would change the code of Make or write a new tool. If you don't write a new tool and just use a subset of Make's functionality for your own Makefiles I'd expect those to be faster than your standard Makefiles, sure. But "existing makefiles" typically use a mixture of GNU make-isms and other things that are not GNU-specific but still slow (e.g. from a skim of the BSD make manual I see lazy variable expansions and control structures). So if you're talking about implementing a subset of Make you're talking about something that's unlikely to be compatible with Makefiles seen in the wild. And then if you're not compatible with Makefiles seen in the wild you've effectively written an incompatible faster subset of Make, which brings us back to what Ninja is. (In case it's not clear, in Chrome's case the same code was used generate Makefiles and Ninja files with some relatively minor output differences -- the tools are that close.) Perhaps there's some "common" subset of Make that covers some high percentage of build files seen in the wild that could be made faster. That could be valuable to organizations who want faster builds without changing anything. I vaguely recall seeing some commercial software that did this even -- as I recall, their value add was that they had tooling that would figure out how to run your Makefiles in parallel; though Make supports parallel execution, apparently this business found enough customers by just targeting those with underspecified Makefiles that weren't parallel safe and then fixing them! Anyway, all of this is kinda moot because for 99% of projects Make is plenty fast. I even use Make myself for all my personal projects! In Chrome's case (what we wrote Ninja for) there's so much code that the build files themselves are over ten megabytes of text, so to parse that quickly you're at the level of worrying about avoiding memory allocations in the lexer, which is beyond what most people would care about.

rspeer 12y ago

Pretty cool. The fact that I'd have to write my own tool to generate the list of dependencies, instead of using pattern rules, is kind of a downer, but-- Okay, now I'm wondering if I could get Make to generate a Ninja file.

evmar 12y ago
Part of the reason Ninja doesn't support pattern rules is that it would require matching patterns while choosing what to compile. In projects like Chrome (with hundreds of thousands of source files) that can take a while.