4 ms·

It's software, it's always fixable. Really depends on which is cheaper. The transaction cost of change or digging into some ancient code.

by eej71 12y ago

sklogic 12y ago
GNU Make architecture is a way too complicated. The very idea behind Ninja is to have the simplest possible representation which allows fast dependency analysis, leaving all the complexity for the front-ends. It is still possible, of course, to implement a compiler from GNU Make to Ninja (or any similar low-level representation), but in most cases it would be more practical to simply switch to something like CMake, which already translates to Ninja.
- justinsb 12y ago
  I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. But... it sounds like the OP isn't really concerned about CPU efficiency (CPU is cheap compared to all the other resources they're considering). I wonder if it would therefore be possible to convert a _single_ make execution to ninja (perhaps by instrumenting make). Then, each build would entail running the converted ninja file, while simultaneously re-running the make->ninja conversion. If the make->ninja conversion produces a different execution plan, throw away the ninja build and start again. This is based on my guess that the build execution changes comparatively rarely: new files or header includes are much rarer than 'simple' code changes. Edit: I think ninja is also smart enough to know when an artifact has been built using a different command / dependency, so even the 'cache miss' case where we restart the ninja build should be fast.
  - sklogic 12y ago
    In such scenario, ninja (or alike) would be the same thing to make as ccache to cc. But, the biggest single performance improvement of ninja comes from the fact that it's a single source for the whole tree - while make calls itself recursively, slowing down the dependency discovery and defying the caching potential.
    - justinsb 12y ago
      I don't think I've properly explained the idea... The goal is to get the same performance as ninja, in the common case when the dependencies/build steps haven't changed. We somehow use the makefile to generate the ninja file; likely by executing make and capturing the dependencies & commands (without executing the build commands). But we don't encode the full complexity of the makefile in the ninja rules; if someone adds a file or changes a flag or even adds a #include directive, our ninja rules will be out of date. So we are able to run the build entirely in ninja, but we may not be building what the makefile would build. So we have to 'waste' some CPU time running the make -> ninja conversion in parallel to verify that our ninja build is still correct, but as long as 'total ninja build time' > 'make to ninja conversion time', we come out ahead in wall clock time.
      - jessaustin 12y ago
        I've never used ninja, but ISTM the ninja setup would just need a few dependencies on the Makefiles themselves in order to build its own Ninjafiles (or whatever they're called). After all, if it's a build system, it surely knows whether or not depended-upon files have changed! Then there would be no need for "parallel" runs or error-prone diff'ing.
    - Nursie 12y ago
      Make only calls itself recursively if you build the Makefiles that way. It's possible to have flat, all in one Makefiles and use include files for different targets etc.
  - adekok 12y ago
    > I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. See my other comments for a set of GNU Make scripts I've been using: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake The craziest thing I did was realize that the rules were being evaluated recursively, and calling "info" (i.e. "print) was just as easy as calling "eval". 30 minutes later, I had: $ make bsd.mak It would spit out a 6000 line BSD-style Makefile. Just variable declarations (organized and commented!) along with rules. It's probably not hard to change the framework to spit out ninja files. It's just text. I'll go soak my head in cold water now.

It's software, it's always fixable. Really depends on which is cheaper. The transaction cost of change or digging into some ancient code.

by eej71 12y ago

It's software, it's always fixable. Really depends on which is cheaper. The transaction cost of change or digging into some ancient code.

sklogic 12y ago

GNU Make architecture is a way too complicated. The very idea behind Ninja is to have the simplest possible representation which allows fast dependency analysis, leaving all the complexity for the front-ends. It is still possible, of course, to implement a compiler from GNU Make to Ninja (or any similar low-level representation), but in most cases it would be more practical to simply switch to something like CMake, which already translates to Ninja.

justinsb 12y ago
I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. But... it sounds like the OP isn't really concerned about CPU efficiency (CPU is cheap compared to all the other resources they're considering). I wonder if it would therefore be possible to convert a _single_ make execution to ninja (perhaps by instrumenting make). Then, each build would entail running the converted ninja file, while simultaneously re-running the make->ninja conversion. If the make->ninja conversion produces a different execution plan, throw away the ninja build and start again. This is based on my guess that the build execution changes comparatively rarely: new files or header includes are much rarer than 'simple' code changes. Edit: I think ninja is also smart enough to know when an artifact has been built using a different command / dependency, so even the 'cache miss' case where we restart the ninja build should be fast.
- sklogic 12y ago
  In such scenario, ninja (or alike) would be the same thing to make as ccache to cc. But, the biggest single performance improvement of ninja comes from the fact that it's a single source for the whole tree - while make calls itself recursively, slowing down the dependency discovery and defying the caching potential.
  - justinsb 12y ago
    I don't think I've properly explained the idea... The goal is to get the same performance as ninja, in the common case when the dependencies/build steps haven't changed. We somehow use the makefile to generate the ninja file; likely by executing make and capturing the dependencies & commands (without executing the build commands). But we don't encode the full complexity of the makefile in the ninja rules; if someone adds a file or changes a flag or even adds a #include directive, our ninja rules will be out of date. So we are able to run the build entirely in ninja, but we may not be building what the makefile would build. So we have to 'waste' some CPU time running the make -> ninja conversion in parallel to verify that our ninja build is still correct, but as long as 'total ninja build time' > 'make to ninja conversion time', we come out ahead in wall clock time.
    - jessaustin 12y ago
      I've never used ninja, but ISTM the ninja setup would just need a few dependencies on the Makefiles themselves in order to build its own Ninjafiles (or whatever they're called). After all, if it's a build system, it surely knows whether or not depended-upon files have changed! Then there would be no need for "parallel" runs or error-prone diff'ing.
  - Nursie 12y ago
    Make only calls itself recursively if you build the Makefiles that way. It's possible to have flat, all in one Makefiles and use include files for different targets etc.
- adekok 12y ago
  > I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. See my other comments for a set of GNU Make scripts I've been using: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake The craziest thing I did was realize that the rules were being evaluated recursively, and calling "info" (i.e. "print) was just as easy as calling "eval". 30 minutes later, I had: $ make bsd.mak It would spit out a 6000 line BSD-style Makefile. Just variable declarations (organized and commented!) along with rules. It's probably not hard to change the framework to spit out ninja files. It's just text. I'll go soak my head in cold water now.