5 ms·

Looking for help improving performance of GNU Make

94 points by motiejus 12y ago

sklogic 12y ago
Unfixable. Use ninja instead.
- eej71 12y ago
  It's software, it's always fixable. Really depends on which is cheaper. The transaction cost of change or digging into some ancient code.
  - sklogic 12y ago
    GNU Make architecture is a way too complicated. The very idea behind Ninja is to have the simplest possible representation which allows fast dependency analysis, leaving all the complexity for the front-ends. It is still possible, of course, to implement a compiler from GNU Make to Ninja (or any similar low-level representation), but in most cases it would be more practical to simply switch to something like CMake, which already translates to Ninja.
    - justinsb 12y ago
      I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. But... it sounds like the OP isn't really concerned about CPU efficiency (CPU is cheap compared to all the other resources they're considering). I wonder if it would therefore be possible to convert a _single_ make execution to ninja (perhaps by instrumenting make). Then, each build would entail running the converted ninja file, while simultaneously re-running the make->ninja conversion. If the make->ninja conversion produces a different execution plan, throw away the ninja build and start again. This is based on my guess that the build execution changes comparatively rarely: new files or header includes are much rarer than 'simple' code changes. Edit: I think ninja is also smart enough to know when an artifact has been built using a different command / dependency, so even the 'cache miss' case where we restart the ninja build should be fast.
      - sklogic 12y ago
        In such scenario, ninja (or alike) would be the same thing to make as ccache to cc. But, the biggest single performance improvement of ninja comes from the fact that it's a single source for the whole tree - while make calls itself recursively, slowing down the dependency discovery and defying the caching potential.
        justinsb 12y ago
        I don't think I've properly explained the idea... The goal is to get the same performance as ninja, in the common case when the dependencies/build steps haven't changed. We somehow use the makefile to generate the ninja file; likely by executing make and capturing the dependencies & commands (without executing the build commands). But we don't encode the full complexity of the makefile in the ninja rules; if someone adds a file or changes a flag or even adds a #include directive, our ninja rules will be out of date. So we are able to run the build entirely in ninja, but we may not be building what the makefile would build. So we have to 'waste' some CPU time running the make -> ninja conversion in parallel to verify that our ninja build is still correct, but as long as 'total ninja build time' > 'make to ninja conversion time', we come out ahead in wall clock time.
        jessaustin 12y ago
        I've never used ninja, but ISTM the ninja setup would just need a few dependencies on the Makefiles themselves in order to build its own Ninjafiles (or whatever they're called). After all, if it's a build system, it surely knows whether or not depended-upon files have changed! Then there would be no need for "parallel" runs or error-prone diff'ing.
        Nursie 12y ago
        Make only calls itself recursively if you build the Makefiles that way. It's possible to have flat, all in one Makefiles and use include files for different targets etc.
      - adekok 12y ago
        > I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. See my other comments for a set of GNU Make scripts I've been using: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake The craziest thing I did was realize that the rules were being evaluated recursively, and calling "info" (i.e. "print) was just as easy as calling "eval". 30 minutes later, I had: $ make bsd.mak It would spit out a 6000 line BSD-style Makefile. Just variable declarations (organized and commented!) along with rules. It's probably not hard to change the framework to spit out ninja files. It's just text. I'll go soak my head in cold water now.
- onli 12y ago
  You might actually be right. Even though I didn't use ninja, given their description a specialized build system for fast build that circumvents the problems they mention that make their build slow might be the solution. But are they not giving way not enough information to seriously assess their situation? That - combined with the shortness of your comment and that people could think it's a joke - would explain your downvotes.
  - sklogic 12y ago
    At least they've explained that dependency resolution seems to be the bottleneck, and Ninja solves this particular source of congestion most nicely.
teddyh 12y ago
Summary: > What we see is that if we have more than 30 or so clients connected to the build Make is often unable to generate tasks fast enough to keep them busy. […] > Our management doesn't want us taking time to learn the internals of make and see if we can improve it ourselves, but they are willing to throw money at the problem, either by offering some sort of donation/bounty for performance improvements or directly hiring an expert on a contract basis; any improvements achieved would of course be submitted back to the community as patches.
andrewchambers 12y ago
Make has so much legacy cruft handling, people just moved to other systems that are simpler. Most developers don't even know most of the features make, and people don't really want to learn such an old program through and through.
- sudowhodoido 12y ago
  I think that avoiding complexity in the build system is the priority rather than avoiding a product that has cruft in it. Make can be simple, elegant and productive even on large projects if you think before you do something. For example SCons, CMake suffer from the same number of problems that manifest themselves differently. This comes from many years of experience of watching people hang themselves with make, then hang themselves with the next best thing, then hang themselves with another thing, then say "all build systems are shit" and write their own which is inferior to make to start with. The problem is always the product, not the build tool. All the useful features of make can be learned in a couple of hours. For ref, I'm going to get shot about "cross platform builds" and stuff in reply to this comment. You're right. Use nmake on windows and make on *nix platforms.
- norswap 12y ago
  Mmh, which build system do you think are simpler than Make? Make has a lot of quirks, but at its core it is a simple system.
  - hendzen 12y ago
    Cmake/ninja is becoming the standard for new C++ projects.
    - davexunit 12y ago
      Cmake is a fucking nightmare to work with. Autotools are light years better and more robust than cmake.
      - bodyfour 12y ago
jordigh 12y ago
I really hope someone takes the job offer, fixes the problem however possible, and then publicises the process. This is what free software is supposed to be about: you can hire anyone to fix it. Let's put theory to practice now and actually hire someone to do it. Edit: You know what, I'm now determined to now headhunt this person. It's the principle of the thing. Once I get the original poster's permission, I am reposting this ad on freedomsponsors.org, bountysource.com, and on the GNU job list: http://www.fsf.org/resources/jobs http://www.fsf.org/resources/jobs
deadgrey19 12y ago
If they're building C/C++ with it then this is a good option: https://github.com/Zomojo/Cake https://github.com/Zomojo/Cake - which is really just a front end to make, but produces really fast makefiles. It was initially developed to allow interactive speed iteration of compiling large C/C++ projects for data analysis. It does require that your source code comply with a few basic rules (just that c/cpp files and headers have the same name). But beyond that it runs automatically and quickly.
deleted 12y ago
[deleted]
adekok 12y ago
I've had luck using https://github.com/dmoulding/boilermake https://github.com/dmoulding/boilermake It's a GNU Make framework which allows you to create non-recursive Makefiles. I've re-worked it a bit, and used it in my pet project: https://github.com/freeradius/freeradius-server/ https://github.com/freeradius/freeradius-server/ 100K+ LoC, ~400 source files, multiple subdirectories, building libraries, executables, etc. Most of the slowness of Make is due to people creating recursive Makefiles. Running endless shell programs is very slow. Make by itself is surprisingly fast. The other benefit of non-recursive Make is that the builds can now go in parallel. What used to be a ~90s build is now ~7s on a multi-core machine. I also replace libtool (shudder) with jlibtool. That was taken from Apache, and modified to fix bugs work better, etc. That sped up the build a lot, too. Maybe I'm crazy, but these tools have worked across all Unix-y platforms I've tried. And I needed a migration path from recursive Make / libtool to something better. Dropping the build system on the floor and hoping for something better wasn't an option. This tool suite works much like the old one, but it's simpler, faster, and doesn't require additional packages (python, etc.) to do the build. Everything outside of Make is self-contained, and is included in the FreeRADIUS distribution. My projects: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake https://github.com/alandekok/jlibtool https://github.com/alandekok/jlibtool https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake
- sp332 12y ago
  They said the build is now so parallel - distributed to 30 machines - that make is having trouble generating jobs fast enough.
- chubot 12y ago
  If anyone wants to do non-recursive make for big projects, I would take a look at what the Android platform does. They have this elaborate Android.mk makefile fragment system. At the beginning of the build you go and find all the fragments, then process/concatenate them into one huge makefile, and do a single make invocation. It results in a very parallelizable build. It seems kinda crazy and hacky but it definitely works better than recursive make. For some reason the best docs seem to be on this Korean site: http://www.kandroid.org/ndk/docs/ANDROID-MK.html http://www.kandroid.org/ndk/docs/ANDROID-MK.html
anjbe 12y ago
Last month at EuroBSDCon, Marc Espie gave a neat talk describing some of the performance improvements he’s made to OpenBSD’s Make (and some positive words about Ninja as well) – http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.pdf http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.p...

94 points by motiejus 12y ago

sklogic 12y ago

Unfixable. Use ninja instead.

eej71 12y ago
It's software, it's always fixable. Really depends on which is cheaper. The transaction cost of change or digging into some ancient code.
- sklogic 12y ago
  GNU Make architecture is a way too complicated. The very idea behind Ninja is to have the simplest possible representation which allows fast dependency analysis, leaving all the complexity for the front-ends. It is still possible, of course, to implement a compiler from GNU Make to Ninja (or any similar low-level representation), but in most cases it would be more practical to simply switch to something like CMake, which already translates to Ninja.
  - justinsb 12y ago
    I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. But... it sounds like the OP isn't really concerned about CPU efficiency (CPU is cheap compared to all the other resources they're considering). I wonder if it would therefore be possible to convert a _single_ make execution to ninja (perhaps by instrumenting make). Then, each build would entail running the converted ninja file, while simultaneously re-running the make->ninja conversion. If the make->ninja conversion produces a different execution plan, throw away the ninja build and start again. This is based on my guess that the build execution changes comparatively rarely: new files or header includes are much rarer than 'simple' code changes. Edit: I think ninja is also smart enough to know when an artifact has been built using a different command / dependency, so even the 'cache miss' case where we restart the ninja build should be fast.
    - sklogic 12y ago
      In such scenario, ninja (or alike) would be the same thing to make as ccache to cc. But, the biggest single performance improvement of ninja comes from the fact that it's a single source for the whole tree - while make calls itself recursively, slowing down the dependency discovery and defying the caching potential.
      - justinsb 12y ago
        I don't think I've properly explained the idea... The goal is to get the same performance as ninja, in the common case when the dependencies/build steps haven't changed. We somehow use the makefile to generate the ninja file; likely by executing make and capturing the dependencies & commands (without executing the build commands). But we don't encode the full complexity of the makefile in the ninja rules; if someone adds a file or changes a flag or even adds a #include directive, our ninja rules will be out of date. So we are able to run the build entirely in ninja, but we may not be building what the makefile would build. So we have to 'waste' some CPU time running the make -> ninja conversion in parallel to verify that our ninja build is still correct, but as long as 'total ninja build time' > 'make to ninja conversion time', we come out ahead in wall clock time.
        jessaustin 12y ago
        I've never used ninja, but ISTM the ninja setup would just need a few dependencies on the Makefiles themselves in order to build its own Ninjafiles (or whatever they're called). After all, if it's a build system, it surely knows whether or not depended-upon files have changed! Then there would be no need for "parallel" runs or error-prone diff'ing.
      - Nursie 12y ago
        Make only calls itself recursively if you build the Makefiles that way. It's possible to have flat, all in one Makefiles and use include files for different targets etc.
    - adekok 12y ago
      > I do think it would be interesting to implement a Make -> Ninja compiler; the devil is in extracting all the correct dependencies. See my other comments for a set of GNU Make scripts I've been using: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake The craziest thing I did was realize that the rules were being evaluated recursively, and calling "info" (i.e. "print) was just as easy as calling "eval". 30 minutes later, I had: $ make bsd.mak It would spit out a 6000 line BSD-style Makefile. Just variable declarations (organized and commented!) along with rules. It's probably not hard to change the framework to spit out ninja files. It's just text. I'll go soak my head in cold water now.
onli 12y ago
You might actually be right. Even though I didn't use ninja, given their description a specialized build system for fast build that circumvents the problems they mention that make their build slow might be the solution. But are they not giving way not enough information to seriously assess their situation? That - combined with the shortness of your comment and that people could think it's a joke - would explain your downvotes.
- sklogic 12y ago
  At least they've explained that dependency resolution seems to be the bottleneck, and Ninja solves this particular source of congestion most nicely.

teddyh 12y ago

Summary: > What we see is that if we have more than 30 or so clients connected to the build Make is often unable to generate tasks fast enough to keep them busy. […] > Our management doesn't want us taking time to learn the internals of make and see if we can improve it ourselves, but they are willing to throw money at the problem, either by offering some sort of donation/bounty for performance improvements or directly hiring an expert on a contract basis; any improvements achieved would of course be submitted back to the community as patches.

andrewchambers 12y ago

Make has so much legacy cruft handling, people just moved to other systems that are simpler. Most developers don't even know most of the features make, and people don't really want to learn such an old program through and through.

sudowhodoido 12y ago
I think that avoiding complexity in the build system is the priority rather than avoiding a product that has cruft in it. Make can be simple, elegant and productive even on large projects if you think before you do something. For example SCons, CMake suffer from the same number of problems that manifest themselves differently. This comes from many years of experience of watching people hang themselves with make, then hang themselves with the next best thing, then hang themselves with another thing, then say "all build systems are shit" and write their own which is inferior to make to start with. The problem is always the product, not the build tool. All the useful features of make can be learned in a couple of hours. For ref, I'm going to get shot about "cross platform builds" and stuff in reply to this comment. You're right. Use nmake on windows and make on *nix platforms.
norswap 12y ago
Mmh, which build system do you think are simpler than Make? Make has a lot of quirks, but at its core it is a simple system.
- hendzen 12y ago
  Cmake/ninja is becoming the standard for new C++ projects.
  - davexunit 12y ago
    Cmake is a fucking nightmare to work with. Autotools are light years better and more robust than cmake.
    - bodyfour 12y ago

jordigh 12y ago

I really hope someone takes the job offer, fixes the problem however possible, and then publicises the process. This is what free software is supposed to be about: you can hire anyone to fix it. Let's put theory to practice now and actually hire someone to do it. Edit: You know what, I'm now determined to now headhunt this person. It's the principle of the thing. Once I get the original poster's permission, I am reposting this ad on freedomsponsors.org, bountysource.com, and on the GNU job list: http://www.fsf.org/resources/jobs http://www.fsf.org/resources/jobs

deadgrey19 12y ago

If they're building C/C++ with it then this is a good option: https://github.com/Zomojo/Cake https://github.com/Zomojo/Cake - which is really just a front end to make, but produces really fast makefiles. It was initially developed to allow interactive speed iteration of compiling large C/C++ projects for data analysis. It does require that your source code comply with a few basic rules (just that c/cpp files and headers have the same name). But beyond that it runs automatically and quickly.

deleted 12y ago

[deleted]

adekok 12y ago

I've had luck using https://github.com/dmoulding/boilermake https://github.com/dmoulding/boilermake It's a GNU Make framework which allows you to create non-recursive Makefiles. I've re-worked it a bit, and used it in my pet project: https://github.com/freeradius/freeradius-server/ https://github.com/freeradius/freeradius-server/ 100K+ LoC, ~400 source files, multiple subdirectories, building libraries, executables, etc. Most of the slowness of Make is due to people creating recursive Makefiles. Running endless shell programs is very slow. Make by itself is surprisingly fast. The other benefit of non-recursive Make is that the builds can now go in parallel. What used to be a ~90s build is now ~7s on a multi-core machine. I also replace libtool (shudder) with jlibtool. That was taken from Apache, and modified to fix bugs work better, etc. That sped up the build a lot, too. Maybe I'm crazy, but these tools have worked across all Unix-y platforms I've tried. And I needed a migration path from recursive Make / libtool to something better. Dropping the build system on the floor and hoping for something better wasn't an option. This tool suite works much like the old one, but it's simpler, faster, and doesn't require additional packages (python, etc.) to do the build. Everything outside of Make is self-contained, and is included in the FreeRADIUS distribution. My projects: https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake https://github.com/alandekok/jlibtool https://github.com/alandekok/jlibtool https://github.com/alandekok/boilermake https://github.com/alandekok/boilermake

sp332 12y ago
They said the build is now so parallel - distributed to 30 machines - that make is having trouble generating jobs fast enough.
chubot 12y ago
If anyone wants to do non-recursive make for big projects, I would take a look at what the Android platform does. They have this elaborate Android.mk makefile fragment system. At the beginning of the build you go and find all the fragments, then process/concatenate them into one huge makefile, and do a single make invocation. It results in a very parallelizable build. It seems kinda crazy and hacky but it definitely works better than recursive make. For some reason the best docs seem to be on this Korean site: http://www.kandroid.org/ndk/docs/ANDROID-MK.html http://www.kandroid.org/ndk/docs/ANDROID-MK.html

anjbe 12y ago

Last month at EuroBSDCon, Marc Espie gave a neat talk describing some of the performance improvements he’s made to OpenBSD’s Make (and some positive words about Ninja as well) – http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.pdf http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.p...