4 ms·

On one project that supports both tup and cmake: building from scratch with tup takes 8 seconds, ninja takes 18s and "make -j" takes 22s. If you are a heavy sh

On one project that supports both tup and cmake: building from scratch with tup takes 8 seconds, ninja takes 18s and "make -j" takes 22s. If you are a heavy shell user you'll also appreciate that tup can be executed from any subdir within the project, no need to keep around extra terminal just for compiling.

kazinator 4y ago
That means that the upper bound on doing nothing but compiling the stuff is 8s. to which make adds 14s of pure waste. That's a weird situation which seems like having a low probability of being broadly representative.
- drothlis 4y ago
  Yeah, meaningful benchmarks take a lot of work. - Does the reported ninja time include the cmake step that generates the ninja files? (which typically doesn't need to be done for incremental builds). - "make -j" doesn't limit the number of jobs, so you can run into resource contention if building too many things at once. Ninja (by default) limits the parallel jobs to the number of CPUs. - Disabling make's built-in rules (with `-r -R`) can make a big difference to performance. - Make spends a lot of time processing the ".d" files with header dependency information -- 98% of the time in my (artificial) benchmarks!
  - jmiskovic 4y ago
    This wasn't a "meaningful benchmark", just a single data point to show how tup makes practical sense, when everyone else was criticizing "yet another build system" on theoretical points. I included the time to generate ninja files, the measured time is for project to build from scratch. Tup generates its initial files _much_ faster (~0.2 seconds compared to ~4 seconds). The strength of tup is actually in incremental builds, but that's harder to measure: you get shorter times with larger relative variations between repeated tries. I wasn't interested in working around the bad make defaults, I also didn't spend any time optimizing tup flags.
  - kazinator 4y ago
    There shouldn't be any .d files in a from-scratch clean build. They get generated during compilation and are not needed initially because every object file is out of date due to its nonexistence. However, they do add a lot of bulk once made and increase with the size of the project. A lot of the information is redundant. Among the files you may find many pairs which only differ in the principal file, the headers being the same: foo.o: foo.c $SOME_HEADERS bar.o: bar.c $SOME_HEADERS In theory, if we have GNU Make, we should be able to compress these into static pattern rules. Concretely, in the above case, the pair condenses to the static pattern rule: foo.o bar.o: %.o: %.c $SOME_HEADERS When, we check whether, say, bar.o is out of date, we find that it as a target in the left side of the rule. Then in the middle, the name matches the %.o target pattern, identifying the stem as "bar", and from there the full dependency rule is instantiated with all the prerequisites. To generalize that, we partition the $(OBJECTS) into subsets based on which patterns they belong to. Hopefully there are a lot fewer subsets than files. Then for each subset we do # The a.h b.h subset $(OBJS_0): %.o: %.c a.h b.h # The a.h d.h subset $(OBJS_1): %.o: %.c a.h d.h and so on. The only problem is managing these subsets (in particular, incrementally) in such a way that more time isn't spent doing that than time saved loading the rules. Stale dependencies can be troublesome when header files are renamed or deleted; this sort of scheme could make it worse. One way to handle these subsets incrementally would be to keep them in some separate database outside of the Makefile. Whenever a file is compiled and the raw .d is generated, we run a script which looks up the header signature and matches it to a subset. Then it assigns that file to the correct subset (perhaps deleting it from an existing one). A step at the start of the build would regenerate the dependency file seen by make if the database indicates that it's out of date.
    - drothlis 4y ago
      Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.
actionfromafar 4y ago
I suppose these numbers must converge for all three as number of files go up? (And compile time for each file is not negligible.) I say this because in my use case ninja spends almost all of its time running compiler processes, not resolving deps.
- drothlis 4y ago
  I did hit a weird edge case in Make's job scheduling. Ninja will try to build things in the order that they are listed in the "build.ninja" file; make kind-of does too, but if any dependencies need to be built first, the later targets get pushed to the very end of the job queue. This means that you can end up running many resource-intensive linking jobs all at the same time at the very end of the build: https://lists.gnu.org/archive/html/help-make/2016-11/msg00006.html https://lists.gnu.org/archive/html/help-make/2016-11/msg0000... It's hard to say how much impact this would have in real-world scenarios.
  - actionfromafar 4y ago
    I think I must have hit that case often, I have come to see it as inevitable that Make does a lot of linking at the end...

kazinator 4y ago

That means that the upper bound on doing nothing but compiling the stuff is 8s. to which make adds 14s of pure waste. That's a weird situation which seems like having a low probability of being broadly representative.

drothlis 4y ago
Yeah, meaningful benchmarks take a lot of work. - Does the reported ninja time include the cmake step that generates the ninja files? (which typically doesn't need to be done for incremental builds). - "make -j" doesn't limit the number of jobs, so you can run into resource contention if building too many things at once. Ninja (by default) limits the parallel jobs to the number of CPUs. - Disabling make's built-in rules (with `-r -R`) can make a big difference to performance. - Make spends a lot of time processing the ".d" files with header dependency information -- 98% of the time in my (artificial) benchmarks!
- jmiskovic 4y ago
  This wasn't a "meaningful benchmark", just a single data point to show how tup makes practical sense, when everyone else was criticizing "yet another build system" on theoretical points. I included the time to generate ninja files, the measured time is for project to build from scratch. Tup generates its initial files _much_ faster (~0.2 seconds compared to ~4 seconds). The strength of tup is actually in incremental builds, but that's harder to measure: you get shorter times with larger relative variations between repeated tries. I wasn't interested in working around the bad make defaults, I also didn't spend any time optimizing tup flags.
- kazinator 4y ago
  There shouldn't be any .d files in a from-scratch clean build. They get generated during compilation and are not needed initially because every object file is out of date due to its nonexistence. However, they do add a lot of bulk once made and increase with the size of the project. A lot of the information is redundant. Among the files you may find many pairs which only differ in the principal file, the headers being the same: foo.o: foo.c $SOME_HEADERS bar.o: bar.c $SOME_HEADERS In theory, if we have GNU Make, we should be able to compress these into static pattern rules. Concretely, in the above case, the pair condenses to the static pattern rule: foo.o bar.o: %.o: %.c $SOME_HEADERS When, we check whether, say, bar.o is out of date, we find that it as a target in the left side of the rule. Then in the middle, the name matches the %.o target pattern, identifying the stem as "bar", and from there the full dependency rule is instantiated with all the prerequisites. To generalize that, we partition the $(OBJECTS) into subsets based on which patterns they belong to. Hopefully there are a lot fewer subsets than files. Then for each subset we do # The a.h b.h subset $(OBJS_0): %.o: %.c a.h b.h # The a.h d.h subset $(OBJS_1): %.o: %.c a.h d.h and so on. The only problem is managing these subsets (in particular, incrementally) in such a way that more time isn't spent doing that than time saved loading the rules. Stale dependencies can be troublesome when header files are renamed or deleted; this sort of scheme could make it worse. One way to handle these subsets incrementally would be to keep them in some separate database outside of the Makefile. Whenever a file is compiled and the raw .d is generated, we run a script which looks up the header signature and matches it to a subset. Then it assigns that file to the correct subset (perhaps deleting it from an existing one). A step at the start of the build would regenerate the dependency file seen by make if the database indicates that it's out of date.
  - drothlis 4y ago
    Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.

actionfromafar 4y ago

I suppose these numbers must converge for all three as number of files go up? (And compile time for each file is not negligible.) I say this because in my use case ninja spends almost all of its time running compiler processes, not resolving deps.

drothlis 4y ago
I did hit a weird edge case in Make's job scheduling. Ninja will try to build things in the order that they are listed in the "build.ninja" file; make kind-of does too, but if any dependencies need to be built first, the later targets get pushed to the very end of the job queue. This means that you can end up running many resource-intensive linking jobs all at the same time at the very end of the build: https://lists.gnu.org/archive/html/help-make/2016-11/msg00006.html https://lists.gnu.org/archive/html/help-make/2016-11/msg0000... It's hard to say how much impact this would have in real-world scenarios.
- actionfromafar 4y ago
  I think I must have hit that case often, I have come to see it as inevitable that Make does a lot of linking at the end...