4 ms·

The use case is really niche. The only time it comes in handy is when: 1. You have enormous files AND 2. You have a little number of files.

rumanator 7y ago
Yes, I agree. In a world where all projects are comprised of a myriad source files where each one takes a fraction of a second to compile, this kind of approach sounds a tad pointless.
ris 7y ago
I'm not sure about that. The traditional way of exploiting parallelism for large builds is to work on a per-file basis. Want more parallelism? Fork more jobs. But this can fall apart in a couple of ways: 1. It can get extremely expensive in memory use. Your parallelism limit can be dictated by the memory your machine has. Spending your available cores over fewer simultaneous compilation units could also have benefits arising from data locality, even in situations where a machine's memory headroom isn't a concern. 2. Many large build processes have serialized steps which are currently unable to benefit from any form of parallelism. This is becoming even more the case with the rise of IPO & LTO. But as ever, we'll probably have to wait to find out how large the practical benefits end up being from this work.
mrich 7y ago
It is quite a common case. First you do your scratch make with 800 jobs which maxes out your machine just from preprocessing :) Thereafter you only edit a small portion of the code and recompile a small number of files, which only uses a few cores. It's especially problematic for C++ with all the template instantiations. Some files in our codebase take 40+ sec to compile.
mxcrossb 7y ago
Agreed. I do know some projects that do a lot of code generation that end up with huge files, though this is probably largely caused by C++’s templates forcing too much into headers (which newer versions fix). It’d also be nice if parallel make had better load balancing.

rumanator 7y ago

Yes, I agree. In a world where all projects are comprised of a myriad source files where each one takes a fraction of a second to compile, this kind of approach sounds a tad pointless.

ris 7y ago

I'm not sure about that. The traditional way of exploiting parallelism for large builds is to work on a per-file basis. Want more parallelism? Fork more jobs. But this can fall apart in a couple of ways: 1. It can get extremely expensive in memory use. Your parallelism limit can be dictated by the memory your machine has. Spending your available cores over fewer simultaneous compilation units could also have benefits arising from data locality, even in situations where a machine's memory headroom isn't a concern. 2. Many large build processes have serialized steps which are currently unable to benefit from any form of parallelism. This is becoming even more the case with the rise of IPO & LTO. But as ever, we'll probably have to wait to find out how large the practical benefits end up being from this work.

mrich 7y ago

It is quite a common case. First you do your scratch make with 800 jobs which maxes out your machine just from preprocessing :) Thereafter you only edit a small portion of the code and recompile a small number of files, which only uses a few cores. It's especially problematic for C++ with all the template instantiations. Some files in our codebase take 40+ sec to compile.

mxcrossb 7y ago

Agreed. I do know some projects that do a lot of code generation that end up with huge files, though this is probably largely caused by C++’s templates forcing too much into headers (which newer versions fix). It’d also be nice if parallel make had better load balancing.