3 ms·
The big problem with distributed compilation is not the distribution of jobs (there are tons of ways to do that), but resolving the dependency information corre
by jgrahamc 15y ago
The big problem with distributed compilation is not the distribution of jobs (there are tons of ways to do that), but resolving the dependency information correctly. I founded Electric Cloud (http://www.electric-cloud.com/ http://www.electric-cloud.com/) to solve this problem and we ended up needing to:
1. Completely understand the build process from the build script (e.g. Makefile)
2. Write a parallel distributed, versioned file system so that we could automatically spot the dependencies that were not mentioned in the build script.
Once we'd done that we were able to get very high parallelism from unmodified build scripts which was impossible with tools like make -j, distcc, etc. The reason is that almost all build scripts assume serial builds and thus do things like reusing a temporary file name which breaks parallelism. Or you have nightmares like a shared precompiled header file that everyone is updating.
Once you crank up the parallelism then you can start worrying about disk latency, network latency etc. That led us to do everything in RAM on the worker nodes, our own binary protocol with compression, and peer-to-peer communication inside the worker node cluster to maximize switch utilization and minimize master load.