5 ms·

Build Time! I will pay a great deal of money to reduce my build times. Building is readily parallelizable and with a good enough build scripts and system desig

by sqeaky 9y ago

Build Time! I will pay a great deal of money to reduce my build times. Building is readily parallelizable and with a good enough build scripts and system design there are several smaller linking steps instead of one huge serialized one at the end. On my current system building the code I care about only takes about 2 minutes, about 5MLOCs of C++. Anything more than 10 seconds is enough to get stuck in the time waste that is Reddit... or HN... I am getting back to work now, my build is likely done.

musha68k 9y ago
I now know why they sell so many of those "fidget spinners" these days – it's all the core-starved developers like myself not wanting to get stuck on HN while waiting for their builds to finish ;D
jhasse 9y ago
> and system design there are several smaller linking steps instead of one huge serialized one at the end. Have you tried the GNU Gold linker? It's makes those serialized linking steps very fast and even supports concurrent linking :)
- sqeaky 9y ago
  Yes, and it took about 5% off my link step. It was nice, but no a fundamental game changer. I suspect someone who didn't already a bunch of smaller link steps would benefit more.
  - starsinspace 9y ago
    Gold also supports incremental linking [0], where it basically modifies the existing binary by only replacing the changed parts of it. I've never used it myself so I don't know how well it works, but you might want to look into it if you haven't already. [0] https://gcc.gnu.org/wiki/GoldIncrementalLinking https://gcc.gnu.org/wiki/GoldIncrementalLinking
    - sqeaky 9y ago
      Well that is something I hadn't heard of. Now I am curious how hard it would be to get CMake to do this?
lfowles 9y ago
Oof, I'm limited to building at -j4 due to memory constraints, not CPU constraints.
- sqeaky 9y ago
  How much memory do you have and what are building? Last I was memory constrained I was trying to build Clang on a Raspberry Pi.
  - lfowles 9y ago
    16 gigs working on internal code, but apparently the culprit is boost::units....
JdeBP 9y ago
The problem is that builds are not just a concatentation of CPU-bound things. There's more to a build than that. An interesting item came up recently. On Windows 10 there's a problem with highly parallel builds that is down to the fact that process exit cleanup is serialized inside the GDI on a lock that also needs to be taken during message input. Combine that with a build that splits things out into lots of compilation units that are all built in parallel using a process each, and fun ensues. Ironically, it means that during the build you are not able to get stuck into anything else with a GUI. (-: * https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/ https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and... (https://news.ycombinator.com/item?id=14733829 https://news.ycombinator.com/item?id=14733829)

by sqeaky 9y ago

musha68k 9y ago

I now know why they sell so many of those "fidget spinners" these days – it's all the core-starved developers like myself not wanting to get stuck on HN while waiting for their builds to finish ;D

jhasse 9y ago

> and system design there are several smaller linking steps instead of one huge serialized one at the end. Have you tried the GNU Gold linker? It's makes those serialized linking steps very fast and even supports concurrent linking :)

sqeaky 9y ago
Yes, and it took about 5% off my link step. It was nice, but no a fundamental game changer. I suspect someone who didn't already a bunch of smaller link steps would benefit more.
- starsinspace 9y ago
  Gold also supports incremental linking [0], where it basically modifies the existing binary by only replacing the changed parts of it. I've never used it myself so I don't know how well it works, but you might want to look into it if you haven't already. [0] https://gcc.gnu.org/wiki/GoldIncrementalLinking https://gcc.gnu.org/wiki/GoldIncrementalLinking
  - sqeaky 9y ago
    Well that is something I hadn't heard of. Now I am curious how hard it would be to get CMake to do this?

lfowles 9y ago

Oof, I'm limited to building at -j4 due to memory constraints, not CPU constraints.

sqeaky 9y ago
How much memory do you have and what are building? Last I was memory constrained I was trying to build Clang on a Raspberry Pi.
- lfowles 9y ago
  16 gigs working on internal code, but apparently the culprit is boost::units....

JdeBP 9y ago

The problem is that builds are not just a concatentation of CPU-bound things. There's more to a build than that. An interesting item came up recently. On Windows 10 there's a problem with highly parallel builds that is down to the fact that process exit cleanup is serialized inside the GDI on a lock that also needs to be taken during message input. Combine that with a build that splits things out into lots of compilation units that are all built in parallel using a process each, and fun ensues. Ironically, it means that during the build you are not able to get stuck into anything else with a GUI. (-: * https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/ https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and... (https://news.ycombinator.com/item?id=14733829 https://news.ycombinator.com/item?id=14733829)