5 ms·

> "Of course our build takes long, we have over 10,000 lines of code!" Well, my project misses the 10 minute target by a factor of four, but does include tests

> "Of course our build takes long, we have over 10,000 lines of code!" Well, my project misses the 10 minute target by a factor of four, but does include tests and have 670kloc of C++. I claim continuous in this context, thanks very much.

marcv81 10y ago
Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.
- verytrivial 10y ago
  Indeed it could, and you're looking at the result, down from 2.5h-ish when I arrived. Our pipeline runneth deep ... (Though of course we'll never stop whittling!)
  - marcv81 10y ago
    Aw man!
  - deleted 10y ago
    [deleted]
- MaulingMonkey 10y ago
  I've been on projects that did this - just linking everything back together if you touch a widely used subproject can blow that 10 minute budget too, for a single build flavor. When porting things for game development, I routinely have 30+ build flavors (say, 5 platforms, 6 configs per), and want to test at least a few of them to have even a remote chance of the checkin going green when touching core platform abstractions or config-specific stuff. A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky. Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...
  - solarengineer 10y ago
    I quite agree with you! I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.
pjc50 10y ago
500kloc here! Produces a 200MB binary in debug, 8MB in release. We managed to get it down under about 15 minutes for the most common platform by using concatenation builds. It's much much faster to build half a dozen files than a thousand. At a previous employer we were up to 1.25mloc by the time I left and we had a different solution: ccache+distcc distributing the build across a room full of computers. Few files took longer than 5m to build all by themselves, and the link phase also took about 5m. (Splitting up large C++ projects that have grown organically is really hard. It requires re-architecting and commitment to spend time on this technical debt.)
tyingq 10y ago
Do you experience any real consequence from this, as the author suggests? I don't deny that can happen, but I have seen teams with build times that exceed 10 minutes that don't have issues. People aren't generally dumb, so they find a way to work that doesn't involve constantly sitting around waiting on the build.
- Macha 10y ago
  Moved from a team that had build times of 15 minutes to a team that had build times of hours, it felt crippling :(
  - tyingq 10y ago
    Hours, sure. But your original 15 minute build time exceeds the author's 10 minute cutoff. I'm questioning whether that seemed to be a huge issue.
- verytrivial 10y ago
  We do, yes. The main one being developers will often have to make a judgement call regarding only doing partial builds and/or coverage so that the commit is not always certain to work during CI (despite what they pwomise to do regarding pre-commit testing). It also means that they have a chunk of mental load that lingers after the commit, the bot might DING them and they'll need to switch back. I don't think the distinction between 10 and 40 minutes is as much as the 40 to 2.5 hours difference -- any problem worth solving is going to take you half and hour IMHO (but let's not bike-shed that), but 2.5 hours you can start a whole investigation from scratch etc...
LegNeato 10y ago
If you switch to a modern build system like Bazel or Buck with remote caching, you will likely get that down to < 5 mins per change...
- verytrivial 10y ago
  They look interesting! I keep an eye on these re: Windows support (we already use IncrediBuild there, and distcc and ccache on Linux ...) We have some monster links that can take upwards of 8 minutes on our fastest machine with SSDs etc, so the 5 mins seems keen.

marcv81 10y ago

Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.

verytrivial 10y ago
Indeed it could, and you're looking at the result, down from 2.5h-ish when I arrived. Our pipeline runneth deep ... (Though of course we'll never stop whittling!)
- marcv81 10y ago
  Aw man!
- deleted 10y ago
  [deleted]
MaulingMonkey 10y ago
I've been on projects that did this - just linking everything back together if you touch a widely used subproject can blow that 10 minute budget too, for a single build flavor. When porting things for game development, I routinely have 30+ build flavors (say, 5 platforms, 6 configs per), and want to test at least a few of them to have even a remote chance of the checkin going green when touching core platform abstractions or config-specific stuff. A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky. Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...
- solarengineer 10y ago
  I quite agree with you! I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.

pjc50 10y ago

500kloc here! Produces a 200MB binary in debug, 8MB in release. We managed to get it down under about 15 minutes for the most common platform by using concatenation builds. It's much much faster to build half a dozen files than a thousand. At a previous employer we were up to 1.25mloc by the time I left and we had a different solution: ccache+distcc distributing the build across a room full of computers. Few files took longer than 5m to build all by themselves, and the link phase also took about 5m. (Splitting up large C++ projects that have grown organically is really hard. It requires re-architecting and commitment to spend time on this technical debt.)

tyingq 10y ago

Do you experience any real consequence from this, as the author suggests? I don't deny that can happen, but I have seen teams with build times that exceed 10 minutes that don't have issues. People aren't generally dumb, so they find a way to work that doesn't involve constantly sitting around waiting on the build.

Macha 10y ago
Moved from a team that had build times of 15 minutes to a team that had build times of hours, it felt crippling :(
- tyingq 10y ago
  Hours, sure. But your original 15 minute build time exceeds the author's 10 minute cutoff. I'm questioning whether that seemed to be a huge issue.
verytrivial 10y ago
We do, yes. The main one being developers will often have to make a judgement call regarding only doing partial builds and/or coverage so that the commit is not always certain to work during CI (despite what they pwomise to do regarding pre-commit testing). It also means that they have a chunk of mental load that lingers after the commit, the bot might DING them and they'll need to switch back. I don't think the distinction between 10 and 40 minutes is as much as the 40 to 2.5 hours difference -- any problem worth solving is going to take you half and hour IMHO (but let's not bike-shed that), but 2.5 hours you can start a whole investigation from scratch etc...

LegNeato 10y ago

If you switch to a modern build system like Bazel or Buck with remote caching, you will likely get that down to < 5 mins per change...

verytrivial 10y ago
They look interesting! I keep an eye on these re: Windows support (we already use IncrediBuild there, and distcc and ccache on Linux ...) We have some monster links that can take upwards of 8 minutes on our fastest machine with SSDs etc, so the 5 mins seems keen.