7 ms·

If a Build Takes 4 Hours, Run It Every 4 Hours

17 points by royosherove 6y ago

tlarkworthy 6y ago
Why don't we run it on every change?
- bryanrasmussen 6y ago
  If a build takes 4 hours it argues that a build on that system is a pretty expensive operation, unless changes are very infrequent, which in most systems is not the case, a build run on every change would seem to be highly problematic.
  - tlarkworthy 6y ago
    Bingo it's expensive, which is why they set it to every 24 hours. Also coz they don't want to add the queuing logic, so it takes 4 hours to run is an approximation. Also it messes with other systems so they prefer to run it overnight. This blog doesn't solve a real problem.
    - royosherove 6y ago
      Author here. It's literally the first line in the blog post. And it links to this blog post about static environments: https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments https://pipelinedriven.org/article/the-four-biggest-issues-w...
      - tlarkworthy 6y ago
        That doesn't answer anything. Either you: minimize latency (run every change independent of build time) or run to minimize engineering (24 hours). I dunno why you couple build time to a static period, it does not make sense to me.
  - lonelappde 6y ago
    Builds are cheap compared to running in production. Stop being a cheapskate.
    - bryanrasmussen 6y ago
      I don't understand your running in production comment in this context, nor actually your stop being a cheapskate aimed at me.
- royosherove 6y ago
  That would be ideal, but many organizations don't yet have that capacity for large builds/deploys (yet).
- eska 6y ago
  It's a bandwith vs latency issue. Imagine you make 5 commits in 4 hours. You now have 5 builds queued up and 20 hours of latency. So one would probably prefer to only build the last commit, and maybe queue the older commits up for later. But if you make another change then you'd need to pause the build of the older commit, or you'd be at up to 8 hours of latency instead of 4..
  - gpvos 6y ago
    If you do them in parallel, your latency is still 4 hours. Okay, it's not cheap.
_ph_ 6y ago
I don't know what the author is building that a build is taking 4 hours, but I am aware that there are huge software projects with build times like that and much longer. I agree that getting quick feedback to changes is vital for quick overall development. This shows the values of development stacks which reduce the build times a lot. As a resultmore incremental development can be done. For example, the build times were one of the architectural goals when designing the Go language. Dynamic languages are also a great tool to fight build times. Even if it is unsuitable for large projects to be entirely built on dynamic languages, a great technique is using a mix of static and dynamic. Build the core of the application in a static language, but more as a set of libraries to be called from the dynamic layer. The dynamic layer is where most of the application logic resides, easy to be changed. And parts which have stabilized reasonably can still be moved over to the static part.
- mister_hn 6y ago
  Have you ever tried to build Chromium or Firefox yourself? If you have a slow machine, the build can take even 10 hours or more
  - _ph_ 6y ago
    No, I have not. But wouldn't be those code bases which compile very slow because of being written in C++? C++ seems to baloon in building time for large code bases. On the other side, the big Threadripper manages to compile the Linux Kernel in 40s. I don't think those web browsers are that much bigger than Linux than the compile times differ. Firefox is further slowed down by using more and more Rust based code - Rust is known to compile slowly. However I would say this is a good tradeoff, since Rust brings much more safety to one of the most safety-critical applications on a typical desktop.
    - rhoyerboat 6y ago
      the general issue is that the feature rich web browser contains libraries wrapped around basically every leaf and stem of windows, mac and *nix desktop interface family forests. (if your building on say, a completely fresh stage3 gentoo system, that equals some pretty good linker load. cairo comes to mind as a huge culprit, whether its webkit or mozilla.)
tedunangst 6y ago
Tying your build frequency to build time and then using that as motivation to decrease build time is probably the important bit here, but only if you include the second part.
- royosherove 6y ago
  That's a good way of putting it. In TOC terms, the build time becomes the new constraint to solve, but only after you solve the build frequency constraint.
gregdoesit 6y ago
Alternatively, you can parallelise parts of your build (eg the tests, static analysis etc) and speed it up. Then, if you have more changes coming in than you could parallelise is when you can start to do some advanced things like trying to predict which changes might break the build. There was a longer discussion on one of these approaches done by Uber[1] on HN. Granted it’s a lot of investment, it’s interesting to explore ways of dealing with long builds that need to run frequently. https://news.ycombinator.com/item?id=19692820 https://news.ycombinator.com/item?id=19692820
- royosherove 6y ago
  True, but that type of optimization is quite moot if you're only running the build at night (i.e, that's not the first constraint to solve). In the article I mention that first get the builds to be tight one after another, then work on build times because that becomes the constraint.
- larrybud 6y ago
  Or, just parallelize the entire build. eg, if you have four build servers, you could kick off a build every hour. Easy using VM's, trivial in the cloud.
noisy_boy 6y ago
We have Maven builds that complete in a few mins via Jenkins but I have been exploring if it can be done even faster. The builds are written using the Jenkins declarative pipeline with parallelization etc but feels like it is just not fast enough (the slowness is probably because of the whole lot of intermediate IO/back-and-forth between Jenkins master and slaves). Wish there was a build setup that was faster while being as feature rich as Jenkins while supporting Maven builds.
- mister_hn 6y ago
  Try the flag -T XC where X is the number of threads you want (similar to make -jX)
  - noisy_boy 6y ago
    Yeah - I did try it + the other oft-recommended optimizations. I have arrived at the conclusion that the actual build is not taking the majority of the overall build time, it is the various before and after that happens (e.g. checkout git repo, deciding build parameters, sonar qube scans, creating git tags and so on). Since the Jenkins pipeline basically runs shell scripts to do these things in each stage, the overall build just feels slow. Collapsing the stages would make things more messy, keeping them separate is cleaner but means more per-stage overhead.
    - mister_hn 6y ago
      I would suggest to keep the shell scripts calls to the minimum (e.g. compact commands together if possible). You can still have nice stages separations, but try to avoid the pitfall of doing too much shell invocations. A good trick is to create a fake "multi-module reactor" pom.xml and grouping all the modules logically there, then you build like mvn clean install -f reactor-pom.xml -T 8C
      - noisy_boy 6y ago
        Thanks for the suggestions. We have very few multi-module projects. Most of the time, the time is spent in the various stages - almost all of them shell scripts. I guess there is overhead of initializing the stages/spawning etc. I also think that Jenkins could be doing some serialization to be able to resume a failed build - which we don't really need as we'll probably start a fresh build anyway. That's my next area to explore.
barry27 6y ago
I didn't read the article or any of the comments. I think simple logic like this is what is lacking nowadays. If a build takes 4 hours, run it every 4 hours. Yes. I didn't read the article or any of the comments. I don't care about the article, and certainly not about the comments. But if a build takes 4 hours, run it every 4 hours...that speaks to me.
draw_down 6y ago
A 4 hour build is going to slow you down no matter what. Ok, run it every 4 hours. Now when the build breaks you get to hunt through 4 hours worth of commits to find the problem. Sorry but this is still hell. Fix the build so it doesn’t take 4 hours and run it on each change.
DmitryOlshansky 6y ago
Why not every hour or even half-hour? Provided that you can run multiple builds in parallel but each of them has somehow limited parallelization.
teleforce 6y ago
If the long build time for static typing programming language is the concerns perhaps you can try have look at D programming language since it is optimized for fast compilation [1]. Around last year someone even managed to speed up its compilation time even further and it is also featured in Hacker News but cannot find the link now. I think in the future most of the static typing programming languages will be mainly based on Single Static Assignment (SSA) technique then the compilation could really fly. [1]https://www.drdobbs.com/cpp/increasing-compiler-speed-by-over-75/240158941 https://www.drdobbs.com/cpp/increasing-compiler-speed-by-ove... [2]https://www.cs.princeton.edu/~appel/papers/ssafun.pdf https://www.cs.princeton.edu/~appel/papers/ssafun.pdf

17 points by royosherove 6y ago

tlarkworthy 6y ago

Why don't we run it on every change?

bryanrasmussen 6y ago
If a build takes 4 hours it argues that a build on that system is a pretty expensive operation, unless changes are very infrequent, which in most systems is not the case, a build run on every change would seem to be highly problematic.
- tlarkworthy 6y ago
  Bingo it's expensive, which is why they set it to every 24 hours. Also coz they don't want to add the queuing logic, so it takes 4 hours to run is an approximation. Also it messes with other systems so they prefer to run it overnight. This blog doesn't solve a real problem.
  - royosherove 6y ago
    Author here. It's literally the first line in the blog post. And it links to this blog post about static environments: https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments https://pipelinedriven.org/article/the-four-biggest-issues-w...
    - tlarkworthy 6y ago
      That doesn't answer anything. Either you: minimize latency (run every change independent of build time) or run to minimize engineering (24 hours). I dunno why you couple build time to a static period, it does not make sense to me.
- lonelappde 6y ago
  Builds are cheap compared to running in production. Stop being a cheapskate.
  - bryanrasmussen 6y ago
    I don't understand your running in production comment in this context, nor actually your stop being a cheapskate aimed at me.
royosherove 6y ago
That would be ideal, but many organizations don't yet have that capacity for large builds/deploys (yet).
eska 6y ago
It's a bandwith vs latency issue. Imagine you make 5 commits in 4 hours. You now have 5 builds queued up and 20 hours of latency. So one would probably prefer to only build the last commit, and maybe queue the older commits up for later. But if you make another change then you'd need to pause the build of the older commit, or you'd be at up to 8 hours of latency instead of 4..
- gpvos 6y ago
  If you do them in parallel, your latency is still 4 hours. Okay, it's not cheap.

_ph_ 6y ago

I don't know what the author is building that a build is taking 4 hours, but I am aware that there are huge software projects with build times like that and much longer. I agree that getting quick feedback to changes is vital for quick overall development. This shows the values of development stacks which reduce the build times a lot. As a resultmore incremental development can be done. For example, the build times were one of the architectural goals when designing the Go language. Dynamic languages are also a great tool to fight build times. Even if it is unsuitable for large projects to be entirely built on dynamic languages, a great technique is using a mix of static and dynamic. Build the core of the application in a static language, but more as a set of libraries to be called from the dynamic layer. The dynamic layer is where most of the application logic resides, easy to be changed. And parts which have stabilized reasonably can still be moved over to the static part.

mister_hn 6y ago
Have you ever tried to build Chromium or Firefox yourself? If you have a slow machine, the build can take even 10 hours or more
- _ph_ 6y ago
  No, I have not. But wouldn't be those code bases which compile very slow because of being written in C++? C++ seems to baloon in building time for large code bases. On the other side, the big Threadripper manages to compile the Linux Kernel in 40s. I don't think those web browsers are that much bigger than Linux than the compile times differ. Firefox is further slowed down by using more and more Rust based code - Rust is known to compile slowly. However I would say this is a good tradeoff, since Rust brings much more safety to one of the most safety-critical applications on a typical desktop.
  - rhoyerboat 6y ago
    the general issue is that the feature rich web browser contains libraries wrapped around basically every leaf and stem of windows, mac and *nix desktop interface family forests. (if your building on say, a completely fresh stage3 gentoo system, that equals some pretty good linker load. cairo comes to mind as a huge culprit, whether its webkit or mozilla.)

tedunangst 6y ago

Tying your build frequency to build time and then using that as motivation to decrease build time is probably the important bit here, but only if you include the second part.

royosherove 6y ago
That's a good way of putting it. In TOC terms, the build time becomes the new constraint to solve, but only after you solve the build frequency constraint.

gregdoesit 6y ago

Alternatively, you can parallelise parts of your build (eg the tests, static analysis etc) and speed it up. Then, if you have more changes coming in than you could parallelise is when you can start to do some advanced things like trying to predict which changes might break the build. There was a longer discussion on one of these approaches done by Uber[1] on HN. Granted it’s a lot of investment, it’s interesting to explore ways of dealing with long builds that need to run frequently. https://news.ycombinator.com/item?id=19692820 https://news.ycombinator.com/item?id=19692820

royosherove 6y ago
True, but that type of optimization is quite moot if you're only running the build at night (i.e, that's not the first constraint to solve). In the article I mention that first get the builds to be tight one after another, then work on build times because that becomes the constraint.
larrybud 6y ago
Or, just parallelize the entire build. eg, if you have four build servers, you could kick off a build every hour. Easy using VM's, trivial in the cloud.

noisy_boy 6y ago

We have Maven builds that complete in a few mins via Jenkins but I have been exploring if it can be done even faster. The builds are written using the Jenkins declarative pipeline with parallelization etc but feels like it is just not fast enough (the slowness is probably because of the whole lot of intermediate IO/back-and-forth between Jenkins master and slaves). Wish there was a build setup that was faster while being as feature rich as Jenkins while supporting Maven builds.

mister_hn 6y ago
Try the flag -T XC where X is the number of threads you want (similar to make -jX)
- noisy_boy 6y ago
  Yeah - I did try it + the other oft-recommended optimizations. I have arrived at the conclusion that the actual build is not taking the majority of the overall build time, it is the various before and after that happens (e.g. checkout git repo, deciding build parameters, sonar qube scans, creating git tags and so on). Since the Jenkins pipeline basically runs shell scripts to do these things in each stage, the overall build just feels slow. Collapsing the stages would make things more messy, keeping them separate is cleaner but means more per-stage overhead.
  - mister_hn 6y ago
    I would suggest to keep the shell scripts calls to the minimum (e.g. compact commands together if possible). You can still have nice stages separations, but try to avoid the pitfall of doing too much shell invocations. A good trick is to create a fake "multi-module reactor" pom.xml and grouping all the modules logically there, then you build like mvn clean install -f reactor-pom.xml -T 8C
    - noisy_boy 6y ago
      Thanks for the suggestions. We have very few multi-module projects. Most of the time, the time is spent in the various stages - almost all of them shell scripts. I guess there is overhead of initializing the stages/spawning etc. I also think that Jenkins could be doing some serialization to be able to resume a failed build - which we don't really need as we'll probably start a fresh build anyway. That's my next area to explore.

barry27 6y ago

I didn't read the article or any of the comments. I think simple logic like this is what is lacking nowadays. If a build takes 4 hours, run it every 4 hours. Yes. I didn't read the article or any of the comments. I don't care about the article, and certainly not about the comments. But if a build takes 4 hours, run it every 4 hours...that speaks to me.

draw_down 6y ago

A 4 hour build is going to slow you down no matter what. Ok, run it every 4 hours. Now when the build breaks you get to hunt through 4 hours worth of commits to find the problem. Sorry but this is still hell. Fix the build so it doesn’t take 4 hours and run it on each change.

DmitryOlshansky 6y ago

Why not every hour or even half-hour? Provided that you can run multiple builds in parallel but each of them has somehow limited parallelization.

teleforce 6y ago

If the long build time for static typing programming language is the concerns perhaps you can try have look at D programming language since it is optimized for fast compilation [1]. Around last year someone even managed to speed up its compilation time even further and it is also featured in Hacker News but cannot find the link now. I think in the future most of the static typing programming languages will be mainly based on Single Static Assignment (SSA) technique then the compilation could really fly. [1]https://www.drdobbs.com/cpp/increasing-compiler-speed-by-over-75/240158941 https://www.drdobbs.com/cpp/increasing-compiler-speed-by-ove... [2]https://www.cs.princeton.edu/~appel/papers/ssafun.pdf https://www.cs.princeton.edu/~appel/papers/ssafun.pdf