6 ms·

It’s hard to know whether to cache CI or not. On one hand, without the cache builds can be very slow. But on the other hand, you’ll see in a lot of projects r

It’s hard to know whether to cache CI or not. On one hand, without the cache builds can be very slow. But on the other hand, you’ll see in a lot of projects random commits like “blow away corrupted cache”, which makes you wonder whether building the cache from scratch is an important step of reproducible builds. I personally rather let the builds run longer and be absolutely certain. Maybe there’s a good middle ground for dev commits vs final merge commits, but unfortunately there’s no machinery in ie GitHub to specify a commit as final before merge.

whazor 5y ago
I remember there is an empty cache button in the UI of Gitlab.
- dnsmichi 5y ago
  Yep, in the pipeline view at the top right. https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache
exdsq 5y ago
It's not perfect but you could have a word in the commit message that the pipeline looks for and acts upon, so default using cache but let you not use it with "NO-CACHE: <message>"
- sharken 5y ago
  Great idea for a DevOps role, but the average developer will most likely not be aware of these features.
  - exdsq 5y ago
    Really? I assumed developers managed that bit of the CI pipeline
    - sharken 5y ago
      Most developers here are focused on business related development, to the point that many technical details are forgotten. It might be different in other companies though.
- aetherspawn 5y ago
  I had thought about this as an option. It's a good idea, but maybe it's not promoting a good separation of concerns between the CI machinery and the source control system.
IanCal 5y ago
Would a two step process work there? A staging branch which is always built from scratch and which is then merged into main? Or main & then tagged commits for releases?
whateveracct 5y ago
NixOS Gitlab Runners are quite nice in this regard. Caching for "free" (cost of admission: learning Nix)
- iduoad 5y ago
  Finally found a reason to learn Nix, thank you!
  - whateveracct 5y ago
    It is a lot of learning - you have to both run gitlab-runner on NixOS _and_ Nixify your project to get caching. If you climb the learning curve that far though - you'll be hooked.
dannyz 5y ago
I agree, tracking down a failed CI build that ends up being a cache problem is much more frustrating to me than waiting a little longer for the build.
- sharken 5y ago
  The attention to pipeline speed is great, i don't think I have seen anything this detailed before. Having implemented a pipeline cache to reduce wall time with 15 minutes for each execution, i wouldn't want to go back to no cache. But it's important to be resilient to a faulty cache. If that is hard to achieve, then it makes good sense to avoid caching.
- maccard 5y ago
  We cache builds on my work c++ projects. A clean build would be 2+ hours, using cached artifacts is about 5 minutes.
  - dnsmichi 5y ago
    Can you share more details about the size of the project (files count, build file size) and which caching settings you are using? I assume it is ccache and am curious how you use it.
    - maccard 5y ago
      It's a ue4 based game project. Unsure of the size offhand, sorry (and it's Sunday, so I'll check tomorrow). > I assume it is ccache and am curious how you use it. Nope. We just use the same machines for the same jobs (we have 3 agents, and we use teamcity which lets us pin our configurations to agents), and incremental builds. Really basic stuff. The machines are managed by terraform and packer and we rebuild the images once a month or so, so semi frequently pay the full compile hit. I am going to look into actual persistent caching for Mac very soon, it's likely we'll go with fastbuild!
nhoughto 5y ago
yep, we do a scheduled daily build with all caching off, to test that it all works without caching, and a good way to rebuild your caches without old dependencies/files etc. Good backstop.
dnsmichi 5y ago
True, CI caches are only one way to look at slow pipelines. When you are in control of the infrastructure where jobs and runners consume resources, other strategies might also help. Package dependency installs taking some time? Move the runners into network segments with blazing fast connections, next to mirrors of package registries (if not already provided). Possibility to use a CDN in front of the runners? Make sure the application code is capable of doing so. C/C++ with ccache can consume a lot of disk space, and caching itself can slow down the pipelines. Calculate the cost of pipeline runs and employees waiting for builds to finish, and compare it to buying more compute resources with xyz cores, lots of memory and direct SSD disk access to speed up the builds. That said, many different environments and workflows make decisions harder. Better observability for CI/CD is needed, having better insights into pipelines and see if and how efficiency improvements can have an impact. I have shared more thoughts in https://news.ycombinator.com/item?id=29520577 https://news.ycombinator.com/item?id=29520577

whazor 5y ago

I remember there is an empty cache button in the UI of Gitlab.

dnsmichi 5y ago
Yep, in the pipeline view at the top right. https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache

exdsq 5y ago

It's not perfect but you could have a word in the commit message that the pipeline looks for and acts upon, so default using cache but let you not use it with "NO-CACHE: <message>"

sharken 5y ago
Great idea for a DevOps role, but the average developer will most likely not be aware of these features.
- exdsq 5y ago
  Really? I assumed developers managed that bit of the CI pipeline
  - sharken 5y ago
    Most developers here are focused on business related development, to the point that many technical details are forgotten. It might be different in other companies though.
aetherspawn 5y ago
I had thought about this as an option. It's a good idea, but maybe it's not promoting a good separation of concerns between the CI machinery and the source control system.

IanCal 5y ago

Would a two step process work there? A staging branch which is always built from scratch and which is then merged into main? Or main & then tagged commits for releases?

whateveracct 5y ago

NixOS Gitlab Runners are quite nice in this regard. Caching for "free" (cost of admission: learning Nix)

iduoad 5y ago
Finally found a reason to learn Nix, thank you!
- whateveracct 5y ago
  It is a lot of learning - you have to both run gitlab-runner on NixOS _and_ Nixify your project to get caching. If you climb the learning curve that far though - you'll be hooked.

dannyz 5y ago

I agree, tracking down a failed CI build that ends up being a cache problem is much more frustrating to me than waiting a little longer for the build.

sharken 5y ago
The attention to pipeline speed is great, i don't think I have seen anything this detailed before. Having implemented a pipeline cache to reduce wall time with 15 minutes for each execution, i wouldn't want to go back to no cache. But it's important to be resilient to a faulty cache. If that is hard to achieve, then it makes good sense to avoid caching.
maccard 5y ago
We cache builds on my work c++ projects. A clean build would be 2+ hours, using cached artifacts is about 5 minutes.
- dnsmichi 5y ago
  Can you share more details about the size of the project (files count, build file size) and which caching settings you are using? I assume it is ccache and am curious how you use it.
  - maccard 5y ago
    It's a ue4 based game project. Unsure of the size offhand, sorry (and it's Sunday, so I'll check tomorrow). > I assume it is ccache and am curious how you use it. Nope. We just use the same machines for the same jobs (we have 3 agents, and we use teamcity which lets us pin our configurations to agents), and incremental builds. Really basic stuff. The machines are managed by terraform and packer and we rebuild the images once a month or so, so semi frequently pay the full compile hit. I am going to look into actual persistent caching for Mac very soon, it's likely we'll go with fastbuild!

nhoughto 5y ago

yep, we do a scheduled daily build with all caching off, to test that it all works without caching, and a good way to rebuild your caches without old dependencies/files etc. Good backstop.

dnsmichi 5y ago

True, CI caches are only one way to look at slow pipelines. When you are in control of the infrastructure where jobs and runners consume resources, other strategies might also help. Package dependency installs taking some time? Move the runners into network segments with blazing fast connections, next to mirrors of package registries (if not already provided). Possibility to use a CDN in front of the runners? Make sure the application code is capable of doing so. C/C++ with ccache can consume a lot of disk space, and caching itself can slow down the pipelines. Calculate the cost of pipeline runs and employees waiting for builds to finish, and compare it to buying more compute resources with xyz cores, lots of memory and direct SSD disk access to speed up the builds. That said, many different environments and workflows make decisions harder. Better observability for CI/CD is needed, having better insights into pipelines and see if and how efficiency improvements can have an impact. I have shared more thoughts in https://news.ycombinator.com/item?id=29520577 https://news.ycombinator.com/item?id=29520577