3 ms·
I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-cons
by maxmcd 2y ago
I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-consuming way to cache things.
Maybe building from scratch all the time is a good correctness decision? Maybe stale values in disks is a tricky enough issue to want to avoid entirely?
If you keep a stack of disks around and grab a free one when the job starts you'd end up with good speedup a lot of the time. If cost is an issue you can expire them quickly. I regularly see CI jobs spending >50% of their time downloading the same things, or compiling the same things, over and over. How many times have I triggered an action that compiled the exact same sqlite source code? Tens of thousands?
Maybe this is fine, I dunno.
- parentheses 2y agoI agree. The notion that everything must be docker is nice in principle but requires a lot of performance optimization work early on. Earlier than one would need with "sticky disks" as you called them.
- adityamaru 2y agoThis is exactly the sort of insight that led us to work on Blacksmith. Since we own the hardware we run CI jobs on there are some exciting things we can do to make these "sticky disks" work the way you describe it. Stay tuned!
- deleted 2y ago[deleted]
- withinboredom 2y agoInteresting. I remember working on a project where the first clean build would always fail, and only incremental builds could succeed. I was a junior at the time, so this was 15-20 years ago. I remember spending some time trying to get it to succeed from a clean build and my lead pulling me aside: he said it was an easy fix, but if we fixed it, the ops guys would insist on building from scratch for every build. So please, stop. Personally, unless you have an exotic build env, it’s usually faster and easier to simply build in the runner. If you need a docker image at the end, build a dockerfile that simply copies the artifacts from disk.
- aayushshah15 2y agoA subtle challenge with "sticky disks" is that it requires your workflow steps to be idempotent beyond the point of "resumption", which can be tricky in a lot of cases.
- kylegalbraith 2y agoCouldn't agree more. Somewhere, we lost the concept of disks in CI unless you run it yourself, and a lot of build tools could benefit from having them. We came to the same conclusion and built Depot around this exact workflow for the Docker image build problem. We're now bringing that same tech into GitHub Actions workflows.