3 ms·

There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI st

There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI startup I worked at, self-hosting runners that used our own AMIs saved us a lot of time and egress costs, because so many CI steps begin by pulling some image, whether it be the base image or one you built at a previous step and pushed to your container registry. We also ran CI steps on some projects that required ARM or GPU. You also have the option of keeping a runner up during the busiest part of the day to benefit from cached images. We'd automatically rebuild the runner AMIs once a week and set runner lifetime to max 24 hours, ensuring freshness and ensuring that we didn't run into disk space issues.

SOLAR_FIELDS 3y ago
We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy
- barefeg 3y ago
  Could you give more details/resources on the distributed in cluster buildkit cache?
  - baby_souffle 3y ago
    I would also like to know more
  - SOLAR_FIELDS 3y ago
    We use Dagger’s implementation. The basic approach is to have a buildkit engine run as a daemonset on the cluster, and clients specify the same docker socket that buildkit uses. The magic is in cache synchronization, eg only lazily pulling layers as the client requests them. This is scalable but obviously since caching is hard there are some complexities with efficient synchronization of cache layers and cache volumes. This is currently a long lived service that runs as a deployment alongside a bunch of ephemeral runners to manage the cache synchronization. There are several other different architectures that range from simpler to more complex. The architecture I recommend people start out with is a single long lived beefy buildkit instance that a bunch of runners share, since that is much much simpler to implement. It of course has the downside that you have to refresh/rebuild the cache if the instance ever goes down. For runs that need read/write locks on volumes (eg Gradle build cache) my recommendation after trial and error to rsync those to the runners and then rsync them back after the run completes so you don’t have a bunch of locks fighting each other for the same folder.

SOLAR_FIELDS 3y ago

We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy

barefeg 3y ago
Could you give more details/resources on the distributed in cluster buildkit cache?
- baby_souffle 3y ago
  I would also like to know more
- SOLAR_FIELDS 3y ago
  We use Dagger’s implementation. The basic approach is to have a buildkit engine run as a daemonset on the cluster, and clients specify the same docker socket that buildkit uses. The magic is in cache synchronization, eg only lazily pulling layers as the client requests them. This is scalable but obviously since caching is hard there are some complexities with efficient synchronization of cache layers and cache volumes. This is currently a long lived service that runs as a deployment alongside a bunch of ephemeral runners to manage the cache synchronization. There are several other different architectures that range from simpler to more complex. The architecture I recommend people start out with is a single long lived beefy buildkit instance that a bunch of runners share, since that is much much simpler to implement. It of course has the downside that you have to refresh/rebuild the cache if the instance ever goes down. For runs that need read/write locks on volumes (eg Gradle build cache) my recommendation after trial and error to rsync those to the runners and then rsync them back after the run completes so you don’t have a bunch of locks fighting each other for the same folder.