3 ms·
I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x. One frustrating part is that a single GitHub runner can han
by e1g 4y ago
I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x.
One frustrating part is that a single GitHub runner can handle one job at a time, and our repos have 4-8 highly parallelized jobs. So we need multiple runners. To do this in an economic way, I made a Docker image and run 10 instances of it a single dedicated host from Hetzner. For ~$50/mo, we have unlimited CI and job run as fast as theoretically possible.
The CI box has no inbound ports open to the internet, and the OS is set to auto-update, so the maintenance burden is low.
- Delemono 4y agoThat sounds more like expected than an issue. You might not want to overload one instance. And running it like you do, is a no brainer anyway
- e1g 4y agoExecuting a GitHub runner in a container is a no-brainer, but it's still easy to overload the _host_ with too many jobs/containers, so managing resources is always up to the operator. I understand why GitHub has this constraint - to avoid clashes between jobs. If multiple jobs (from the same repository) are executing simultaneously within the same stateful environment, they are more likely to clash over shared resources (/tmp, cleanup tasks, database names, etc.). However, even if my jobs are clean and idempotent, GitHub is nudging me to think about runners as "VMs" rather than "containers" (because often CI jobs involve their own containers, and docker-in-docker is a pain), and "self-hosting a bunch of CI VMs" becomes expensive rather quick.
- naikrovek 4y agowell, GHA supports running actions in containers, which is the best way to control the environment your CI/CD runs in, so putting your runner in a container won't work for everyone. but, if it does work for someone, doing what you've done will give a much better experience to their developers. my employer uses single-user VMs for it's runners; it works well, but sometimes actions invocation is high and it can take a few minutes for a runner to come around to taking my job. that would be much less of a problem with dockerized runners.
- how_gauche 4y agoDeveloper velocity has been my project for the last quarter at work: we just switched from GitHub runners to Hetzner machines, and saw similar improvements to average build times. Between self hosting our runners and switching to bazel which caches aggressively, we've driven down average CI test time from 23 min to 2m40.
- jokethrowaway 4y agoCan your jobs use dockers or do you run into the docker-in-docker issue?
- e1g 4y agoNo Docker for us. I baked all dependencies (DB/Redis/etc) into the main image with the GitHub runner to maximize performance. This works well for us as a relatively small product team that tightly controls all environments. Modularity and flexibility come with a cost, and it's not worth it (for us). One alternative is to use Firecracker on the CI host and run entire VMs for jobs (still quick). However, I am not comfortable with that stack to do "set & forget" implementation with no ongoing burden.