6 ms·

Meta: Absolutely superb readme for an open-source project. Can anyone point to a better opensource readme today? That said, do many see value is in this? The c

Meta: Absolutely superb readme for an open-source project. Can anyone point to a better opensource readme today? That said, do many see value is in this? The current stance is you either self-host or accept paying github's runner rates out of laziness. In the end it's all very mild when it comes to extra costs. When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI and in the end finding "oh it was a re-provisioning of compute", so...let's run it again and hope that goes well, actually no...let's schedule it for 11pm Eastern which probably isn't that optimal. I can see it working for some, but horrendous for others. Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. Still, a marvellous readme. If only others in open-source would take note that this is how you do it.

lukevp 3y ago
Cost isn’t the only vector. Someone may want to host the runners within a given VPC so they have access to private resources / PrivateLink etc. the job API is a lot narrower of an attack surface area than giving your auth info to GitHub that has enough power to do CI/CD infra changes and such. There’s also people who use GitHub Enterprise who still need to host a runner fleet, so they could benefit from this too.
- Grimburger 3y ago
  Sure, but does my point about fallible spot not still stand?
  - quickthrower2 3y ago
    I agree usually “CI is running” means “human is sorta blocked waiting on this”* means the compute cost to run it reliably is probably negligible. Whether you this on the Github network or your own private cloud is then an orthogonal concern. Spot is silly although spot with immediate fallback to in demand might be a neat hack. * Dependabot and AI PRs might be exceptions.
- stoobs 3y ago
  This. We’re using the this at my day job for the requirement for the runner to access private resources such as internal api’s etc
- raffraffraff 3y ago
  There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI startup I worked at, self-hosting runners that used our own AMIs saved us a lot of time and egress costs, because so many CI steps begin by pulling some image, whether it be the base image or one you built at a previous step and pushed to your container registry. We also ran CI steps on some projects that required ARM or GPU. You also have the option of keeping a runner up during the busiest part of the day to benefit from cached images. We'd automatically rebuild the runner AMIs once a week and set runner lifetime to max 24 hours, ensuring freshness and ensuring that we didn't run into disk space issues.
  - SOLAR_FIELDS 3y ago
    We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy
    - barefeg 3y ago
      Could you give more details/resources on the distributed in cluster buildkit cache?
      - baby_souffle 3y ago
        I would also like to know more
      - SOLAR_FIELDS 3y ago
        We use Dagger’s implementation. The basic approach is to have a buildkit engine run as a daemonset on the cluster, and clients specify the same docker socket that buildkit uses. The magic is in cache synchronization, eg only lazily pulling layers as the client requests them. This is scalable but obviously since caching is hard there are some complexities with efficient synchronization of cache layers and cache volumes. This is currently a long lived service that runs as a deployment alongside a bunch of ephemeral runners to manage the cache synchronization. There are several other different architectures that range from simpler to more complex. The architecture I recommend people start out with is a single long lived beefy buildkit instance that a bunch of runners share, since that is much much simpler to implement. It of course has the downside that you have to refresh/rebuild the cache if the instance ever goes down. For runs that need read/write locks on volumes (eg Gradle build cache) my recommendation after trial and error to rsync those to the runners and then rsync them back after the run completes so you don’t have a bunch of locks fighting each other for the same folder.
- skwashd 3y ago
  Add network egress controls and telemetry as two more benefits of self hosted runners. Denied network connections are a sign there could be a problem with a dependency.
tlarkworthy 3y ago
Absolutely, there are lots of workloads that do not run on actions and require custom hardware (GPU? Big boxes) or greater privileges to Linux API than actions provide, or just better caching.
- dataking 3y ago
  This. You even need to self-host if you want to test code that uses AVX512 outside of an emulator since the default runners do not support that. Same if you want to test aarch64-specific code paths on Linux, Windows, or macOS.
andrus 3y ago
> That said, do many see value is in this? GitHub don't host runners for Linux × arm64, so if you need this, you need to self-host. You can also run custom AMIs with pre-installed packages, which can speed up workflows that depend on those packages. > When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI… I'm on a small team who've been running the Philips Lab self-hosted runners for the past year. It hasn't been difficult to operate. Once deployed, it pretty much "just works". In my experience, the things that go wrong originate from the GitHub workflows themselves. We usually have to review workflow logs regardless of whether the workflow uses a self-hosted runner or not.
- Grimburger 3y ago
  >GitHub don't host runners for Linux × arm64 Is this not possible via QEMU like everyone has been doing for a very long time now? The real issue here seems to be large companies wanting their own syntactical turf everyone abides by that they can later profit from.
Aeolun 3y ago
> Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. We have extremely bursty CI loads. One push can kick off up to 60 different CI jobs. If a few people push at the same time, we easily have 300 jobs running in parallel. This happens the whole day long. It absolutely makes sense to self host this, since the cost to run this on Github runners would be prohibitive (Github runners are 8x the cost of the equivalent AWS instance I think). All our runners are ephemeral, so we only pay for them when they are actually running jobs. After all the jobs are over, the runners are immediately scaled to zero. I build this whole thing myself, so it's a bit sad to find someone had already built the whole thing before.
candiddevmike 3y ago
ARC is great for running GitHub Actions on Kubernetes: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I think Kubernetes is a better platform than EC2 for runners, it's faster and more integrated with your tooling (if you're using Kubernetes).
- uuddlrlrbas 3y ago
  Yeah we use this as well. It really is awesome and highly recommended and fairly straight forward to setup.

lukevp 3y ago

Cost isn’t the only vector. Someone may want to host the runners within a given VPC so they have access to private resources / PrivateLink etc. the job API is a lot narrower of an attack surface area than giving your auth info to GitHub that has enough power to do CI/CD infra changes and such. There’s also people who use GitHub Enterprise who still need to host a runner fleet, so they could benefit from this too.

Grimburger 3y ago
Sure, but does my point about fallible spot not still stand?
- quickthrower2 3y ago
  I agree usually “CI is running” means “human is sorta blocked waiting on this”* means the compute cost to run it reliably is probably negligible. Whether you this on the Github network or your own private cloud is then an orthogonal concern. Spot is silly although spot with immediate fallback to in demand might be a neat hack. * Dependabot and AI PRs might be exceptions.
stoobs 3y ago
This. We’re using the this at my day job for the requirement for the runner to access private resources such as internal api’s etc
raffraffraff 3y ago
There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI startup I worked at, self-hosting runners that used our own AMIs saved us a lot of time and egress costs, because so many CI steps begin by pulling some image, whether it be the base image or one you built at a previous step and pushed to your container registry. We also ran CI steps on some projects that required ARM or GPU. You also have the option of keeping a runner up during the busiest part of the day to benefit from cached images. We'd automatically rebuild the runner AMIs once a week and set runner lifetime to max 24 hours, ensuring freshness and ensuring that we didn't run into disk space issues.
- SOLAR_FIELDS 3y ago
  We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy
  - barefeg 3y ago
    Could you give more details/resources on the distributed in cluster buildkit cache?
    - baby_souffle 3y ago
      I would also like to know more
    - SOLAR_FIELDS 3y ago
      We use Dagger’s implementation. The basic approach is to have a buildkit engine run as a daemonset on the cluster, and clients specify the same docker socket that buildkit uses. The magic is in cache synchronization, eg only lazily pulling layers as the client requests them. This is scalable but obviously since caching is hard there are some complexities with efficient synchronization of cache layers and cache volumes. This is currently a long lived service that runs as a deployment alongside a bunch of ephemeral runners to manage the cache synchronization. There are several other different architectures that range from simpler to more complex. The architecture I recommend people start out with is a single long lived beefy buildkit instance that a bunch of runners share, since that is much much simpler to implement. It of course has the downside that you have to refresh/rebuild the cache if the instance ever goes down. For runs that need read/write locks on volumes (eg Gradle build cache) my recommendation after trial and error to rsync those to the runners and then rsync them back after the run completes so you don’t have a bunch of locks fighting each other for the same folder.
skwashd 3y ago
Add network egress controls and telemetry as two more benefits of self hosted runners. Denied network connections are a sign there could be a problem with a dependency.

tlarkworthy 3y ago

Absolutely, there are lots of workloads that do not run on actions and require custom hardware (GPU? Big boxes) or greater privileges to Linux API than actions provide, or just better caching.

dataking 3y ago
This. You even need to self-host if you want to test code that uses AVX512 outside of an emulator since the default runners do not support that. Same if you want to test aarch64-specific code paths on Linux, Windows, or macOS.

andrus 3y ago

> That said, do many see value is in this? GitHub don't host runners for Linux × arm64, so if you need this, you need to self-host. You can also run custom AMIs with pre-installed packages, which can speed up workflows that depend on those packages. > When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI… I'm on a small team who've been running the Philips Lab self-hosted runners for the past year. It hasn't been difficult to operate. Once deployed, it pretty much "just works". In my experience, the things that go wrong originate from the GitHub workflows themselves. We usually have to review workflow logs regardless of whether the workflow uses a self-hosted runner or not.

Grimburger 3y ago
>GitHub don't host runners for Linux × arm64 Is this not possible via QEMU like everyone has been doing for a very long time now? The real issue here seems to be large companies wanting their own syntactical turf everyone abides by that they can later profit from.

Aeolun 3y ago

> Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. We have extremely bursty CI loads. One push can kick off up to 60 different CI jobs. If a few people push at the same time, we easily have 300 jobs running in parallel. This happens the whole day long. It absolutely makes sense to self host this, since the cost to run this on Github runners would be prohibitive (Github runners are 8x the cost of the equivalent AWS instance I think). All our runners are ephemeral, so we only pay for them when they are actually running jobs. After all the jobs are over, the runners are immediately scaled to zero. I build this whole thing myself, so it's a bit sad to find someone had already built the whole thing before.

candiddevmike 3y ago

ARC is great for running GitHub Actions on Kubernetes: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I think Kubernetes is a better platform than EC2 for runners, it's faster and more integrated with your tooling (if you're using Kubernetes).

uuddlrlrbas 3y ago
Yeah we use this as well. It really is awesome and highly recommended and fairly straight forward to setup.