7 ms·

Terraform module for scalable GitHub action runners on AWS

120 points by ta988 3y ago

leipert 3y ago
This looks cool! I will have a look how it works. It’s funny because just the other day I thought about implementing the same for GitLab in order to learn more about terraform. (Note: I work for GitLab, but as a Frontend Engineer and usually not on CI topics)
senzilla 3y ago
I love how, in the last few years, open source infrastructure has become more of a thing. We're using this Philips Labs module at $dayjob. It's a great piece of work!
Grimburger 3y ago
Meta: Absolutely superb readme for an open-source project. Can anyone point to a better opensource readme today? That said, do many see value is in this? The current stance is you either self-host or accept paying github's runner rates out of laziness. In the end it's all very mild when it comes to extra costs. When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI and in the end finding "oh it was a re-provisioning of compute", so...let's run it again and hope that goes well, actually no...let's schedule it for 11pm Eastern which probably isn't that optimal. I can see it working for some, but horrendous for others. Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. Still, a marvellous readme. If only others in open-source would take note that this is how you do it.
- lukevp 3y ago
  Cost isn’t the only vector. Someone may want to host the runners within a given VPC so they have access to private resources / PrivateLink etc. the job API is a lot narrower of an attack surface area than giving your auth info to GitHub that has enough power to do CI/CD infra changes and such. There’s also people who use GitHub Enterprise who still need to host a runner fleet, so they could benefit from this too.
  - Grimburger 3y ago
    Sure, but does my point about fallible spot not still stand?
    - quickthrower2 3y ago
      I agree usually “CI is running” means “human is sorta blocked waiting on this”* means the compute cost to run it reliably is probably negligible. Whether you this on the Github network or your own private cloud is then an orthogonal concern. Spot is silly although spot with immediate fallback to in demand might be a neat hack. * Dependabot and AI PRs might be exceptions.
  - stoobs 3y ago
    This. We’re using the this at my day job for the requirement for the runner to access private resources such as internal api’s etc
  - raffraffraff 3y ago
    There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI startup I worked at, self-hosting runners that used our own AMIs saved us a lot of time and egress costs, because so many CI steps begin by pulling some image, whether it be the base image or one you built at a previous step and pushed to your container registry. We also ran CI steps on some projects that required ARM or GPU. You also have the option of keeping a runner up during the busiest part of the day to benefit from cached images. We'd automatically rebuild the runner AMIs once a week and set runner lifetime to max 24 hours, ensuring freshness and ensuring that we didn't run into disk space issues.
    - SOLAR_FIELDS 3y ago
      We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy
- tlarkworthy 3y ago
  Absolutely, there are lots of workloads that do not run on actions and require custom hardware (GPU? Big boxes) or greater privileges to Linux API than actions provide, or just better caching.
  - dataking 3y ago
    This. You even need to self-host if you want to test code that uses AVX512 outside of an emulator since the default runners do not support that. Same if you want to test aarch64-specific code paths on Linux, Windows, or macOS.
- andrus 3y ago
  > That said, do many see value is in this? GitHub don't host runners for Linux × arm64, so if you need this, you need to self-host. You can also run custom AMIs with pre-installed packages, which can speed up workflows that depend on those packages. > When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI… I'm on a small team who've been running the Philips Lab self-hosted runners for the past year. It hasn't been difficult to operate. Once deployed, it pretty much "just works". In my experience, the things that go wrong originate from the GitHub workflows themselves. We usually have to review workflow logs regardless of whether the workflow uses a self-hosted runner or not.
  - Grimburger 3y ago
    >GitHub don't host runners for Linux × arm64 Is this not possible via QEMU like everyone has been doing for a very long time now? The real issue here seems to be large companies wanting their own syntactical turf everyone abides by that they can later profit from.
- Aeolun 3y ago
  > Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. We have extremely bursty CI loads. One push can kick off up to 60 different CI jobs. If a few people push at the same time, we easily have 300 jobs running in parallel. This happens the whole day long. It absolutely makes sense to self host this, since the cost to run this on Github runners would be prohibitive (Github runners are 8x the cost of the equivalent AWS instance I think). All our runners are ephemeral, so we only pay for them when they are actually running jobs. After all the jobs are over, the runners are immediately scaled to zero. I build this whole thing myself, so it's a bit sad to find someone had already built the whole thing before.
- candiddevmike 3y ago
  ARC is great for running GitHub Actions on Kubernetes: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I think Kubernetes is a better platform than EC2 for runners, it's faster and more integrated with your tooling (if you're using Kubernetes).
  - uuddlrlrbas 3y ago
    Yeah we use this as well. It really is awesome and highly recommended and fairly straight forward to setup.
moltar 3y ago
There are also official Heml charts available for ARC: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I am in the process of setting it up on a cheap Hetzner box. If it works, would be a great deal! You can get a 64 GB RAM box for 35 EUR/mo at server auctions with unlimited traffic. I don't mention CPU or GPU, as typically this isn't a bottleneck for my projects. Plus, I can configure cache sharing via host-mounted dir. E.g. pnpm cache can be all in one place, and be locally available to pods via a mounted dir. Same for the Docker image cache. This would speed up CI runs and also reduce network traffic by a huge margin. GitHub Actions effectively has no local caching. There's an action for caching, but it uses a blob storage for cache artifacts. Which then gets network fetched, gzip'ed and gunzip'ed each time, and from my experience this has never been a gain for medium to large npm projects, as they have thousands of small .js files in node_modules, and thus takes a long time to compress and decompress. I think npm edge cache servers are already so optimized and fast, that in my experience almost always it's faster to install from npm directly. I even tested this on AWS, where the cache was stored in S3, in the same region as CodeBuild (CI), and direct installs from npm were still faster by about 30%. So other than adding more hardware resources, local caching is the only way to significantly speed up GH Actions, from my experience, and thus you must have your runner.
- ozgune 3y ago
  (Disclaimer: I'm a cofounder at Ubicloud.) Hey there, we offer Ubicloud Runners that are 10x cheaper than GitHub and bill them by the minute. You get a fresh VM with each job; and we use Hetzner as our underlying provider. If you're already setting up a Hetzner box, I'd love to get your input. Any thoughts or feedback for us? https://www.ubicloud.com/use-cases/github-actions https://www.ubicloud.com/use-cases/github-actions https://github.com/ubicloud/ubicloud/blob/main/routes/web/webhook/github.rb https://github.com/ubicloud/ubicloud/blob/main/routes/web/we... (our github actions integration is also openly available)
  - moltar 3y ago
    Does it address my main issue of cache sharing? If cache is still over the network, then it's not optimal, as it's the bottleneck for everything. E.g. our node_modules is 3 GB, and once you gzip and gunzip it takes minutes. With pnpm cache store we can reduce this to seconds.
gchamonlive 3y ago
The only frustrating part of this project is GitHub actions itself. It could be so much better (https://news.ycombinator.com/item?id=37612420 https://news.ycombinator.com/item?id=37612420). Part of the hassle is figuring out why on earth actions/checkout stopped working in your self-hosted infrastructure... Why you suddenly have what once were ephemeral SQL containers persisting and blocking specific ports in subsequent runs... And why that sweet python provisioning action isn't doing its job anymore... It all comes down to the instances. GitHub runners are insanely complex and also the best part of actions, along with community driven reusable workflows. They not only pack so much stuff without conflict, they also spin up really fast (forgot which virtualization infrastructure they use). So when you move to self-hosted you now have instances that take longer to be ready, increasing cold start times from pipelines, that will require that you install everything you need in it to run your jobs, and that, depending on how you set them, will require more thorough cleanup after each job. The scalability is really good though, and all the benefits already cited, like having access to private resources on your VPC, and delegating permissions to instance profile through IAM roles, make this project a godsend.
- suryao 3y ago
  We maintain a little action called Action-Debugger that let's you ssh into a running GitHub action workflow to help debug pesky issues. It has a few additional features when you use it with our runners but works very well by itself. https://docs.warpbuild.com/tools/action-debugger https://docs.warpbuild.com/tools/action-debugger https://github.com/WarpBuilds/action-debugger https://github.com/WarpBuilds/action-debugger
damianh 3y ago
If you are interested in having self-hosted auto scaling github actions runners but don't want to manage your own webhook handling infrastructure, https://cirun.io https://cirun.io is worth a look.
_joel 3y ago
I used this at the last gig. It works really well +1.
kgjkgk 3y ago
[flagged]
romanhn 3y ago
I was using it for my project because the workflows required access to VPC resources (the database, etc). It worked... ok. There would be times when the runner didn't start, or parallel runs would not work, felt a bit random. Possibly was my config, there were quite a few knobs and dials. Eventually I stumbled on the idea of running the VPC-requiring commands from an AWS CodeBuild script, and invoking it from a workflow executed on a GitHub-owned runner. Works beautifully and I was able to remove a ton of complexity from my infra that this Terraform module adds.
eez0 3y ago
Been using this for 2+ years. My experience is that it works until it does not, and then you are down on a rabbit hole trying to figure out why. The only reason we keep using it is because we have other priorities, but once we have more spare time, this module is going to hell, it's not reliable at all.
- ta988 3y ago
  Interesting, the only issues we have had were when AWS didn't have enough capacity. And you can see that in the logs of the scale up lambda.
- xjia 3y ago
  I had a similar experience with ARC (actions-runner-controller). One of the machines in the fleet failed to sync its clock via NTP. Once a job X got scheduled to it, the runner pod failed authentication due to incorrect clock time, and then the whole ARC system started to behave incorrectly: job X was stuck without runners, until another workflow job Y was created, and then X got run but Y became stuck. There were also other wierd behaviors like this so I eventually rebuilt everything based on VMs and stopped using ARC. Using VMs also allowed me to support the use of the official runner images [0], which is good for compatibility. I feel more people would benefit from managed "self-hosted" runners, so I started DimeRun [1] to provide cheaper GHA runners for people who don't have the time/willingness to troubleshoot low-level infra issues. [0]: https://github.com/actions/runner-images https://github.com/actions/runner-images [1]: https://dime.run https://dime.run
  - eez0 3y ago
    Exactly what you're are describing is what I explained to my colleagues as "stealing runners" :) If something fails and you don't have idle runners (hence wasting unnecessary resources), things start to snowball.
    - ta988 3y ago
      The module posted here as a way to avoid that where only runners requested by a job can be used by it (or idle runners if you have those)
Halan 3y ago
Been using Scalesets + Karpenter and the only problem is the occasional GH availability taking a dip
erulabs 3y ago
With Karpenter becoming an official EKS add on, I’m really hoping the new “scalesets” gets the same treatment and then we’re somewhat close to a an out of the box ci on EKS experience. Right now I’m using Karpenter, ARC, EFS and buildkit and it’s great, but it was also like a month of setup and is nontrivially complex.
SpookyChoice 3y ago
We evaluated this but shied away from the complexity... In the end we build a slim version ourself containing 1. A codebuild job running a container image which starts a runner with custom labels 2. A lambda to receive the webhook from github and run the codebuild job on demand.
- Shorn 3y ago
  I haven't implemented it myself - I just assumed this would be the way to folks would do this. What would implementing all the infra give you in terms of benefits over codebuild? The readme doesn't mention anything about codebuild that I can see.
  - kichik 3y ago
    CodeBuild can be a bit slow to start sometimes. It also doesn't support Docker on Windows builds and it's limited to 8 hours per run. Shameless plug: I wrote a CDK construct that lets you create runners on demand in response to a GitHub webhook (so basically what was suggested here only with handling of more corner cases that came up). It lets you start runners in CodeBuild, Fargate, ECS, EC2, or Lambda so you can pick whichever works best for you. There is a table in the readme that shows the difference between them and why you might want to choose one over the other. https://github.com/CloudSnorkel/cdk-github-runners https://github.com/CloudSnorkel/cdk-github-runners
igorzij 3y ago
This is super cool. Private runners are often so much cheaper, especially in large teams. Comes with a "fat tail" of reliability risk though: taking maintenance of runners means that that say 0.01% of builds would be failing for unknown reasons and it'd take significant effort to fix those. But then again that'd probably be only relevant in large organisations which likely have bigger bottlenecks in their devops practice than that; and if that's just about cost then a no-brainer.
rubiconjosh 3y ago
I use this in Gov Cloud with GitHub Enterprise Server. It is working great as a POC. It will be in production with a lot of usage soon so I can evaluate how it scales.

120 points by ta988 3y ago

This looks cool! I will have a look how it works. It’s funny because just the other day I thought about implementing the same for GitLab in order to learn more about terraform. (Note: I work for GitLab, but as a Frontend Engineer and usually not on CI topics)

senzilla 3y ago

I love how, in the last few years, open source infrastructure has become more of a thing. We're using this Philips Labs module at $dayjob. It's a great piece of work!

Grimburger 3y ago

Meta: Absolutely superb readme for an open-source project. Can anyone point to a better opensource readme today? That said, do many see value is in this? The current stance is you either self-host or accept paying github's runner rates out of laziness. In the end it's all very mild when it comes to extra costs. When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI and in the end finding "oh it was a re-provisioning of compute", so...let's run it again and hope that goes well, actually no...let's schedule it for 11pm Eastern which probably isn't that optimal. I can see it working for some, but horrendous for others. Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. Still, a marvellous readme. If only others in open-source would take note that this is how you do it.

lukevp 3y ago
Cost isn’t the only vector. Someone may want to host the runners within a given VPC so they have access to private resources / PrivateLink etc. the job API is a lot narrower of an attack surface area than giving your auth info to GitHub that has enough power to do CI/CD infra changes and such. There’s also people who use GitHub Enterprise who still need to host a runner fleet, so they could benefit from this too.
- Grimburger 3y ago
  Sure, but does my point about fallible spot not still stand?
  - quickthrower2 3y ago
    I agree usually “CI is running” means “human is sorta blocked waiting on this”* means the compute cost to run it reliably is probably negligible. Whether you this on the Github network or your own private cloud is then an orthogonal concern. Spot is silly although spot with immediate fallback to in demand might be a neat hack. * Dependabot and AI PRs might be exceptions.
- stoobs 3y ago
  This. We’re using the this at my day job for the requirement for the runner to access private resources such as internal api’s etc
- raffraffraff 3y ago
  There's also the ability to pre-provision your runners with all of your base container images. Base images don't change that often and can be large. At an AI startup I worked at, self-hosting runners that used our own AMIs saved us a lot of time and egress costs, because so many CI steps begin by pulling some image, whether it be the base image or one you built at a previous step and pushed to your container registry. We also ran CI steps on some projects that required ARM or GPU. You also have the option of keeping a runner up during the busiest part of the day to benefit from cached images. We'd automatically rebuild the runner AMIs once a week and set runner lifetime to max 24 hours, ensuring freshness and ensuring that we didn't run into disk space issues.
  - SOLAR_FIELDS 3y ago
    We were baking AMI’s like this for a little while but our current setup is quite even fancier - you can do everything in containers and then a distributed in cluster buildkit cache and registry be hosted and backed by S3, and not only do you get a remote reusable cache that syncs extremely fast but you no longer have to rely on external artifact registries being flaky, which if you pull from Dockerhub frequently know this is an actual significant problem. We actually looked at using this project from OP at the beginning of our ci journey but eventually decided on actions-runner-controller due to its higher level of robustness and configurabiltiy
tlarkworthy 3y ago
Absolutely, there are lots of workloads that do not run on actions and require custom hardware (GPU? Big boxes) or greater privileges to Linux API than actions provide, or just better caching.
- dataking 3y ago
  This. You even need to self-host if you want to test code that uses AVX512 outside of an emulator since the default runners do not support that. Same if you want to test aarch64-specific code paths on Linux, Windows, or macOS.
andrus 3y ago
> That said, do many see value is in this? GitHub don't host runners for Linux × arm64, so if you need this, you need to self-host. You can also run custom AMIs with pre-installed packages, which can speed up workflows that depend on those packages. > When things go wrong, you need people going through logs working out that there's nothing actually wrong with CI… I'm on a small team who've been running the Philips Lab self-hosted runners for the past year. It hasn't been difficult to operate. Once deployed, it pretty much "just works". In my experience, the things that go wrong originate from the GitHub workflows themselves. We usually have to review workflow logs regardless of whether the workflow uses a self-hosted runner or not.
- Grimburger 3y ago
  >GitHub don't host runners for Linux × arm64 Is this not possible via QEMU like everyone has been doing for a very long time now? The real issue here seems to be large companies wanting their own syntactical turf everyone abides by that they can later profit from.
Aeolun 3y ago
> Maybe if you have burstable CI loads it's worth doing and re-rolling on failure. We have extremely bursty CI loads. One push can kick off up to 60 different CI jobs. If a few people push at the same time, we easily have 300 jobs running in parallel. This happens the whole day long. It absolutely makes sense to self host this, since the cost to run this on Github runners would be prohibitive (Github runners are 8x the cost of the equivalent AWS instance I think). All our runners are ephemeral, so we only pay for them when they are actually running jobs. After all the jobs are over, the runners are immediately scaled to zero. I build this whole thing myself, so it's a bit sad to find someone had already built the whole thing before.
candiddevmike 3y ago
ARC is great for running GitHub Actions on Kubernetes: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I think Kubernetes is a better platform than EC2 for runners, it's faster and more integrated with your tooling (if you're using Kubernetes).
- uuddlrlrbas 3y ago
  Yeah we use this as well. It really is awesome and highly recommended and fairly straight forward to setup.

moltar 3y ago

There are also official Heml charts available for ARC: https://github.com/actions/actions-runner-controller https://github.com/actions/actions-runner-controller I am in the process of setting it up on a cheap Hetzner box. If it works, would be a great deal! You can get a 64 GB RAM box for 35 EUR/mo at server auctions with unlimited traffic. I don't mention CPU or GPU, as typically this isn't a bottleneck for my projects. Plus, I can configure cache sharing via host-mounted dir. E.g. pnpm cache can be all in one place, and be locally available to pods via a mounted dir. Same for the Docker image cache. This would speed up CI runs and also reduce network traffic by a huge margin. GitHub Actions effectively has no local caching. There's an action for caching, but it uses a blob storage for cache artifacts. Which then gets network fetched, gzip'ed and gunzip'ed each time, and from my experience this has never been a gain for medium to large npm projects, as they have thousands of small .js files in node_modules, and thus takes a long time to compress and decompress. I think npm edge cache servers are already so optimized and fast, that in my experience almost always it's faster to install from npm directly. I even tested this on AWS, where the cache was stored in S3, in the same region as CodeBuild (CI), and direct installs from npm were still faster by about 30%. So other than adding more hardware resources, local caching is the only way to significantly speed up GH Actions, from my experience, and thus you must have your runner.

ozgune 3y ago
(Disclaimer: I'm a cofounder at Ubicloud.) Hey there, we offer Ubicloud Runners that are 10x cheaper than GitHub and bill them by the minute. You get a fresh VM with each job; and we use Hetzner as our underlying provider. If you're already setting up a Hetzner box, I'd love to get your input. Any thoughts or feedback for us? https://www.ubicloud.com/use-cases/github-actions https://www.ubicloud.com/use-cases/github-actions https://github.com/ubicloud/ubicloud/blob/main/routes/web/webhook/github.rb https://github.com/ubicloud/ubicloud/blob/main/routes/web/we... (our github actions integration is also openly available)
- moltar 3y ago
  Does it address my main issue of cache sharing? If cache is still over the network, then it's not optimal, as it's the bottleneck for everything. E.g. our node_modules is 3 GB, and once you gzip and gunzip it takes minutes. With pnpm cache store we can reduce this to seconds.

gchamonlive 3y ago

The only frustrating part of this project is GitHub actions itself. It could be so much better (https://news.ycombinator.com/item?id=37612420 https://news.ycombinator.com/item?id=37612420). Part of the hassle is figuring out why on earth actions/checkout stopped working in your self-hosted infrastructure... Why you suddenly have what once were ephemeral SQL containers persisting and blocking specific ports in subsequent runs... And why that sweet python provisioning action isn't doing its job anymore... It all comes down to the instances. GitHub runners are insanely complex and also the best part of actions, along with community driven reusable workflows. They not only pack so much stuff without conflict, they also spin up really fast (forgot which virtualization infrastructure they use). So when you move to self-hosted you now have instances that take longer to be ready, increasing cold start times from pipelines, that will require that you install everything you need in it to run your jobs, and that, depending on how you set them, will require more thorough cleanup after each job. The scalability is really good though, and all the benefits already cited, like having access to private resources on your VPC, and delegating permissions to instance profile through IAM roles, make this project a godsend.

suryao 3y ago
We maintain a little action called Action-Debugger that let's you ssh into a running GitHub action workflow to help debug pesky issues. It has a few additional features when you use it with our runners but works very well by itself. https://docs.warpbuild.com/tools/action-debugger https://docs.warpbuild.com/tools/action-debugger https://github.com/WarpBuilds/action-debugger https://github.com/WarpBuilds/action-debugger

damianh 3y ago

If you are interested in having self-hosted auto scaling github actions runners but don't want to manage your own webhook handling infrastructure, https://cirun.io https://cirun.io is worth a look.

_joel 3y ago

I used this at the last gig. It works really well +1.

kgjkgk 3y ago

[flagged]

romanhn 3y ago

I was using it for my project because the workflows required access to VPC resources (the database, etc). It worked... ok. There would be times when the runner didn't start, or parallel runs would not work, felt a bit random. Possibly was my config, there were quite a few knobs and dials. Eventually I stumbled on the idea of running the VPC-requiring commands from an AWS CodeBuild script, and invoking it from a workflow executed on a GitHub-owned runner. Works beautifully and I was able to remove a ton of complexity from my infra that this Terraform module adds.

eez0 3y ago

Been using this for 2+ years. My experience is that it works until it does not, and then you are down on a rabbit hole trying to figure out why. The only reason we keep using it is because we have other priorities, but once we have more spare time, this module is going to hell, it's not reliable at all.

ta988 3y ago
Interesting, the only issues we have had were when AWS didn't have enough capacity. And you can see that in the logs of the scale up lambda.
xjia 3y ago
I had a similar experience with ARC (actions-runner-controller). One of the machines in the fleet failed to sync its clock via NTP. Once a job X got scheduled to it, the runner pod failed authentication due to incorrect clock time, and then the whole ARC system started to behave incorrectly: job X was stuck without runners, until another workflow job Y was created, and then X got run but Y became stuck. There were also other wierd behaviors like this so I eventually rebuilt everything based on VMs and stopped using ARC. Using VMs also allowed me to support the use of the official runner images [0], which is good for compatibility. I feel more people would benefit from managed "self-hosted" runners, so I started DimeRun [1] to provide cheaper GHA runners for people who don't have the time/willingness to troubleshoot low-level infra issues. [0]: https://github.com/actions/runner-images https://github.com/actions/runner-images [1]: https://dime.run https://dime.run
- eez0 3y ago
  Exactly what you're are describing is what I explained to my colleagues as "stealing runners" :) If something fails and you don't have idle runners (hence wasting unnecessary resources), things start to snowball.
  - ta988 3y ago
    The module posted here as a way to avoid that where only runners requested by a job can be used by it (or idle runners if you have those)

Halan 3y ago

Been using Scalesets + Karpenter and the only problem is the occasional GH availability taking a dip

erulabs 3y ago

With Karpenter becoming an official EKS add on, I’m really hoping the new “scalesets” gets the same treatment and then we’re somewhat close to a an out of the box ci on EKS experience. Right now I’m using Karpenter, ARC, EFS and buildkit and it’s great, but it was also like a month of setup and is nontrivially complex.

SpookyChoice 3y ago

We evaluated this but shied away from the complexity... In the end we build a slim version ourself containing 1. A codebuild job running a container image which starts a runner with custom labels 2. A lambda to receive the webhook from github and run the codebuild job on demand.

Shorn 3y ago
I haven't implemented it myself - I just assumed this would be the way to folks would do this. What would implementing all the infra give you in terms of benefits over codebuild? The readme doesn't mention anything about codebuild that I can see.
- kichik 3y ago
  CodeBuild can be a bit slow to start sometimes. It also doesn't support Docker on Windows builds and it's limited to 8 hours per run. Shameless plug: I wrote a CDK construct that lets you create runners on demand in response to a GitHub webhook (so basically what was suggested here only with handling of more corner cases that came up). It lets you start runners in CodeBuild, Fargate, ECS, EC2, or Lambda so you can pick whichever works best for you. There is a table in the readme that shows the difference between them and why you might want to choose one over the other. https://github.com/CloudSnorkel/cdk-github-runners https://github.com/CloudSnorkel/cdk-github-runners

igorzij 3y ago

This is super cool. Private runners are often so much cheaper, especially in large teams. Comes with a "fat tail" of reliability risk though: taking maintenance of runners means that that say 0.01% of builds would be failing for unknown reasons and it'd take significant effort to fix those. But then again that'd probably be only relevant in large organisations which likely have bigger bottlenecks in their devops practice than that; and if that's just about cost then a no-brainer.

rubiconjosh 3y ago

I use this in Gov Cloud with GitHub Enterprise Server. It is working great as a POC. It will be in production with a lot of usage soon so I can evaluate how it scales.