7 ms·

Making GitHub CI workflow 3x faster

54 points by quyleanh 6y ago

judge2020 6y ago
I realize this isn't about GitHub Actions, but I wish they would get on with different machine sizes for GitHub's own runners[0]. Many huge builds can end up taking 2x-3x the time on the slow 2 core machine they give you compared to a local high-end CPU. 0: https://github.com/github/roadmap/issues/95 https://github.com/github/roadmap/issues/95
- 0xcoffee 6y ago
  I feel like this is a common problem with almost all CI/CD offerings. At my company we ended up just installing a local build agent on a powerful pc, and using that for the 'frequent, but not super important' builds, such as building/test for a PR. The builds for acceptance/prod we still do on remote CI/CD, but those happen much more rarely. Also the advantage of a local CI/CD is that it is much easier to setup caching for things like Node Modules, NuGet etc. We use Azure Devops which has a pretty good local CI/CD story, installing an agent was quiet easy for us.
- eganjs 6y ago
  You should check out this [0] GitHub action workflow. It uses docker machine to create a high resourced runner on demand in the cloud and then runs the subsequent job on it. The workflow is intended for machine learning but I'm sure it could be adapted. I must disclose that I haven't actually used it myself yet (because I haven't felt the need) but it does look very promising. 0: https://github.com/iterative/cml_cloud_case/blob/master/.github/workflows/cml.yaml https://github.com/iterative/cml_cloud_case/blob/master/.git...
- coolsunglasses 6y ago
  With GitLab CI you can task any machine you want to your builds. I've slaved my workstations to my CI builds because they're ~2-5x faster than the AWS hardware most companies use.
  - judge2020 6y ago
    Same for GitHub[0], but i'm specifically talking about the machines they give you. 0: https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners https://docs.github.com/en/free-pro-team@latest/actions/host...
- larrybud 6y ago
  This is one of the key use-cases for self-hosted runners. https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners https://docs.github.com/en/free-pro-team@latest/actions/host...
yellow_lead 6y ago
This doesn't seem like all that great of a solution to me. The strict requirement to have a passing build is replaced with the fear that you'll be the one that broke the build. So you'll end up running these tests before you merge anyways, right? What has changed?
- anuila 6y ago
  I think the difference is that they expect to not break tests most of the time. So most of the time this just saves time. It’s like when you merge a documentation PR while the tests are still running: You’re pretty sure that the tests won’t break. With this solution however you’ll still be pinged if they do break, later.
  - malkia 6y ago
    I mean the change can be reverted, if it's not fixed, automatically or not.
  - stkdump 6y ago
    If one developer doesn't break it 98% of the time, then for 200 devs you have a chance of .98^200=2% of not breaking it. And for one person trying to quickly get the fix ready while he might have an urgent family situation you have 199 devs who can't get their stuff in, of which at least 20 are under pressure to deploy their fixes quickly themself. It sounds like creating a hard problem for individuals for solving a mediocre problem for the team.
    - conradludgate 6y ago
      I dont see why this would block other devs merging in their work. And what kind of teams do you work in where you can't politely ask for someone's help when you have to deal with a personal issue?
      - stkdump 6y ago
        > I dont see why this would block other devs merging in their work. Read the article. It says: > If the CI job remains broken for more than 72 hours, all deployments to GitHub.com are halted Edit: it even creates the perverse incentive of trying to get your own stuff in quickly before the 72h window closes that s/o else caused.
monkin 6y ago
This is such a deep dive I almost drown in information...
stkdump 6y ago
Stopping the world after 72h of CI fails seems like a dangerous decision, especially if you have a large team working on a monorepo. Note that fails can easily 'pile up' and the 72h are not necessarily all caused by the same change. I would probably get paranoid, feeling the constant pressure not to be part of the cause for such an occourance.
- rendall 6y ago
  Yes, I didn't understand this. I've even been a part of workflows that halt all subsequent PRs until the CI tests pass, and I still don't understand the situation this is designed to prevent. Why not simply disallow the PR to merge unless CI passes? Why stop all integrations? There must be something I'm missing. Edit: I think I get it. Normally PRs to Github.com non-enterprise go through as normal. 45 minutes later, after that merge, the 2 expensive Enterprise CI tests complete and that's when the developer gets the 72 hour timer. The dev has the option of reverting their change and trying again.
soulchild37 6y ago
TL;DR: Move fast and break things
IshKebab 6y ago
It's not very clear but it seems to suggest they deploy to production before all the tests pass? That seems silly. Why not just wait until they've passed? Or does "deploy" really mean "merge"?
- jsmith45 6y ago
  Why should deploying to production be be prevented by needing to wait for tests that do not apply to that environment, but only to a separate product (GitHub Enterprise Server)? GitHub takes continuous deployment very seriously, with dozens of deployments every day. My understanding is that they try to avoid having code sitting around only deployed to an internal environment any more than truly necessary. They want to be able to fix code issues discovered in production within a shockingly short time frame if the issue is not complex.
rurban 6y ago
The title misses a Our

54 points by quyleanh 6y ago

judge2020 6y ago

I realize this isn't about GitHub Actions, but I wish they would get on with different machine sizes for GitHub's own runners[0]. Many huge builds can end up taking 2x-3x the time on the slow 2 core machine they give you compared to a local high-end CPU. 0: https://github.com/github/roadmap/issues/95 https://github.com/github/roadmap/issues/95

0xcoffee 6y ago
I feel like this is a common problem with almost all CI/CD offerings. At my company we ended up just installing a local build agent on a powerful pc, and using that for the 'frequent, but not super important' builds, such as building/test for a PR. The builds for acceptance/prod we still do on remote CI/CD, but those happen much more rarely. Also the advantage of a local CI/CD is that it is much easier to setup caching for things like Node Modules, NuGet etc. We use Azure Devops which has a pretty good local CI/CD story, installing an agent was quiet easy for us.
eganjs 6y ago
You should check out this [0] GitHub action workflow. It uses docker machine to create a high resourced runner on demand in the cloud and then runs the subsequent job on it. The workflow is intended for machine learning but I'm sure it could be adapted. I must disclose that I haven't actually used it myself yet (because I haven't felt the need) but it does look very promising. 0: https://github.com/iterative/cml_cloud_case/blob/master/.github/workflows/cml.yaml https://github.com/iterative/cml_cloud_case/blob/master/.git...
coolsunglasses 6y ago
With GitLab CI you can task any machine you want to your builds. I've slaved my workstations to my CI builds because they're ~2-5x faster than the AWS hardware most companies use.
- judge2020 6y ago
  Same for GitHub[0], but i'm specifically talking about the machines they give you. 0: https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners https://docs.github.com/en/free-pro-team@latest/actions/host...
larrybud 6y ago
This is one of the key use-cases for self-hosted runners. https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners https://docs.github.com/en/free-pro-team@latest/actions/host...

yellow_lead 6y ago

This doesn't seem like all that great of a solution to me. The strict requirement to have a passing build is replaced with the fear that you'll be the one that broke the build. So you'll end up running these tests before you merge anyways, right? What has changed?

anuila 6y ago
I think the difference is that they expect to not break tests most of the time. So most of the time this just saves time. It’s like when you merge a documentation PR while the tests are still running: You’re pretty sure that the tests won’t break. With this solution however you’ll still be pinged if they do break, later.
- malkia 6y ago
  I mean the change can be reverted, if it's not fixed, automatically or not.
- stkdump 6y ago
  If one developer doesn't break it 98% of the time, then for 200 devs you have a chance of .98^200=2% of not breaking it. And for one person trying to quickly get the fix ready while he might have an urgent family situation you have 199 devs who can't get their stuff in, of which at least 20 are under pressure to deploy their fixes quickly themself. It sounds like creating a hard problem for individuals for solving a mediocre problem for the team.
  - conradludgate 6y ago
    I dont see why this would block other devs merging in their work. And what kind of teams do you work in where you can't politely ask for someone's help when you have to deal with a personal issue?
    - stkdump 6y ago
      > I dont see why this would block other devs merging in their work. Read the article. It says: > If the CI job remains broken for more than 72 hours, all deployments to GitHub.com are halted Edit: it even creates the perverse incentive of trying to get your own stuff in quickly before the 72h window closes that s/o else caused.

monkin 6y ago

This is such a deep dive I almost drown in information...

stkdump 6y ago

Stopping the world after 72h of CI fails seems like a dangerous decision, especially if you have a large team working on a monorepo. Note that fails can easily 'pile up' and the 72h are not necessarily all caused by the same change. I would probably get paranoid, feeling the constant pressure not to be part of the cause for such an occourance.

rendall 6y ago
Yes, I didn't understand this. I've even been a part of workflows that halt all subsequent PRs until the CI tests pass, and I still don't understand the situation this is designed to prevent. Why not simply disallow the PR to merge unless CI passes? Why stop all integrations? There must be something I'm missing. Edit: I think I get it. Normally PRs to Github.com non-enterprise go through as normal. 45 minutes later, after that merge, the 2 expensive Enterprise CI tests complete and that's when the developer gets the 72 hour timer. The dev has the option of reverting their change and trying again.

soulchild37 6y ago

TL;DR: Move fast and break things

IshKebab 6y ago

It's not very clear but it seems to suggest they deploy to production before all the tests pass? That seems silly. Why not just wait until they've passed? Or does "deploy" really mean "merge"?

jsmith45 6y ago
Why should deploying to production be be prevented by needing to wait for tests that do not apply to that environment, but only to a separate product (GitHub Enterprise Server)? GitHub takes continuous deployment very seriously, with dozens of deployments every day. My understanding is that they try to avoid having code sitting around only deployed to an internal environment any more than truly necessary. They want to be able to fix code issues discovered in production within a shockingly short time frame if the issue is not complex.

rurban 6y ago

The title misses a Our