11 ms·

GitHub Actions limitations and gotchas

217 points by 147 5y ago

BugsJustFindMe 5y ago
Is it possible yet to (without hoops) have an action workflow that runs on pull requests to <default_branch> without having to name that branch explicitly so that you can have a write-once-run-everywhere-forever process across repos that might change their default branch name? Last I saw there was a template parameter, but those get populated at creation not at runtime.
bob1029 5y ago
The only thing we've ever trusted GH actions with is enforcing check builds pass before a PR is allowed to merge. Everything else is managed via a custom tool we use for packaging & deploying our product. Even our simple "run this 1 build command and ensure exit code == 0" action seems to have a semi-weekly issue like stuck "waiting for status" and other unexplained failures throughout. We don't want to put any more eggs into that particular basket right now.
judge2020 5y ago
The cache action section focuses on GHES which is a shame, since the GitHub.com cache action is also pretty bad. - can't manually delete cache (although they're considering changing that[0]) - only saves cache at the end of a workflow, and only if the workflow succeeds. This could be solved with CircleCI's approach of having a save-cache step and a restore-cache step. - Cache is super slow for self-hosted runners, so it makes more sense to have a local cache instead of using the action - only 5gb of storage size. This was supposed to be increased via billable cache storage[1], but it's been on the backburner since July 2020 - In addition to the above, you can't use a different storage backend on the official action (which would allow storing over 5GB of cache via your own storage). The best workaround is to use a user-provided action which utilizes the s3 api[2]. 0: https://github.com/actions/cache/issues/632 https://github.com/actions/cache/issues/632 1: https://github.com/github/roadmap/issues/66 https://github.com/github/roadmap/issues/66 2: https://github.com/actions/cache/issues/354#issuecomment-854762521 https://github.com/actions/cache/issues/354#issuecomment-854... 2>
- trulyme 5y ago
  - can't cache on tags [0] Caching is a joke. But then again, comparing GHA to GitLab CI is like comparing a pogo stick to a bicycle. Sure, both can be used to get there, but one ride is a bit... bumpy. (I use both and I am not affiliated to either.) [0] https://github.com/actions/cache/issues/556 https://github.com/actions/cache/issues/556
- codys 5y ago
  I'm heavily using cache on self hosted runners that are running in ec2 machines on aws. It's my experience that the cache isn't "super slow". Because of a bug (in my action) I was pushing/pulling gigabytes of data into the cache. On large files I was seeing ~125MB/s (that's Byte) download speeds. I haven't used the GitHub hosted runners much so I'm not sure how it compares though.
  - judge2020 5y ago
    The hosted runners seem to be very varied in performance. Just today an actions/checkout action on GitHub's hosted Linux box went down to 1 megabit per second and was going to take 30+ minutes to clone a 3gb repository. Re-running the workflow an hour later had it running fine.
lmeyerov 5y ago
We've been loving GHA: CI/CD-as-code, pull requests capture deploy history (CI, stage/restage, deploy), and labeling a PR with 'release' is enough to generate our on-prem + multi-cloud artifacts. Our main gotchas are roughly: - GH-hosted runners have too little RAM/HD for big docker software. They push you to self-hosted runners for that, which is fine in theory, but GHA/Azure doesn't actually support serverless runners, so that falls flat in practice. We don't want to be turning machines on/off, that's GHA's job. We experimented with GHA -> Packer -> Azure for serverless, but it was slow and Packer frequently leaves zombie machines, so we went back to tweaking the low-RAM runners provided by our enterprise plan. - Security: We want contactors etc. to be able to run limited GHA CI jobs and use that quota, but not higher-trust GHA CD ones. This is tricky at a configuration level. Ex: It seems like we'd need to funny things like main repo for CI w/ CI secrets, and a separate repo for CD w/ CD secrets, and only give untrusted folks access to the CD-cred repo. We've thought of other possibilities as well, but in general, it's frightening. - Big Docker images: We do spend more time than I'd like messing with optimizing Docker caching as GPU containers are embarrassingly big (we use dockerhub vs github's due to sizes/pricing/etc), think both multi-stage containers + multi-step jobs (monorepo/microservices). I think they're in a good position to speed that up! I'm optimistic about these, but tricky to align with MS/GH PM personal team priorities :)
- yebyen 5y ago
  I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images and pushing out releases, and CD is responsible for production. There is no crossover between these roles, and this makes it very secure. You use a policy CRD called ImagePolicy to declare what versions should be matched and deployed automatically. You can have them deployed directly to production, or if it makes you more comfortable with your contractors, to an "auto-PR" branch which simply queues them up for a release engineer with the required juice to approve and merge the changes to prod. There is no deploy job in CI, since the deployment in prod is whatever happens to be on the main branch in the config repo. This is a protected branch which only releng can merge changes into, usually only when the release checks are satisfied. Flux (which scans any image repos) updates those manifests when there is a release, either directly or via pull request. This strategy I think would be able to address your Security concerns regarding contractors, by reducing the responsibility of your CI system to only CI, and not to handle CD anymore. This is the subject of my talk at KubeCon[1], although you might not be able to get that from the title, (I'm presenting this topic for a Jenkins audience, but the focus is on Flux and how Flux works) so I'm hoping it should be applicable to a broader audience, certainly inclusive of GitHub users :) [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-with-declarative-everything-kingdon-barrett-weaveworks?iframe=no https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-wi...
m_a_g 5y ago
One extremely painful "gotcha" we encountered was trying to push code to a protected branch from inside a workflow. With the default GITHUB_TOKEN, you can't push to protected branches. If you decide to use personal access tokens, you can push to protected branches, BUT that will trigger other workflows. That can cause an infinite loop of workflows. We still couldn't figure out how to push code to a protected branch without triggering the same/other workflows.
- simonw 5y ago
  One trick that might work is to include a token such as 'DO_NOT_TRIGGER_ACTIONS' in your automated commit message, then have your other action skip any commits that include that token using this in the workflow: if: "!contains(github.event.head_commit.message, 'DO_NOT_TRIGGER_ACTIONS')"
  - m_a_g 5y ago
    good idea, thanks!
- chad_oliver 5y ago
  > We still couldn't figure out how to push code to a protected branch without triggering the same/other workflows. If you include `[skip ci]` anywhere in the commit message, that commit won't trigger any github actions. This is a built-in behavior; you don't have to manually check for that string in any downstream actions. I spent a long time trying to work around the same constraint before my colleague pointed out that `[skip ci]` was a thing.
- dcherman 5y ago
  You can also use a deploy key. That's how I do it while avoiding adding a PAT to a shared repo
oefrha 5y ago
Also, scheduled jobs are a joke, they are routinely one to several hours late. I have to use the API to manually trigger jobs (workflow_dispatch) whenever I need scheduling that is remotely time sensitive.
- 147 5y ago
  I've heard of issues with scheduled jobs on Github.com but we didn't have any issues so far on GHES.
- chrisfosterelli 5y ago
  This was the first thing that came to mind for me too. I don't need it to be exact, sure, but the scheduled jobs are more than an hour off more often than not. Sometimes several hours as you mentioned. Frequently they would also just not run at all! It was so bad I ended up scheduling my cron jobs to run 4 - 5 hours early and just sit there idling until the actual execution time came around haha. Ultimately not running at all started to be a real pain point, I gave up and paid $5 for a monthly box to sit there and run crond to save on the headache.
- seanhandley 5y ago
  Yes their scheduler is woeful. I have a few suspicions that it's intentionally done this way for balancing demand spikes and that jobs scheduled for the top of hour are worst affected.
Octabrain 5y ago
In the company I am working we’ve decided to use Github actions for our CI pipelines instead of deploying any “on prem” solution. I’ve worked a lot in the past with Jenkins and Travis and I also played a bit at home with Github actions. Now that I am using it in real world scenarios I have to say that I am a bit disappointed. In my opinion, as soon as you try to do something a bit complex you end up having to implement some nasty hack. Also I found that the Github Actions marketplace is a bit of a sink. You have to spend a good amount of time browsing it for finding something decent that hasn’t been discontinued, or is a pointless fork or at least is actively maintained. This happens even for basic functionalities. I known is a fairly recent platform but I was expecting much more compared to what other services offer.
- takeda 5y ago
  This is why I like GitLab much more. They implemented a proper CI. I think what GitHub was going for is to create some kind of ecosystem for proprietary 3rd party applications and use that as a revenue. That approach only crippled the whole functionality.
  - jaxn 5y ago
    Maybe it's just me, but I refuse to use third party actions in our build and deployment steps. It seems like such an obvious and avoidable attack surface to me, and one where an attached can have root on our production instances and at least some access to all developer machines.
    - ameliaquining 5y ago
      You can solve that by pinning commit hashes, so that nobody can change the code of the actions you use without your consent. You can then use Dependabot to automatically get PRs to update to the latest version of each action when it comes out. You still get the chance to review each PR before it goes in. I wish GitHub would implement a security setting requiring repositories to do this within an organization.
- ak217 5y ago
cebert 5y ago
GitHub Actions are a fantastic experience for serverless applications. I am working on a serverless project where we use GitHub Actions exclusively for CI/CD as well as running automated tests. We rely heavily on Lambda, S3, and DynamoDB. Our client app is static JS files we serve over Cloud Font. GitHub Actions make our piplines accessible to any developer on the team. Since we only pay for what we use with our serverless infrastructure, we can even deploy each pull request to the cloud rather inexpensively and leverage GitHub's environments to help manage the cleanup for us. This allows our team members to review and test changes in their browser before we pull them into our development branch. We additionally can run Playwright E2E tests to verify that none of our critical user workflow scenarios have broken resulting from the PR changes. I love this development experience and would have a hard time going back to anything else.
- shoo 5y ago
  thank you for sharing -- as i was reading through i tried to understand what about the workflow was specific to github actions vs other CI automation. > GitHub Actions make our pipelines accessible to any developer on the team Do you reckon this accessibility is a combination of (i) storing the pipeline definitions in the application's source repo, where application developers can find them easily, not hidden/scattered elsewhere in other repos or behind management UIs, and (ii) a relatively simple and documented pipeline syntax? The first example I can think of a tool that supported this workflow was Travis CI ~ 2011 - 2012. Appveyor offered similar capabilities quite early as well. Same workflow can be done with Gitlab, Google cloud build. > we can even deploy each pull request to the cloud rather inexpensively and leverage GitHub's environments to help manage the cleanup for us. This allows our team members to review and test changes in their browser before we pull them into our development branch Yeah, this kind of workflow is great. Another way this kind of workflow can be done is to create simple command line tools that developers can use to create and destroy temporary test environments running their speculative changes. In some cases, for rapid experimentation, it can be great to be able to spin up N temporary environments in parallel with different changes without tying it to pull requests. But I can see that tying the temporary environment lifecycle to the lifecycle of a PR might make it easier to share demos of proposed changes with reviewers. Out of curiosity, how reliable do you find the environment cleanup is? I remember building a similar create-temp-environment / destroy-temp-environment workflow for ephemeral databases running in AWS RDS driven by jenkins pipelines. It took a few months of tweaking to figure out how to ensure the RDS databases got torn down correctly and not "leaked" even if the jenkins master or workers failed midway through pipeline execution. From memory we had a bunch of exception handling in a jenkins groovy scripted pipeline that would run on the master jenkins to try to do cleanup, and even that wouldn't work all of the time, so we had a second cleanup job on a cron-schedule to detect and kill leaked resources.
simonw 5y ago
"You can’t have actions that call other actions" - I think it's possible to use the repository_dispatch trigger described at https://docs.github.com/en/actions/reference/events-that-trigger-workflows#repository_dispatch https://docs.github.com/en/actions/reference/events-that-tri... for this - you'd need a separate GitHub personal access token, but using that it should be possible to trigger a workflow in any other repository you own from an API call in another action.
- 147 5y ago
  I meant that in the context of composite actions. You can't package up a composite action that can call other actions, at least not yet.
  - arccy 5y ago
    Pretty new: https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu...
    - 147 5y ago
      Thanks! This is super exciting. Unfortunately, since we use GHES we'll be at least 6 months away from having this though :(.
    - jordemort 5y ago
      Ooh, I didn't realize this had shipped. Off to refactor all my boilerplate into its own repo...
- akerl_ 5y ago
  The issue is that once you add a PAT, you’ve imbued the build with your permissions. You can bound which permissions by call type (so, for example, you can make a token that can’t modify your profile info), but you can’t bound it by repository. So if you need the build to trigger write actions on one repo, now it has them for all repos.
thinkafterbef 5y ago
Disclaimer - incoming self promotion. Actually, most points in the article are the basis on why we created BuildJet. We initially tried to solve these annoyances by creating a CI with speed and the YAML config as a USP. We got 4x speed and a much better YAML config structure, but despite these improvements we noticed that it people had a mental barrier to migrate to a new unknown CI. But like OP we always enjoyed the experience of using GitHub Actions, so with this in mind. We decided to build BuildJet for GitHub Action[1] that uses the same infrastructure but plugs right into Github Action as a "self-hosted" runner, which is automatically set up for you with OAauth. This resulted on average a 2x speed improvement for half the cost(due to us being close to the metal). Easy to install and easy to revert. [1] https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions
- rexelhoff 5y ago
  Just checked it out. I like the idea of BuildJet. Something dead simple like this is what I've been looking for. The biggest problem is that I don't feel comfortable putting our IP out on a server managed by a small provider. If you had an offering where I could self-host, I'd be very interested. BTW. You have a broken link to "Privacy" from the "Terms" page.
sofixa 5y ago
Wow, reading this and all the comments makes me realise GHA is really far behind Gitlab CI. They've at least been playing catch-up since the MS acquisition, having stagnated for years before that. I really wonder why would anyone self-host GitHub. Gitlab has a much more feature rich, mature and cheap ( there's a perfectly usable free version) offering. Yeah, someone might prefer Github's UX, but is it really worth it to pay for a worse product?
- easton 5y ago
  GitHub’s UX is worth it, IMO. GitLab tries to do too many things and it’s really hard to get my bearings (it’s like Azure DevOps’ problem times 10). And if GitHub Actions doesn’t work, it’s really easy to integrate third party CI solutions.
  - sofixa 5y ago
    > GitHub’s UX is worth it, IMO. GitLab tries to do too many things and it’s really hard to get my bearings (it’s like Azure DevOps’ problem times 10). That's subjective. Gitlab does many more things, but nobody is forcing you to use them, and unused menu buttons don't matter much for the main workflows ( commits, MRs, repo visualisation, wikis, CI/CD).
  - adwww 5y ago
    Yeah I liked Gitlab when we used it for everything, but once we started to integrate with other services (Jira etc) instead of the built in functionality it felt like you were fighting how they had designed it.
    - john_cogs 5y ago
      GitLab team member here. We've put a lot of effort into our integration with Jira. The docs [1] detail how it works and we have a blog [2] with some screenshots so you can see what it looks like in action. What kind of troubles have you encountered? 1 - https://docs.gitlab.com/ee/integration/jira/ https://docs.gitlab.com/ee/integration/jira/ 2 - https://about.gitlab.com/blog/2021/03/25/integrating-gitlab-com-with-atlassian-jira-cloud/ https://about.gitlab.com/blog/2021/03/25/integrating-gitlab-...
chad_oliver 5y ago
> You can’t have actions that call other actions As of August 25, you can! https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu...
deleted 5y ago
[deleted]
chad_oliver 5y ago
The biggest limitation I'm hitting with Github Actions right now is that there's no real support for queueing jobs up. Github actions do support a 'concurrency group' primitive that will prevent two actions in the same concurrency group from running at the same time, but this only allows you to have one item executing and one item queued up. If you try to queue up another one, the first queued item will be cancelled. In our case we've got some cypress tests that we want to run on a specific on-premise server every time we create a pull request. They take about 20 minutes to run, and we're creating a lot of pull requests, so you have to carefully check what github actions are executing before you create a pull request or push new changes to it. I'd love support for proper queues like what teamcity and other CI systems have.
- jhgb 5y ago
  > The biggest limitation I'm hitting with Github Actions right now is that there's no real support for queueing jobs up. Maybe they need to reinvent JES and JCL. :)
  - aidanr 5y ago
    Now that made me smile! Add in a dose of CA-7 and job's a goodun.
  - AltF4me 5y ago
    Turnstyle? We've had similar issues with Terraform deployments. https://github.com/softprops/turnstyle https://github.com/softprops/turnstyle
- AltF4me 5y ago
  Turnstyle? We've had similar issues with Terraform deployments. https://github.com/softprops/turnstyle https://github.com/softprops/turnstyle
staticassertion 5y ago
My experience with GHA is that it can be awesome for things like small projects that want to enforce linters, unit tests, etc. Once you get into more complex things - like building docker images, storing into an artifact repository, baking amis, running integration or end to end tests, etc, it can be a pain. It was a great place for us to start but we've since moved to BuildKite.
- athorax 5y ago
  Agreed. It is great for things that you want to run directly against your code base (like linters, tests, etc.)
throwaway20371 5y ago
> migrate every engineering team at Venmo to GitHub Actions Oh jesus christ. I feel for you dude. We evaluated GHA, and we still are trying to use it, but there is a barrage of problems and limitations, including cost, lack of functionality, and technical issues. It's really only suitable (at scale) for linting, or generating Changelogs, or something else trivial. I use it in my OSS projects to run tests, and it's okay for that (though impossible to just tail a build log when it's large) Drone.io is still an amazingly effective system that matches GHA (and has _crazy_ features like build parameters) but is more flexible. Of course you'll have to pay for commercial licenses, but if it's between paying for GHA or Drone, I highly recommend Drone instead. Drone is stupidly easy to maintain (infrastructure-wise).
elpakal 5y ago
Build engineer at a large Fortune 500 shop here - the largest impediment to us even enabling GitHub Actions is that you either disable them entirely for your org or allow repo admins to enable them _per repo_. We have several hundred repos in one org alone and we cannot simply enable them for everyone with admin access (for us if you create a repo, you get admin access).
- adwww 5y ago
  > cannot simply enable them for everyone with admin access why not? is that a cost issue, perceived security, administration overhead?
  - athorax 5y ago
    Not OP, but in our case allowing essentially the entire company to start to use GHA would result in a huge load increase on our GHE deployment. Even with self-hosted runners per org, there is a lot of overhead that is ran on the github servers themselves. We are working towards rolling it out, but there are definitely some scalability issues.
    - pojzon 5y ago
      We are running dedicated agents per repo, anytime someone creates a repo an agent pool is created for him. Everything is running on AKS. We dont even pay for it THAT much, because container-agents are spinned up when needed. (Also few hundred repos)
    - hardwaresofton 5y ago
      If off-premise is an option for you I’d like to suggest a service I run called SurplusCI (surplusci.com). It’s cheaper than self hosting most of the time and is SHOULD be roughly zero toil for you.
      - jffry 5y ago
        I'm guessing since @athorax mentioned "our GHE deployment" that they're running GHE on-prem instead of using hosted GHE. I'd be surprised if they were amenable to off-prem Actions runner hosting if they chose to run github on-prem
arminiusreturns 5y ago
Does anyone know if on GHE I can bypass actions and just use the builtin git recieve hooks? I really enjoy playing with post-commit scripts.
ChrisMarshallNY 5y ago
I think GH Actions is a pretty cool idea. I don't use them, myself, because, every time I count myself, I keep coming up "1." When I left my last job, and started working on my own, I set up things like CI/D, JIRA, Jenkins, etc. These were the bread and butter for development in my old shop. But they are "Concrete Galoshes"[0], and work very well for teams, as opposed to ICs. As a single developer, working alone, the infrastructure overhead just slowed me down, and, ironically, interfered with Quality. When GH Actions were first announced (I can't remember, but they may have been beta, then), I set up several of them, on my busier projects. They worked great, until I started to introduce some pivots, and I realized that there was actually no advantage to them. I ran the tests manually, anyway, and the Actions just gave me one more thing to tweak. It was annoying, getting the failure messages, when I knew damn well, the project was fine. I'd just forgotten to tweak the Action. I introduce frequent changes, in my work, and that is great. [0] https://littlegreenviper.com/miscellany/concrete-galoshes/ https://littlegreenviper.com/miscellany/concrete-galoshes/
- jayflux 5y ago
  I don’t think you need to be in a team to get good mileage out of Actions. For example. Some open source code I need to publish to package-indexes or marketplaces, and will have an action to do that whenever I publish a release, I’ve found it to be very useful and saves a lot of time.
  - ChrisMarshallNY 5y ago
    I agree. It's just a difference in style. I find mine is quite productive, for the way that I work.
- shadycuz 5y ago
  You can run GitHub actions locally but I forget the name of the tool. You can also run actions on top of Jenkins. https://github.com/DontShaveTheYak/jenkins-std-lib https://github.com/DontShaveTheYak/jenkins-std-lib
  - 5y ago
shadycuz 5y ago
If you use Jenkins and want to try actions then check out https://github.com/DontShaveTheYak/jenkins-std-lib https://github.com/DontShaveTheYak/jenkins-std-lib It let's you run actions on top of Jenkins.
AJRF 5y ago
Has anyone got experience or resources for running a big iOS application on GHA? Is it possible, or is it pretty much a toy right now? Say you have a big CI pipeline like; * Full clean build including dependencies (support Carthage, Cocoapods, SPM) * Running multiple test suites that takes maybe 5+ hours for full suite? * Running simulators for screenshot testing
epage 5y ago
I really like Github Actions but it still only feels appropriate for small projects. Unless I'm missing something, I didn't see a good mechanism for monorepos. I'm thinking in terms of there being shared pieces that get built/tested and then products that sit on top. The complication comes in test avoidance. I don't remember why I didn't like doing this all in one workflow with jobs (though it was going to require an orchestrator job setting variables to choose which downstream jobs to run). For chaining workflows/pipelines, when I looked, you could only trigger other workflows for master, defeating the point. Among my small, open source work, probably my biggest complaint is actions running in forks. Wastes a lot of resources on their side and limits my concurrent runners for projects in my personal space. For companies, depending on the setup, this would eat their compute minutes. Also annoying that PR actions can't post to the PR. I can understand there are security limitations but it makes it so a lot of nice features don't exist for most people.
jeremyepling 5y ago
Thanks for taking the time to share your experience. I work on GitHub Actions and am familiar with the limitations you're experiencing. Below is more info on where we're at with each of these issues, using the numbers in your Table of Contents. Also, we have a lot of other new things planned for Actions as you can see in our roadmap. https://github.com/github/roadmap/projects/1?card_filter_query=label%3Aactions https://github.com/github/roadmap/projects/1?card_filter_que... 2.1 We're starting work on this in the next couple of months. We plan to ship it in early 2022. 2.2 We want to speed up the pace of GitHub Enterprise Server releases, but I don't have more info to share. 2.3 We're looking at ways to not require a GHEC account or "unified" license. 2.4 The limits are much higher with the GitHub hosted runners, but this is a current limit of self-hosted runners. 3.1 It's on our backlog. No date to share. 3.2 I haven't heard this before and thanks for sharing the scenario. We'll think about it. 3.3 This will ship in October. 3.4 We're doing some performance optimizations for GHES 3.4 that should fix this. 3.5 This shipped recently - https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu... 3.6 We have a couple API improvements coming later this year. 3.7 We're looking into this, but no dates to share. We're dedicated to making Actions a great experience. As you would assume, I'm very excited about the future of Actions and getting feedback like this helps us make it better.
- nicois 5y ago
  We have recently switched to GitHub actions and in addition to the above, there are two others which impact us: Sometimes the checkout action fetches the wrong commit! The SHA in the GitHub environment variable says one commit, but the actual code is different(!). Because we don't know why this happens we basically need to do an unshallow fetch of the whole repo to be sure we have what we expect. Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it. This is essential to avoid broken workflows while scaling in. Gitlab supports this via a signal, but there is no equivalent.
  - ethomson 5y ago
    > Sometimes the checkout action fetches the wrong commit! I'm a product manager at GitHub, and I'm investigating this now. Feel free to drop me an email (ethomson@github.com) if you can share more information about the workflow runs that were affected by this. > Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it. We're building this functionality out right now, so that you can have self-hosted runners that will run a single job and we will let you subscribe to webhooks that will let you scale up/down your fleet of runners.
boleary-gl 5y ago
GitLab team-member here. Obviously coming in with a lot of bias, but I wanted to address how each point relates to GitLab CI/CD’s view of the world. I’m also thinking about writing a longer post with more details as I have a lot of thoughts (™) about this topic. 2.1 Caching isn’t available: GitLab has this everywhere. 2.2 GitHub Enterprise Server is behind GitHub Enterprise Cloud: GitLab ships the same code to GitLab.com as it does to our self-managed customers. This was a tough decision but has a lot of benefits...the central being feature parity and scalability for self-managed folks 2.3 Using Public GitHub.com Actions: This is a symptom more than the problem itself - relying on third-party plugins for build jobs is scary, and leads to many of the same issues we’ve seen in the Jenkins ecosystem - easy to get started, hard to maintain. 2.4 Dockerhub pull rate limiting: for self-hosted runners, you can use a registry mirror or Dependency Proxy to reduce your number of pulls from Docker Hub. The key is the entire platform has to be there to enable the right workflows. 3.1 No dropdowns for manually triggered jobs: GitLab also doesn’t have drop downs, but does have the ability to pre-fill these values. 3.2 Self-hosted runner default labels: I think this is also more of a symptom than a problem. 3.3 Being able to tag and use runners for specific tasks is key - so I understand the frustration and we’ve spent a lot of time on this. 3.4 You can’t restart a single job of a workflow: You can do this with GitLab. 3.5 Slow log output: I haven’t seen this be a problem, and is a benefit of our scalability features being built into the self-managed code. 3.6 You can’t have actions that call other actions: There are lots of ways to relate pipelines (parent/child, triggers. etc.) in GitLab. 3.7 Metrics and observability: The GitLab runner has Prometheus build in, and the dashboards we use to manage GitLab.com are partially public: https://dashboards.gitlab.com https://dashboards.gitlab.com 3.8 Workflow YAML Syntax can confusing: This can be really hard to get right. I learned to stop worrying and love the YAML long ago, and I know we’ve got through a lot of iterations to try and get this right. I'd love to know where folks think I got this assessment wrong. And is there value in writing more about it? (edited for line spacing)
- noptd 5y ago
  >2.2 GitHub Enterprise Server is behind GitHub Enterprise Cloud: GitLab ships the same code to GitLab.com as it does to our self-managed customers. Kudos to Gitlab for not utilizing the GHE model in order to convert users to their hosted solutions.
thedougd 5y ago
Surprised there's no mention of the inability to view logs generated before the page was loaded until the job completes. This one drives me crazy when I have long running, silent activity. My wishlist item would be more variants of Windows server versions so that we could build Windows containers for more versions of Windows. I realize the fault lies with Windows containers pinning the container base version to the host version, but I'm still stuck with the burden. I think GitHub Actions got the model correct, using everything as events to trigger any number of workflows. This is far simpler to maintain than a single workflow with conditionals and wait states that you see with other systems.
Aeolun 5y ago
I was planning to migrate to Github from Gitlab, but hearing about all these missing basic features (no caching, no restarting individual jobs?) I think I’ll stick with Gitlab for the forseeable future.
- WillDaSilva 5y ago
  Why were you planning to migrate in the first place?
  - Aeolun 5y ago
    Gitlab isn’t supported by the enterprise, so I basically have to maintsin the whole thing myself. We also don’t have a license, so it’s missing some much needed features. Apparently it’s still better than Github though xD
gitgud 5y ago
> For the vast majority of use cases, the YAML syntax is sane and is similar to other CI systems. It gets super clumsy when you want to assign an output of a step to a variable that you can refer to later. The verbosity of accessing output has the added benefit of making it much clearer that the 2 workflow steps are tightly interdependent on each other.
billsdatabikes 5y ago
We use GitHub Actions and we like it. Though the current version feels like a very lukewarm compromise from the original one they had before the Microsoft acquisition. I am rather disappointed to see it become "just another CI" when the original release was so much more. I know, I know, I know, Microsoft bought GitHub so what should I expect, but either way, disappointed when the original had such amazing potential.
smcleod 5y ago
I've been testing out Github Actions for a few weeks now, for the most part I really like it, there are a few features missing but I think the fundamentals of the product are solid with the public catalogue of actions being the killer feature. The biggest issue I have is around self-hosted runners. 1. There's no official auto-scaling runner option, so even if you're paying Github (aka Microsoft) for Enterprise - they're not going to support your auto-scaling EKS/GKE/EC2/whatever runners. 2. You can't register self-hosted runners without a Personal Access Token - the key word being _Personal_. Your automation code for provisioning runners should not rely on an individuals Github access token just to register, they need to have a system like GitLab has where you can generate a registration token per-organisation/team/repo that allows you to programmatically register runners.
- eberkund 5y ago
  I think I recall seeing something on the GitHub forums where the recommended approach was the create a separate service user account and all your non-personal-PATs with that account.
  - zeisss 5y ago
    Which means you have to pay more money. If we want fine-tune the permissions of each repo we have to create one user per repository.
- 147 5y ago
  We use https://github.com/actions-runner-controller/actions-runner-controller https://github.com/actions-runner-controller/actions-runner-... to auto scale on EKS. It also allows you to use a GitHub App for the runner registration instead of personal access tokens. Also it seems like the project is receives support from GitHub because they’re getting early access to test out features.
  - smcleod 5y ago
    Thanks, I had a play with this and submitted a PR to add some configuration options to the CRD. It seemed decent but I hit two problems: 1. I didn't manage to get autoscaling to work - I suspect my helm templates might have been incorrect. 2. Docker-in-Docker (DIND) I know works, but one of the clients I'm working with has switched to containerd and the controller got a little confused by Docker-in-Containerd (DINC - you heard it here first!), I know really they should be using Kaniko/Buildah etc... but their devs aren't ready to make the change yet.
Waterluvian 5y ago
I’ve been loving GH Actions a ton but one issue I ran into is that I have a couple tests that get run and then if it’s a merge to prod branch, an image build action. The problem is that I cannot make the image build action happen IFF both testing actions pass. I had to combine all three actions into one. It works. But now there’s a “skipped” step that’s skipped 99% of the time and makes no sense for a lot of PRs. It also means I have a Frankenstein monster action that does three long lists of very different things. All just so I can make 3 depend on 1 and 2. The other problem is that to develop and Test an action, I have to just push to origin a thousand times. My kingdom for a CI system that _trivially_ enables me to install a single one-liner program that lets me locally test my actions at near 1:1 compatibility.
Dowwie 5y ago
I've yet to come across a complete tutorial for setting up cached docker builds (one involving a rust compilation) within a Github Actions workflow. I've been figuring it out from pieces of info scattered across blog posts and github repos. How is anyone managing this today? How/where are you persisting cached objects?

217 points by 147 5y ago

Is it possible yet to (without hoops) have an action workflow that runs on pull requests to <default_branch> without having to name that branch explicitly so that you can have a write-once-run-everywhere-forever process across repos that might change their default branch name? Last I saw there was a template parameter, but those get populated at creation not at runtime.

bob1029 5y ago

The only thing we've ever trusted GH actions with is enforcing check builds pass before a PR is allowed to merge. Everything else is managed via a custom tool we use for packaging & deploying our product. Even our simple "run this 1 build command and ensure exit code == 0" action seems to have a semi-weekly issue like stuck "waiting for status" and other unexplained failures throughout. We don't want to put any more eggs into that particular basket right now.

judge2020 5y ago

The cache action section focuses on GHES which is a shame, since the GitHub.com cache action is also pretty bad. - can't manually delete cache (although they're considering changing that[0]) - only saves cache at the end of a workflow, and only if the workflow succeeds. This could be solved with CircleCI's approach of having a save-cache step and a restore-cache step. - Cache is super slow for self-hosted runners, so it makes more sense to have a local cache instead of using the action - only 5gb of storage size. This was supposed to be increased via billable cache storage[1], but it's been on the backburner since July 2020 - In addition to the above, you can't use a different storage backend on the official action (which would allow storing over 5GB of cache via your own storage). The best workaround is to use a user-provided action which utilizes the s3 api[2]. 0: https://github.com/actions/cache/issues/632 https://github.com/actions/cache/issues/632 1: https://github.com/github/roadmap/issues/66 https://github.com/github/roadmap/issues/66 2: https://github.com/actions/cache/issues/354#issuecomment-854762521 https://github.com/actions/cache/issues/354#issuecomment-854... 2>

trulyme 5y ago
- can't cache on tags [0] Caching is a joke. But then again, comparing GHA to GitLab CI is like comparing a pogo stick to a bicycle. Sure, both can be used to get there, but one ride is a bit... bumpy. (I use both and I am not affiliated to either.) [0] https://github.com/actions/cache/issues/556 https://github.com/actions/cache/issues/556
codys 5y ago
I'm heavily using cache on self hosted runners that are running in ec2 machines on aws. It's my experience that the cache isn't "super slow". Because of a bug (in my action) I was pushing/pulling gigabytes of data into the cache. On large files I was seeing ~125MB/s (that's Byte) download speeds. I haven't used the GitHub hosted runners much so I'm not sure how it compares though.
- judge2020 5y ago
  The hosted runners seem to be very varied in performance. Just today an actions/checkout action on GitHub's hosted Linux box went down to 1 megabit per second and was going to take 30+ minutes to clone a 3gb repository. Re-running the workflow an hour later had it running fine.

lmeyerov 5y ago

We've been loving GHA: CI/CD-as-code, pull requests capture deploy history (CI, stage/restage, deploy), and labeling a PR with 'release' is enough to generate our on-prem + multi-cloud artifacts. Our main gotchas are roughly: - GH-hosted runners have too little RAM/HD for big docker software. They push you to self-hosted runners for that, which is fine in theory, but GHA/Azure doesn't actually support serverless runners, so that falls flat in practice. We don't want to be turning machines on/off, that's GHA's job. We experimented with GHA -> Packer -> Azure for serverless, but it was slow and Packer frequently leaves zombie machines, so we went back to tweaking the low-RAM runners provided by our enterprise plan. - Security: We want contactors etc. to be able to run limited GHA CI jobs and use that quota, but not higher-trust GHA CD ones. This is tricky at a configuration level. Ex: It seems like we'd need to funny things like main repo for CI w/ CI secrets, and a separate repo for CD w/ CD secrets, and only give untrusted folks access to the CD-cred repo. We've thought of other possibilities as well, but in general, it's frightening. - Big Docker images: We do spend more time than I'd like messing with optimizing Docker caching as GPU containers are embarrassingly big (we use dockerhub vs github's due to sizes/pricing/etc), think both multi-stage containers + multi-step jobs (monorepo/microservices). I think they're in a good position to speed that up! I'm optimistic about these, but tricky to align with MS/GH PM personal team priorities :)

yebyen 5y ago
I work on Flux, and for the reasons you mentioned we advocate for a strict separation between CI and CD, where CI is responsible for building and testing images and pushing out releases, and CD is responsible for production. There is no crossover between these roles, and this makes it very secure. You use a policy CRD called ImagePolicy to declare what versions should be matched and deployed automatically. You can have them deployed directly to production, or if it makes you more comfortable with your contractors, to an "auto-PR" branch which simply queues them up for a release engineer with the required juice to approve and merge the changes to prod. There is no deploy job in CI, since the deployment in prod is whatever happens to be on the main branch in the config repo. This is a protected branch which only releng can merge changes into, usually only when the release checks are satisfied. Flux (which scans any image repos) updates those manifests when there is a release, either directly or via pull request. This strategy I think would be able to address your Security concerns regarding contractors, by reducing the responsibility of your CI system to only CI, and not to handle CD anymore. This is the subject of my talk at KubeCon[1], although you might not be able to get that from the title, (I'm presenting this topic for a Jenkins audience, but the focus is on Flux and how Flux works) so I'm hoping it should be applicable to a broader audience, certainly inclusive of GitHub users :) [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-with-declarative-everything-kingdon-barrett-weaveworks?iframe=no https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-wi...

m_a_g 5y ago

One extremely painful "gotcha" we encountered was trying to push code to a protected branch from inside a workflow. With the default GITHUB_TOKEN, you can't push to protected branches. If you decide to use personal access tokens, you can push to protected branches, BUT that will trigger other workflows. That can cause an infinite loop of workflows. We still couldn't figure out how to push code to a protected branch without triggering the same/other workflows.

simonw 5y ago
One trick that might work is to include a token such as 'DO_NOT_TRIGGER_ACTIONS' in your automated commit message, then have your other action skip any commits that include that token using this in the workflow: if: "!contains(github.event.head_commit.message, 'DO_NOT_TRIGGER_ACTIONS')"
- m_a_g 5y ago
  good idea, thanks!
chad_oliver 5y ago
> We still couldn't figure out how to push code to a protected branch without triggering the same/other workflows. If you include `[skip ci]` anywhere in the commit message, that commit won't trigger any github actions. This is a built-in behavior; you don't have to manually check for that string in any downstream actions. I spent a long time trying to work around the same constraint before my colleague pointed out that `[skip ci]` was a thing.
dcherman 5y ago
You can also use a deploy key. That's how I do it while avoiding adding a PAT to a shared repo

oefrha 5y ago

Also, scheduled jobs are a joke, they are routinely one to several hours late. I have to use the API to manually trigger jobs (workflow_dispatch) whenever I need scheduling that is remotely time sensitive.

147 5y ago
I've heard of issues with scheduled jobs on Github.com but we didn't have any issues so far on GHES.
chrisfosterelli 5y ago
This was the first thing that came to mind for me too. I don't need it to be exact, sure, but the scheduled jobs are more than an hour off more often than not. Sometimes several hours as you mentioned. Frequently they would also just not run at all! It was so bad I ended up scheduling my cron jobs to run 4 - 5 hours early and just sit there idling until the actual execution time came around haha. Ultimately not running at all started to be a real pain point, I gave up and paid $5 for a monthly box to sit there and run crond to save on the headache.
seanhandley 5y ago
Yes their scheduler is woeful. I have a few suspicions that it's intentionally done this way for balancing demand spikes and that jobs scheduled for the top of hour are worst affected.

Octabrain 5y ago

In the company I am working we’ve decided to use Github actions for our CI pipelines instead of deploying any “on prem” solution. I’ve worked a lot in the past with Jenkins and Travis and I also played a bit at home with Github actions. Now that I am using it in real world scenarios I have to say that I am a bit disappointed. In my opinion, as soon as you try to do something a bit complex you end up having to implement some nasty hack. Also I found that the Github Actions marketplace is a bit of a sink. You have to spend a good amount of time browsing it for finding something decent that hasn’t been discontinued, or is a pointless fork or at least is actively maintained. This happens even for basic functionalities. I known is a fairly recent platform but I was expecting much more compared to what other services offer.

takeda 5y ago
This is why I like GitLab much more. They implemented a proper CI. I think what GitHub was going for is to create some kind of ecosystem for proprietary 3rd party applications and use that as a revenue. That approach only crippled the whole functionality.
- jaxn 5y ago
  Maybe it's just me, but I refuse to use third party actions in our build and deployment steps. It seems like such an obvious and avoidable attack surface to me, and one where an attached can have root on our production instances and at least some access to all developer machines.
  - ameliaquining 5y ago
    You can solve that by pinning commit hashes, so that nobody can change the code of the actions you use without your consent. You can then use Dependabot to automatically get PRs to update to the latest version of each action when it comes out. You still get the chance to review each PR before it goes in. I wish GitHub would implement a security setting requiring repositories to do this within an organization.
ak217 5y ago

cebert 5y ago

GitHub Actions are a fantastic experience for serverless applications. I am working on a serverless project where we use GitHub Actions exclusively for CI/CD as well as running automated tests. We rely heavily on Lambda, S3, and DynamoDB. Our client app is static JS files we serve over Cloud Font. GitHub Actions make our piplines accessible to any developer on the team. Since we only pay for what we use with our serverless infrastructure, we can even deploy each pull request to the cloud rather inexpensively and leverage GitHub's environments to help manage the cleanup for us. This allows our team members to review and test changes in their browser before we pull them into our development branch. We additionally can run Playwright E2E tests to verify that none of our critical user workflow scenarios have broken resulting from the PR changes. I love this development experience and would have a hard time going back to anything else.

shoo 5y ago
thank you for sharing -- as i was reading through i tried to understand what about the workflow was specific to github actions vs other CI automation. > GitHub Actions make our pipelines accessible to any developer on the team Do you reckon this accessibility is a combination of (i) storing the pipeline definitions in the application's source repo, where application developers can find them easily, not hidden/scattered elsewhere in other repos or behind management UIs, and (ii) a relatively simple and documented pipeline syntax? The first example I can think of a tool that supported this workflow was Travis CI ~ 2011 - 2012. Appveyor offered similar capabilities quite early as well. Same workflow can be done with Gitlab, Google cloud build. > we can even deploy each pull request to the cloud rather inexpensively and leverage GitHub's environments to help manage the cleanup for us. This allows our team members to review and test changes in their browser before we pull them into our development branch Yeah, this kind of workflow is great. Another way this kind of workflow can be done is to create simple command line tools that developers can use to create and destroy temporary test environments running their speculative changes. In some cases, for rapid experimentation, it can be great to be able to spin up N temporary environments in parallel with different changes without tying it to pull requests. But I can see that tying the temporary environment lifecycle to the lifecycle of a PR might make it easier to share demos of proposed changes with reviewers. Out of curiosity, how reliable do you find the environment cleanup is? I remember building a similar create-temp-environment / destroy-temp-environment workflow for ephemeral databases running in AWS RDS driven by jenkins pipelines. It took a few months of tweaking to figure out how to ensure the RDS databases got torn down correctly and not "leaked" even if the jenkins master or workers failed midway through pipeline execution. From memory we had a bunch of exception handling in a jenkins groovy scripted pipeline that would run on the master jenkins to try to do cleanup, and even that wouldn't work all of the time, so we had a second cleanup job on a cron-schedule to detect and kill leaked resources.

simonw 5y ago

"You can’t have actions that call other actions" - I think it's possible to use the repository_dispatch trigger described at https://docs.github.com/en/actions/reference/events-that-trigger-workflows#repository_dispatch https://docs.github.com/en/actions/reference/events-that-tri... for this - you'd need a separate GitHub personal access token, but using that it should be possible to trigger a workflow in any other repository you own from an API call in another action.

147 5y ago
I meant that in the context of composite actions. You can't package up a composite action that can call other actions, at least not yet.
- arccy 5y ago
  Pretty new: https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu...
  - 147 5y ago
    Thanks! This is super exciting. Unfortunately, since we use GHES we'll be at least 6 months away from having this though :(.
  - jordemort 5y ago
    Ooh, I didn't realize this had shipped. Off to refactor all my boilerplate into its own repo...
akerl_ 5y ago
The issue is that once you add a PAT, you’ve imbued the build with your permissions. You can bound which permissions by call type (so, for example, you can make a token that can’t modify your profile info), but you can’t bound it by repository. So if you need the build to trigger write actions on one repo, now it has them for all repos.

thinkafterbef 5y ago

Disclaimer - incoming self promotion. Actually, most points in the article are the basis on why we created BuildJet. We initially tried to solve these annoyances by creating a CI with speed and the YAML config as a USP. We got 4x speed and a much better YAML config structure, but despite these improvements we noticed that it people had a mental barrier to migrate to a new unknown CI. But like OP we always enjoyed the experience of using GitHub Actions, so with this in mind. We decided to build BuildJet for GitHub Action[1] that uses the same infrastructure but plugs right into Github Action as a "self-hosted" runner, which is automatically set up for you with OAauth. This resulted on average a 2x speed improvement for half the cost(due to us being close to the metal). Easy to install and easy to revert. [1] https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions

rexelhoff 5y ago
Just checked it out. I like the idea of BuildJet. Something dead simple like this is what I've been looking for. The biggest problem is that I don't feel comfortable putting our IP out on a server managed by a small provider. If you had an offering where I could self-host, I'd be very interested. BTW. You have a broken link to "Privacy" from the "Terms" page.

sofixa 5y ago

Wow, reading this and all the comments makes me realise GHA is really far behind Gitlab CI. They've at least been playing catch-up since the MS acquisition, having stagnated for years before that. I really wonder why would anyone self-host GitHub. Gitlab has a much more feature rich, mature and cheap ( there's a perfectly usable free version) offering. Yeah, someone might prefer Github's UX, but is it really worth it to pay for a worse product?

easton 5y ago
GitHub’s UX is worth it, IMO. GitLab tries to do too many things and it’s really hard to get my bearings (it’s like Azure DevOps’ problem times 10). And if GitHub Actions doesn’t work, it’s really easy to integrate third party CI solutions.
- sofixa 5y ago
  > GitHub’s UX is worth it, IMO. GitLab tries to do too many things and it’s really hard to get my bearings (it’s like Azure DevOps’ problem times 10). That's subjective. Gitlab does many more things, but nobody is forcing you to use them, and unused menu buttons don't matter much for the main workflows ( commits, MRs, repo visualisation, wikis, CI/CD).
- adwww 5y ago
  Yeah I liked Gitlab when we used it for everything, but once we started to integrate with other services (Jira etc) instead of the built in functionality it felt like you were fighting how they had designed it.
  - john_cogs 5y ago
    GitLab team member here. We've put a lot of effort into our integration with Jira. The docs [1] detail how it works and we have a blog [2] with some screenshots so you can see what it looks like in action. What kind of troubles have you encountered? 1 - https://docs.gitlab.com/ee/integration/jira/ https://docs.gitlab.com/ee/integration/jira/ 2 - https://about.gitlab.com/blog/2021/03/25/integrating-gitlab-com-with-atlassian-jira-cloud/ https://about.gitlab.com/blog/2021/03/25/integrating-gitlab-...

chad_oliver 5y ago

> You can’t have actions that call other actions As of August 25, you can! https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu...

deleted 5y ago

[deleted]

chad_oliver 5y ago

The biggest limitation I'm hitting with Github Actions right now is that there's no real support for queueing jobs up. Github actions do support a 'concurrency group' primitive that will prevent two actions in the same concurrency group from running at the same time, but this only allows you to have one item executing and one item queued up. If you try to queue up another one, the first queued item will be cancelled. In our case we've got some cypress tests that we want to run on a specific on-premise server every time we create a pull request. They take about 20 minutes to run, and we're creating a lot of pull requests, so you have to carefully check what github actions are executing before you create a pull request or push new changes to it. I'd love support for proper queues like what teamcity and other CI systems have.

jhgb 5y ago
> The biggest limitation I'm hitting with Github Actions right now is that there's no real support for queueing jobs up. Maybe they need to reinvent JES and JCL. :)
- aidanr 5y ago
  Now that made me smile! Add in a dose of CA-7 and job's a goodun.
- AltF4me 5y ago
  Turnstyle? We've had similar issues with Terraform deployments. https://github.com/softprops/turnstyle https://github.com/softprops/turnstyle
AltF4me 5y ago
Turnstyle? We've had similar issues with Terraform deployments. https://github.com/softprops/turnstyle https://github.com/softprops/turnstyle

staticassertion 5y ago

My experience with GHA is that it can be awesome for things like small projects that want to enforce linters, unit tests, etc. Once you get into more complex things - like building docker images, storing into an artifact repository, baking amis, running integration or end to end tests, etc, it can be a pain. It was a great place for us to start but we've since moved to BuildKite.

athorax 5y ago
Agreed. It is great for things that you want to run directly against your code base (like linters, tests, etc.)

throwaway20371 5y ago

> migrate every engineering team at Venmo to GitHub Actions Oh jesus christ. I feel for you dude. We evaluated GHA, and we still are trying to use it, but there is a barrage of problems and limitations, including cost, lack of functionality, and technical issues. It's really only suitable (at scale) for linting, or generating Changelogs, or something else trivial. I use it in my OSS projects to run tests, and it's okay for that (though impossible to just tail a build log when it's large) Drone.io is still an amazingly effective system that matches GHA (and has _crazy_ features like build parameters) but is more flexible. Of course you'll have to pay for commercial licenses, but if it's between paying for GHA or Drone, I highly recommend Drone instead. Drone is stupidly easy to maintain (infrastructure-wise).

elpakal 5y ago

Build engineer at a large Fortune 500 shop here - the largest impediment to us even enabling GitHub Actions is that you either disable them entirely for your org or allow repo admins to enable them _per repo_. We have several hundred repos in one org alone and we cannot simply enable them for everyone with admin access (for us if you create a repo, you get admin access).

adwww 5y ago
> cannot simply enable them for everyone with admin access why not? is that a cost issue, perceived security, administration overhead?
- athorax 5y ago
  Not OP, but in our case allowing essentially the entire company to start to use GHA would result in a huge load increase on our GHE deployment. Even with self-hosted runners per org, there is a lot of overhead that is ran on the github servers themselves. We are working towards rolling it out, but there are definitely some scalability issues.
  - pojzon 5y ago
    We are running dedicated agents per repo, anytime someone creates a repo an agent pool is created for him. Everything is running on AKS. We dont even pay for it THAT much, because container-agents are spinned up when needed. (Also few hundred repos)
  - hardwaresofton 5y ago
    If off-premise is an option for you I’d like to suggest a service I run called SurplusCI (surplusci.com). It’s cheaper than self hosting most of the time and is SHOULD be roughly zero toil for you.
    - jffry 5y ago
      I'm guessing since @athorax mentioned "our GHE deployment" that they're running GHE on-prem instead of using hosted GHE. I'd be surprised if they were amenable to off-prem Actions runner hosting if they chose to run github on-prem

arminiusreturns 5y ago

Does anyone know if on GHE I can bypass actions and just use the builtin git recieve hooks? I really enjoy playing with post-commit scripts.

ChrisMarshallNY 5y ago

I think GH Actions is a pretty cool idea. I don't use them, myself, because, every time I count myself, I keep coming up "1." When I left my last job, and started working on my own, I set up things like CI/D, JIRA, Jenkins, etc. These were the bread and butter for development in my old shop. But they are "Concrete Galoshes"[0], and work very well for teams, as opposed to ICs. As a single developer, working alone, the infrastructure overhead just slowed me down, and, ironically, interfered with Quality. When GH Actions were first announced (I can't remember, but they may have been beta, then), I set up several of them, on my busier projects. They worked great, until I started to introduce some pivots, and I realized that there was actually no advantage to them. I ran the tests manually, anyway, and the Actions just gave me one more thing to tweak. It was annoying, getting the failure messages, when I knew damn well, the project was fine. I'd just forgotten to tweak the Action. I introduce frequent changes, in my work, and that is great. [0] https://littlegreenviper.com/miscellany/concrete-galoshes/ https://littlegreenviper.com/miscellany/concrete-galoshes/

jayflux 5y ago
I don’t think you need to be in a team to get good mileage out of Actions. For example. Some open source code I need to publish to package-indexes or marketplaces, and will have an action to do that whenever I publish a release, I’ve found it to be very useful and saves a lot of time.
- ChrisMarshallNY 5y ago
  I agree. It's just a difference in style. I find mine is quite productive, for the way that I work.
shadycuz 5y ago
You can run GitHub actions locally but I forget the name of the tool. You can also run actions on top of Jenkins. https://github.com/DontShaveTheYak/jenkins-std-lib https://github.com/DontShaveTheYak/jenkins-std-lib
- 5y ago

shadycuz 5y ago

If you use Jenkins and want to try actions then check out https://github.com/DontShaveTheYak/jenkins-std-lib https://github.com/DontShaveTheYak/jenkins-std-lib It let's you run actions on top of Jenkins.

AJRF 5y ago

Has anyone got experience or resources for running a big iOS application on GHA? Is it possible, or is it pretty much a toy right now? Say you have a big CI pipeline like; * Full clean build including dependencies (support Carthage, Cocoapods, SPM) * Running multiple test suites that takes maybe 5+ hours for full suite? * Running simulators for screenshot testing

epage 5y ago

I really like Github Actions but it still only feels appropriate for small projects. Unless I'm missing something, I didn't see a good mechanism for monorepos. I'm thinking in terms of there being shared pieces that get built/tested and then products that sit on top. The complication comes in test avoidance. I don't remember why I didn't like doing this all in one workflow with jobs (though it was going to require an orchestrator job setting variables to choose which downstream jobs to run). For chaining workflows/pipelines, when I looked, you could only trigger other workflows for master, defeating the point. Among my small, open source work, probably my biggest complaint is actions running in forks. Wastes a lot of resources on their side and limits my concurrent runners for projects in my personal space. For companies, depending on the setup, this would eat their compute minutes. Also annoying that PR actions can't post to the PR. I can understand there are security limitations but it makes it so a lot of nice features don't exist for most people.

jeremyepling 5y ago

Thanks for taking the time to share your experience. I work on GitHub Actions and am familiar with the limitations you're experiencing. Below is more info on where we're at with each of these issues, using the numbers in your Table of Contents. Also, we have a lot of other new things planned for Actions as you can see in our roadmap. https://github.com/github/roadmap/projects/1?card_filter_query=label%3Aactions https://github.com/github/roadmap/projects/1?card_filter_que... 2.1 We're starting work on this in the next couple of months. We plan to ship it in early 2022. 2.2 We want to speed up the pace of GitHub Enterprise Server releases, but I don't have more info to share. 2.3 We're looking at ways to not require a GHEC account or "unified" license. 2.4 The limits are much higher with the GitHub hosted runners, but this is a current limit of self-hosted runners. 3.1 It's on our backlog. No date to share. 3.2 I haven't heard this before and thanks for sharing the scenario. We'll think about it. 3.3 This will ship in October. 3.4 We're doing some performance optimizations for GHES 3.4 that should fix this. 3.5 This shipped recently - https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/ https://github.blog/changelog/2021-08-25-github-actions-redu... 3.6 We have a couple API improvements coming later this year. 3.7 We're looking into this, but no dates to share. We're dedicated to making Actions a great experience. As you would assume, I'm very excited about the future of Actions and getting feedback like this helps us make it better.

nicois 5y ago
We have recently switched to GitHub actions and in addition to the above, there are two others which impact us: Sometimes the checkout action fetches the wrong commit! The SHA in the GitHub environment variable says one commit, but the actual code is different(!). Because we don't know why this happens we basically need to do an unshallow fetch of the whole repo to be sure we have what we expect. Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it. This is essential to avoid broken workflows while scaling in. Gitlab supports this via a signal, but there is no equivalent.
- ethomson 5y ago
  > Sometimes the checkout action fetches the wrong commit! I'm a product manager at GitHub, and I'm investigating this now. Feel free to drop me an email (ethomson@github.com) if you can share more information about the workflow runs that were affected by this. > Using autoscaling self hosted runners, it is not currently possible to instruct the agent to finalise the current job but to accept none after it. We're building this functionality out right now, so that you can have self-hosted runners that will run a single job and we will let you subscribe to webhooks that will let you scale up/down your fleet of runners.

boleary-gl 5y ago

GitLab team-member here. Obviously coming in with a lot of bias, but I wanted to address how each point relates to GitLab CI/CD’s view of the world. I’m also thinking about writing a longer post with more details as I have a lot of thoughts (™) about this topic. 2.1 Caching isn’t available: GitLab has this everywhere. 2.2 GitHub Enterprise Server is behind GitHub Enterprise Cloud: GitLab ships the same code to GitLab.com as it does to our self-managed customers. This was a tough decision but has a lot of benefits...the central being feature parity and scalability for self-managed folks 2.3 Using Public GitHub.com Actions: This is a symptom more than the problem itself - relying on third-party plugins for build jobs is scary, and leads to many of the same issues we’ve seen in the Jenkins ecosystem - easy to get started, hard to maintain. 2.4 Dockerhub pull rate limiting: for self-hosted runners, you can use a registry mirror or Dependency Proxy to reduce your number of pulls from Docker Hub. The key is the entire platform has to be there to enable the right workflows. 3.1 No dropdowns for manually triggered jobs: GitLab also doesn’t have drop downs, but does have the ability to pre-fill these values. 3.2 Self-hosted runner default labels: I think this is also more of a symptom than a problem. 3.3 Being able to tag and use runners for specific tasks is key - so I understand the frustration and we’ve spent a lot of time on this. 3.4 You can’t restart a single job of a workflow: You can do this with GitLab. 3.5 Slow log output: I haven’t seen this be a problem, and is a benefit of our scalability features being built into the self-managed code. 3.6 You can’t have actions that call other actions: There are lots of ways to relate pipelines (parent/child, triggers. etc.) in GitLab. 3.7 Metrics and observability: The GitLab runner has Prometheus build in, and the dashboards we use to manage GitLab.com are partially public: https://dashboards.gitlab.com https://dashboards.gitlab.com 3.8 Workflow YAML Syntax can confusing: This can be really hard to get right. I learned to stop worrying and love the YAML long ago, and I know we’ve got through a lot of iterations to try and get this right. I'd love to know where folks think I got this assessment wrong. And is there value in writing more about it? (edited for line spacing)

noptd 5y ago
>2.2 GitHub Enterprise Server is behind GitHub Enterprise Cloud: GitLab ships the same code to GitLab.com as it does to our self-managed customers. Kudos to Gitlab for not utilizing the GHE model in order to convert users to their hosted solutions.

thedougd 5y ago

Surprised there's no mention of the inability to view logs generated before the page was loaded until the job completes. This one drives me crazy when I have long running, silent activity. My wishlist item would be more variants of Windows server versions so that we could build Windows containers for more versions of Windows. I realize the fault lies with Windows containers pinning the container base version to the host version, but I'm still stuck with the burden. I think GitHub Actions got the model correct, using everything as events to trigger any number of workflows. This is far simpler to maintain than a single workflow with conditionals and wait states that you see with other systems.

Aeolun 5y ago

I was planning to migrate to Github from Gitlab, but hearing about all these missing basic features (no caching, no restarting individual jobs?) I think I’ll stick with Gitlab for the forseeable future.

WillDaSilva 5y ago
Why were you planning to migrate in the first place?
- Aeolun 5y ago
  Gitlab isn’t supported by the enterprise, so I basically have to maintsin the whole thing myself. We also don’t have a license, so it’s missing some much needed features. Apparently it’s still better than Github though xD

gitgud 5y ago

> For the vast majority of use cases, the YAML syntax is sane and is similar to other CI systems. It gets super clumsy when you want to assign an output of a step to a variable that you can refer to later. The verbosity of accessing output has the added benefit of making it much clearer that the 2 workflow steps are tightly interdependent on each other.

billsdatabikes 5y ago

We use GitHub Actions and we like it. Though the current version feels like a very lukewarm compromise from the original one they had before the Microsoft acquisition. I am rather disappointed to see it become "just another CI" when the original release was so much more. I know, I know, I know, Microsoft bought GitHub so what should I expect, but either way, disappointed when the original had such amazing potential.

smcleod 5y ago

I've been testing out Github Actions for a few weeks now, for the most part I really like it, there are a few features missing but I think the fundamentals of the product are solid with the public catalogue of actions being the killer feature. The biggest issue I have is around self-hosted runners. 1. There's no official auto-scaling runner option, so even if you're paying Github (aka Microsoft) for Enterprise - they're not going to support your auto-scaling EKS/GKE/EC2/whatever runners. 2. You can't register self-hosted runners without a Personal Access Token - the key word being _Personal_. Your automation code for provisioning runners should not rely on an individuals Github access token just to register, they need to have a system like GitLab has where you can generate a registration token per-organisation/team/repo that allows you to programmatically register runners.

eberkund 5y ago
I think I recall seeing something on the GitHub forums where the recommended approach was the create a separate service user account and all your non-personal-PATs with that account.
- zeisss 5y ago
  Which means you have to pay more money. If we want fine-tune the permissions of each repo we have to create one user per repository.
147 5y ago
We use https://github.com/actions-runner-controller/actions-runner-controller https://github.com/actions-runner-controller/actions-runner-... to auto scale on EKS. It also allows you to use a GitHub App for the runner registration instead of personal access tokens. Also it seems like the project is receives support from GitHub because they’re getting early access to test out features.
- smcleod 5y ago
  Thanks, I had a play with this and submitted a PR to add some configuration options to the CRD. It seemed decent but I hit two problems: 1. I didn't manage to get autoscaling to work - I suspect my helm templates might have been incorrect. 2. Docker-in-Docker (DIND) I know works, but one of the clients I'm working with has switched to containerd and the controller got a little confused by Docker-in-Containerd (DINC - you heard it here first!), I know really they should be using Kaniko/Buildah etc... but their devs aren't ready to make the change yet.

Waterluvian 5y ago

I’ve been loving GH Actions a ton but one issue I ran into is that I have a couple tests that get run and then if it’s a merge to prod branch, an image build action. The problem is that I cannot make the image build action happen IFF both testing actions pass. I had to combine all three actions into one. It works. But now there’s a “skipped” step that’s skipped 99% of the time and makes no sense for a lot of PRs. It also means I have a Frankenstein monster action that does three long lists of very different things. All just so I can make 3 depend on 1 and 2. The other problem is that to develop and Test an action, I have to just push to origin a thousand times. My kingdom for a CI system that _trivially_ enables me to install a single one-liner program that lets me locally test my actions at near 1:1 compatibility.

Dowwie 5y ago

I've yet to come across a complete tutorial for setting up cached docker builds (one involving a rust compilation) within a Github Actions workflow. I've been figuring it out from pieces of info scattered across blog posts and github repos. How is anyone managing this today? How/where are you persisting cached objects?