6 ms·

GitHub Actions is experiencing degraded performance

76 points by NiekvdMaas 6y ago

cyberpunk 6y ago
Is it just me or are these happening a lot more since Microsoft? We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... The runners don’t even speak kubernetes properly; all your steps run inside the same container as the runner process...
- thatwasunusual 6y ago
  This has been discussed before.[0] The short answer seems to be "yes, but it might have its natural reasons." [0] https://news.ycombinator.com/item?id=23676199 https://news.ycombinator.com/item?id=23676199
- wp381640 6y ago
  They've been growing like crazy the past 12 months - I think they've been doing pretty well based on some of the numbers I've heard.
- WnZ39p0Dgydaz1 6y ago
  I'm not sure if it's since Microsoft but this has been happening A LOT. I've experienced roughly 5-10 outages over the past few months. Most were minor, but they still interrupted our development workflows for several hours.
- tester34 6y ago
  If you're trying to blame MS, then I don't think so - as far as I've heard Github works independently from MS
- dustinmoris 6y ago
  I think they migrated the .NET code from Azure DevOps to GitHub Actions a while ago. It was open sourced somewhere but I cannot find the repo now. Not sure if they have made the repo private again but Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. EDIT: Here's the source: https://github.com/actions/runner https://github.com/actions/runner This code is identical to the runner code for Azure DevOps. Also the repo history only goes back a little while, so you know that this wasn't the original code and they migrated the code from Azure to GitHub and then made the C# code public. I highly doubt that GitHub and Azure have independently written the exact same code in C#, especially since GitHub didn't even use C# before Microsoft's acquisition. Seems things have hugely degraded since then. I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. Wait for the next big Microsoft conference when MSFT will shout through megaphones how .NET and C# has been powering millions of customers on GitHub and been a huge "success". So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P
  - cyberpunk 6y ago
    The self hosted runner is definitely using some .net stuff, you get a load of .net related errors if you try and run it on a non-glibc container (e.g alpine w/musl). shudder
    - qayxc 6y ago
      What's wrong with .net? musl also gives a lot of errors with other frameworks that aren't supported, yet no one bats an eye in those cases. What's special about .net? Is it just because it's MS or what?
  - Hawxy 6y ago
    This has been public knowledge for years. Github Actions has been based on Azure Devops Pipelines since mid-2019 when it became a full CI/CD solution instead of just performing task automation. https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners https://docs.github.com/en/actions/reference/specifications-... > The GitHub-hosted runner application is a fork of the Azure Pipelines Agent. There was a related discussion on reddit at the time: https://www.reddit.com/r/programming/comments/cnoq0q/github_actions_now_supports_cicd_free_for_public/ewd8xyi/?context=3 https://www.reddit.com/r/programming/comments/cnoq0q/github_... It allowed Github to get spun up quickly with a competitor for Gitlab's CI/CD, and the community was asking for actions to become a CI/CD solution, so taking advantage of pre-existing infrastructure seems like a win-win for everyone. I don't see why they'd bother going to the effort of reinventing the wheel.
  - throwaway5842 6y ago
    >Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. Github has had availability issues for about a decade now, but I'm sure it's Microsoft's fault that their MySQL clusters have high replication times[0]. > I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. This is not true, no code from GitHub has been "replaced" with .Net since Github never had a CI solution. Github was and still is a Ruby shop running MySQL. My understanding is that large parts of the actions codebase is still written in Ruby, but under the hood the CI jobs themselves run using the Azure Devops Pipelines infrastructure. In general blaming uptime on a programming language is the wrong way to think about it. Outside framework level failures like GC memory leaks it shouldn't really matter what programming language you use when it comes to uptime. Things like architecture, database availability, caching, capacity planning, and redundancy are far more operationally important. >So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P What? C# is consistently in the top 5 most used programming languages in the world in most surveys[1][2]. Why would they care about adding some marginal new project when it's used in almost every Microsoft product? [0]https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... [1]https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ [2]https://pypl.github.io/PYPL.html https://pypl.github.io/PYPL.html
- dkarp 6y ago
  Anecdotal, but where I worked ~5 years ago experienced so many github outages that they started maintaining a gitlab mirror. The outages had blocked enough deployments that it was worth setting up some backup.
- akhong 6y ago
  But I believe GitLab was down as well several hours ago. I think it’s because of free CI minutes resetting at the start of the month.
- rvz 6y ago
  > Is it just me or are these happening a lot more since Microsoft? The hard truth is yes. I have been saying it since the start of it all, Last time it happened was just 20 days ago. [0] > We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... Going all in on something without a backup plan sounds like a dangerous decision to do for critical deployments or serious bug fixes that must be pushed immediately. I really wouldn't want to totally lose control by going all in on GitHub Actions and have no backup or secondary measure whatsoever to urgently fix such serious issues. Instead you have to wait for GitHub to go back online or worse contact the CEO of GitHub to sort it out. Again. No thanks and most certainly no deal. [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426
- jillesvangurp 6y ago
  Actions were added after the acquisition. MS definitely has added a lot of features since then. So, that explains the complexity increase. That and code spaces is quite a complex thing to run for millions of developers. I think their scale is quite a bit more massive than what Gitlab is doing. And it seems a lot of OSS projects use it as well. I actually like Github actions. I also like that there's a rapidly growing number of pre-fab actions in the ecosystem. Makes creating builds a lot easier. If running in the same container is a problem for you, maybe breakup your workflows a bit into several jobs? I think you can actually chain them or run them concurrently. Also, I haven't played with this but you can plug in your own runner.
- wdb 6y ago
  Gitlab runners have been having degraded performance since Thursday
KingOfCoders 6y ago
Happend right when I added a new action for a new project - hope it wasn't me ;-)
- gostsamo 6y ago
  If it was just you, then it is not just you. Github should own the fact that they are responsible for the continued work of their infrastructure and the normal workflow of people can't be responsible for it not working.
- berkes 6y ago
  Wait, did you name that task `$(rm --rf --no-preserve-root /)`? Because if so, it was probably you :)
  - KingOfCoders 6y ago
    Ooopsi.
dochtman 6y ago
What's especially annoying is that most PRs seem to get a green checkmark for me as if checks have passed, when tests actually just haven't been started. It seems like it ought to be possible to do better than this. Right now the risk of accidentally merging a PR that hasn't run through CI is way too large.
- domoritz 6y ago
  If you have required pull request checks, the pull requests are marked as not ready yet. Plus, you get auto merge if you enable it.
leetrout 6y ago
The “subscribe for updates” functionality is nice. Do other services offer things like this for updates on incidents? That’s much better than watching Twitter or refreshing a page!
- mhitza 6y ago
  I think it would be even nicer if all these status pages had push notifications (unless it's somewhere hidden on the page). But I agree it's nicer than shuffling around different websites for the info.
rvz 6y ago
Do you think it makes sense to go all in on GitHub Actions without a backup plan involved? It's time to contact the CEO of GitHub again. Don't say I didn't warn you for months and months on end. [0] [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426
- fuzzypixelz 6y ago
  rvz_told_you_so_linked_list You sir are a legend.
- rvnx 6y ago
  It depends. In most of the scenarios, if GitHub Actions is down for an hour, even a couple of times per year, the deployment of new functionalities may be delayed a bit but that's it. If you average over the year, the productivity gain, simplicity, and price (100% free for public repos!) from using GitHub Actions is still worth it for most of the people. There could have been many uncontrollable reasons for the flow of developers to be interrupted (meeting, lunch, questions from colleagues, a pet in the office, slow internet, traffic with the car, stuck in an elevator) or just big news-related distractions (Christmas, Gamestop stock, Roblox IPO). Unless you have a company somewhat relying on GitHub Actions for critical / business-generating flow (this sounds like an unusual idea to me but ok), it's unlikely to impact the income of the company.
  - rowanseymour 6y ago
    We're all in on Github actions these days but there was a time when we used to maintain our own Jenkins instance. I have spent less time waiting for the occasional Github problem to be resolved than I did trying to keep that thing working.
    - AdamJacobMuller 6y ago
      I agree with Jenkins, but, Gitlab-CI is a joy to work with.
- chrisan 6y ago
lima 6y ago
At this point, our self-hosted CI and code review tooling has had significantly better uptime for the past year. The advantage of having someone else operate it evaporates when it is neutralized by complexity and scale. The failure modes of our own setup are much, much easier to deal with.
- growse 6y ago
  Is anyone using self-hosted runners with public git repos? Github's advice is not to do this because it effectively allows anyone who can open a PR to run untrusted code on your infra. I figure this must be an easily solved problem, but not sure how everyone else is doing it.
  - h43k3r 6y ago
    Sandboxed containers.
    - rvnx 6y ago
      In Facebook for example, in the internal network you can access a lot of machines without authentication (let's say you open a PR to one of their public repo, and this runs in their internal network), and of course the beta of Facebook.
      - jrockway 6y ago
        I don't think "self hosted" has to mean "can access some special internal network". Just put a Threadripper under your desk. Have a coworker do the same for disaster recovery / high availability :)
  - qqii 6y ago
    Wouldn't this be as simple as restricting CI until the repo owner has looked over the PR? It's moving the problem but at the end of the day the repo owner has to look over and choose to accept or reject the PR.
Shadonototro 6y ago
They switched to - Azure - dotnet Results: - Lower uptime - Degraded performance Let's not forget about the global XBOX LIVE outage from last week
- qayxc 6y ago
  So apart from the lack of evidence and data proving the point, correlation ≠ causation. Unless you can demonstrate that no other factors (increase in repos, increase in usage overall, etc.) are involved, your statement is a biased opinion, not a fact. Edit: since you opted to throw XBOXLive into the mix in an edit, what did Sony change? https://www.theverge.com/2021/2/27/22304492/sony-playstation-network-outage-saturday https://www.theverge.com/2021/2/27/22304492/sony-playstation...
  - victords 6y ago
    Not agreeing with his points, but funnily enough, I think Sony does use Azure.
    - Shadonototro 6y ago
      exactly, Sony got pressured to migrate to Azure, even for their game streaming services what's funny is they got the downtime, right after XBOX got his almost like if someone was jealous and wanted them to be "equal" but people won't like this "idea", maybe a little too much "conspiracy theory", i'll agree, my comment should get flagged, or i shouldn't have hit the "reply" button, but it's too late i guess
    - qayxc 6y ago
      That's Playstation Now, their streaming service. All I could find about PSN is that they moved away from AWS [1] in North America. [1] https://www.networkworld.com/article/2186653/sony-division-moves-some-services-from-aws-to-openstack.html https://www.networkworld.com/article/2186653/sony-division-m...

76 points by NiekvdMaas 6y ago

Is it just me or are these happening a lot more since Microsoft? We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... The runners don’t even speak kubernetes properly; all your steps run inside the same container as the runner process...

thatwasunusual 6y ago
This has been discussed before.[0] The short answer seems to be "yes, but it might have its natural reasons." [0] https://news.ycombinator.com/item?id=23676199 https://news.ycombinator.com/item?id=23676199
wp381640 6y ago
They've been growing like crazy the past 12 months - I think they've been doing pretty well based on some of the numbers I've heard.
WnZ39p0Dgydaz1 6y ago
I'm not sure if it's since Microsoft but this has been happening A LOT. I've experienced roughly 5-10 outages over the past few months. Most were minor, but they still interrupted our development workflows for several hours.
tester34 6y ago
If you're trying to blame MS, then I don't think so - as far as I've heard Github works independently from MS
dustinmoris 6y ago
I think they migrated the .NET code from Azure DevOps to GitHub Actions a while ago. It was open sourced somewhere but I cannot find the repo now. Not sure if they have made the repo private again but Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. EDIT: Here's the source: https://github.com/actions/runner https://github.com/actions/runner This code is identical to the runner code for Azure DevOps. Also the repo history only goes back a little while, so you know that this wasn't the original code and they migrated the code from Azure to GitHub and then made the C# code public. I highly doubt that GitHub and Azure have independently written the exact same code in C#, especially since GitHub didn't even use C# before Microsoft's acquisition. Seems things have hugely degraded since then. I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. Wait for the next big Microsoft conference when MSFT will shout through megaphones how .NET and C# has been powering millions of customers on GitHub and been a huge "success". So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P
- cyberpunk 6y ago
  The self hosted runner is definitely using some .net stuff, you get a load of .net related errors if you try and run it on a non-glibc container (e.g alpine w/musl). shudder
  - qayxc 6y ago
    What's wrong with .net? musl also gives a lot of errors with other frameworks that aren't supported, yet no one bats an eye in those cases. What's special about .net? Is it just because it's MS or what?
- Hawxy 6y ago
  This has been public knowledge for years. Github Actions has been based on Azure Devops Pipelines since mid-2019 when it became a full CI/CD solution instead of just performing task automation. https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners https://docs.github.com/en/actions/reference/specifications-... > The GitHub-hosted runner application is a fork of the Azure Pipelines Agent. There was a related discussion on reddit at the time: https://www.reddit.com/r/programming/comments/cnoq0q/github_actions_now_supports_cicd_free_for_public/ewd8xyi/?context=3 https://www.reddit.com/r/programming/comments/cnoq0q/github_... It allowed Github to get spun up quickly with a competitor for Gitlab's CI/CD, and the community was asking for actions to become a CI/CD solution, so taking advantage of pre-existing infrastructure seems like a win-win for everyone. I don't see why they'd bother going to the effort of reinventing the wheel.
- throwaway5842 6y ago
  >Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. Github has had availability issues for about a decade now, but I'm sure it's Microsoft's fault that their MySQL clusters have high replication times[0]. > I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. This is not true, no code from GitHub has been "replaced" with .Net since Github never had a CI solution. Github was and still is a Ruby shop running MySQL. My understanding is that large parts of the actions codebase is still written in Ruby, but under the hood the CI jobs themselves run using the Azure Devops Pipelines infrastructure. In general blaming uptime on a programming language is the wrong way to think about it. Outside framework level failures like GC memory leaks it shouldn't really matter what programming language you use when it comes to uptime. Things like architecture, database availability, caching, capacity planning, and redundancy are far more operationally important. >So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P What? C# is consistently in the top 5 most used programming languages in the world in most surveys[1][2]. Why would they care about adding some marginal new project when it's used in almost every Microsoft product? [0]https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... [1]https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ [2]https://pypl.github.io/PYPL.html https://pypl.github.io/PYPL.html
dkarp 6y ago
Anecdotal, but where I worked ~5 years ago experienced so many github outages that they started maintaining a gitlab mirror. The outages had blocked enough deployments that it was worth setting up some backup.
akhong 6y ago
But I believe GitLab was down as well several hours ago. I think it’s because of free CI minutes resetting at the start of the month.
rvz 6y ago
> Is it just me or are these happening a lot more since Microsoft? The hard truth is yes. I have been saying it since the start of it all, Last time it happened was just 20 days ago. [0] > We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... Going all in on something without a backup plan sounds like a dangerous decision to do for critical deployments or serious bug fixes that must be pushed immediately. I really wouldn't want to totally lose control by going all in on GitHub Actions and have no backup or secondary measure whatsoever to urgently fix such serious issues. Instead you have to wait for GitHub to go back online or worse contact the CEO of GitHub to sort it out. Again. No thanks and most certainly no deal. [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426
jillesvangurp 6y ago
Actions were added after the acquisition. MS definitely has added a lot of features since then. So, that explains the complexity increase. That and code spaces is quite a complex thing to run for millions of developers. I think their scale is quite a bit more massive than what Gitlab is doing. And it seems a lot of OSS projects use it as well. I actually like Github actions. I also like that there's a rapidly growing number of pre-fab actions in the ecosystem. Makes creating builds a lot easier. If running in the same container is a problem for you, maybe breakup your workflows a bit into several jobs? I think you can actually chain them or run them concurrently. Also, I haven't played with this but you can plug in your own runner.
wdb 6y ago
Gitlab runners have been having degraded performance since Thursday

KingOfCoders 6y ago

Happend right when I added a new action for a new project - hope it wasn't me ;-)

gostsamo 6y ago
If it was just you, then it is not just you. Github should own the fact that they are responsible for the continued work of their infrastructure and the normal workflow of people can't be responsible for it not working.
berkes 6y ago
Wait, did you name that task `$(rm --rf --no-preserve-root /)`? Because if so, it was probably you :)
- KingOfCoders 6y ago
  Ooopsi.

dochtman 6y ago

What's especially annoying is that most PRs seem to get a green checkmark for me as if checks have passed, when tests actually just haven't been started. It seems like it ought to be possible to do better than this. Right now the risk of accidentally merging a PR that hasn't run through CI is way too large.

domoritz 6y ago
If you have required pull request checks, the pull requests are marked as not ready yet. Plus, you get auto merge if you enable it.

leetrout 6y ago

The “subscribe for updates” functionality is nice. Do other services offer things like this for updates on incidents? That’s much better than watching Twitter or refreshing a page!

mhitza 6y ago
I think it would be even nicer if all these status pages had push notifications (unless it's somewhere hidden on the page). But I agree it's nicer than shuffling around different websites for the info.

rvz 6y ago

Do you think it makes sense to go all in on GitHub Actions without a backup plan involved? It's time to contact the CEO of GitHub again. Don't say I didn't warn you for months and months on end. [0] [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426

fuzzypixelz 6y ago
rvz_told_you_so_linked_list You sir are a legend.
rvnx 6y ago
It depends. In most of the scenarios, if GitHub Actions is down for an hour, even a couple of times per year, the deployment of new functionalities may be delayed a bit but that's it. If you average over the year, the productivity gain, simplicity, and price (100% free for public repos!) from using GitHub Actions is still worth it for most of the people. There could have been many uncontrollable reasons for the flow of developers to be interrupted (meeting, lunch, questions from colleagues, a pet in the office, slow internet, traffic with the car, stuck in an elevator) or just big news-related distractions (Christmas, Gamestop stock, Roblox IPO). Unless you have a company somewhat relying on GitHub Actions for critical / business-generating flow (this sounds like an unusual idea to me but ok), it's unlikely to impact the income of the company.
- rowanseymour 6y ago
  We're all in on Github actions these days but there was a time when we used to maintain our own Jenkins instance. I have spent less time waiting for the occasional Github problem to be resolved than I did trying to keep that thing working.
  - AdamJacobMuller 6y ago
    I agree with Jenkins, but, Gitlab-CI is a joy to work with.
chrisan 6y ago

lima 6y ago

At this point, our self-hosted CI and code review tooling has had significantly better uptime for the past year. The advantage of having someone else operate it evaporates when it is neutralized by complexity and scale. The failure modes of our own setup are much, much easier to deal with.

growse 6y ago
Is anyone using self-hosted runners with public git repos? Github's advice is not to do this because it effectively allows anyone who can open a PR to run untrusted code on your infra. I figure this must be an easily solved problem, but not sure how everyone else is doing it.
- h43k3r 6y ago
  Sandboxed containers.
  - rvnx 6y ago
    In Facebook for example, in the internal network you can access a lot of machines without authentication (let's say you open a PR to one of their public repo, and this runs in their internal network), and of course the beta of Facebook.
    - jrockway 6y ago
      I don't think "self hosted" has to mean "can access some special internal network". Just put a Threadripper under your desk. Have a coworker do the same for disaster recovery / high availability :)
- qqii 6y ago
  Wouldn't this be as simple as restricting CI until the repo owner has looked over the PR? It's moving the problem but at the end of the day the repo owner has to look over and choose to accept or reject the PR.

Shadonototro 6y ago

They switched to - Azure - dotnet Results: - Lower uptime - Degraded performance Let's not forget about the global XBOX LIVE outage from last week

qayxc 6y ago
So apart from the lack of evidence and data proving the point, correlation ≠ causation. Unless you can demonstrate that no other factors (increase in repos, increase in usage overall, etc.) are involved, your statement is a biased opinion, not a fact. Edit: since you opted to throw XBOXLive into the mix in an edit, what did Sony change? https://www.theverge.com/2021/2/27/22304492/sony-playstation-network-outage-saturday https://www.theverge.com/2021/2/27/22304492/sony-playstation...
- victords 6y ago
  Not agreeing with his points, but funnily enough, I think Sony does use Azure.
  - Shadonototro 6y ago
    exactly, Sony got pressured to migrate to Azure, even for their game streaming services what's funny is they got the downtime, right after XBOX got his almost like if someone was jealous and wanted them to be "equal" but people won't like this "idea", maybe a little too much "conspiracy theory", i'll agree, my comment should get flagged, or i shouldn't have hit the "reply" button, but it's too late i guess
  - qayxc 6y ago
    That's Playstation Now, their streaming service. All I could find about PSN is that they moved away from AWS [1] in North America. [1] https://www.networkworld.com/article/2186653/sony-division-moves-some-services-from-aws-to-openstack.html https://www.networkworld.com/article/2186653/sony-division-m...