7 ms·

Is it just me or are these happening a lot more since Microsoft? We unfortunately made the call to go all in in actions; and it really makes me miss gitlab...

Is it just me or are these happening a lot more since Microsoft? We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... The runners don’t even speak kubernetes properly; all your steps run inside the same container as the runner process...

thatwasunusual 6y ago
This has been discussed before.[0] The short answer seems to be "yes, but it might have its natural reasons." [0] https://news.ycombinator.com/item?id=23676199 https://news.ycombinator.com/item?id=23676199
wp381640 6y ago
They've been growing like crazy the past 12 months - I think they've been doing pretty well based on some of the numbers I've heard.
WnZ39p0Dgydaz1 6y ago
I'm not sure if it's since Microsoft but this has been happening A LOT. I've experienced roughly 5-10 outages over the past few months. Most were minor, but they still interrupted our development workflows for several hours.
tester34 6y ago
If you're trying to blame MS, then I don't think so - as far as I've heard Github works independently from MS
dustinmoris 6y ago
I think they migrated the .NET code from Azure DevOps to GitHub Actions a while ago. It was open sourced somewhere but I cannot find the repo now. Not sure if they have made the repo private again but Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. EDIT: Here's the source: https://github.com/actions/runner https://github.com/actions/runner This code is identical to the runner code for Azure DevOps. Also the repo history only goes back a little while, so you know that this wasn't the original code and they migrated the code from Azure to GitHub and then made the C# code public. I highly doubt that GitHub and Azure have independently written the exact same code in C#, especially since GitHub didn't even use C# before Microsoft's acquisition. Seems things have hugely degraded since then. I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. Wait for the next big Microsoft conference when MSFT will shout through megaphones how .NET and C# has been powering millions of customers on GitHub and been a huge "success". So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P
- cyberpunk 6y ago
  The self hosted runner is definitely using some .net stuff, you get a load of .net related errors if you try and run it on a non-glibc container (e.g alpine w/musl). shudder
  - qayxc 6y ago
    What's wrong with .net? musl also gives a lot of errors with other frameworks that aren't supported, yet no one bats an eye in those cases. What's special about .net? Is it just because it's MS or what?
- Hawxy 6y ago
  This has been public knowledge for years. Github Actions has been based on Azure Devops Pipelines since mid-2019 when it became a full CI/CD solution instead of just performing task automation. https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners https://docs.github.com/en/actions/reference/specifications-... > The GitHub-hosted runner application is a fork of the Azure Pipelines Agent. There was a related discussion on reddit at the time: https://www.reddit.com/r/programming/comments/cnoq0q/github_actions_now_supports_cicd_free_for_public/ewd8xyi/?context=3 https://www.reddit.com/r/programming/comments/cnoq0q/github_... It allowed Github to get spun up quickly with a competitor for Gitlab's CI/CD, and the community was asking for actions to become a CI/CD solution, so taking advantage of pre-existing infrastructure seems like a win-win for everyone. I don't see why they'd bother going to the effort of reinventing the wheel.
- throwaway5842 6y ago
  >Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. Github has had availability issues for about a decade now, but I'm sure it's Microsoft's fault that their MySQL clusters have high replication times[0]. > I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. This is not true, no code from GitHub has been "replaced" with .Net since Github never had a CI solution. Github was and still is a Ruby shop running MySQL. My understanding is that large parts of the actions codebase is still written in Ruby, but under the hood the CI jobs themselves run using the Azure Devops Pipelines infrastructure. In general blaming uptime on a programming language is the wrong way to think about it. Outside framework level failures like GC memory leaks it shouldn't really matter what programming language you use when it comes to uptime. Things like architecture, database availability, caching, capacity planning, and redundancy are far more operationally important. >So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P What? C# is consistently in the top 5 most used programming languages in the world in most surveys[1][2]. Why would they care about adding some marginal new project when it's used in almost every Microsoft product? [0]https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... [1]https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ [2]https://pypl.github.io/PYPL.html https://pypl.github.io/PYPL.html
dkarp 6y ago
Anecdotal, but where I worked ~5 years ago experienced so many github outages that they started maintaining a gitlab mirror. The outages had blocked enough deployments that it was worth setting up some backup.
akhong 6y ago
But I believe GitLab was down as well several hours ago. I think it’s because of free CI minutes resetting at the start of the month.
rvz 6y ago
> Is it just me or are these happening a lot more since Microsoft? The hard truth is yes. I have been saying it since the start of it all, Last time it happened was just 20 days ago. [0] > We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... Going all in on something without a backup plan sounds like a dangerous decision to do for critical deployments or serious bug fixes that must be pushed immediately. I really wouldn't want to totally lose control by going all in on GitHub Actions and have no backup or secondary measure whatsoever to urgently fix such serious issues. Instead you have to wait for GitHub to go back online or worse contact the CEO of GitHub to sort it out. Again. No thanks and most certainly no deal. [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426
jillesvangurp 6y ago
Actions were added after the acquisition. MS definitely has added a lot of features since then. So, that explains the complexity increase. That and code spaces is quite a complex thing to run for millions of developers. I think their scale is quite a bit more massive than what Gitlab is doing. And it seems a lot of OSS projects use it as well. I actually like Github actions. I also like that there's a rapidly growing number of pre-fab actions in the ecosystem. Makes creating builds a lot easier. If running in the same container is a problem for you, maybe breakup your workflows a bit into several jobs? I think you can actually chain them or run them concurrently. Also, I haven't played with this but you can plug in your own runner.
wdb 6y ago
Gitlab runners have been having degraded performance since Thursday

thatwasunusual 6y ago

This has been discussed before.[0] The short answer seems to be "yes, but it might have its natural reasons." [0] https://news.ycombinator.com/item?id=23676199 https://news.ycombinator.com/item?id=23676199

wp381640 6y ago

They've been growing like crazy the past 12 months - I think they've been doing pretty well based on some of the numbers I've heard.

WnZ39p0Dgydaz1 6y ago

I'm not sure if it's since Microsoft but this has been happening A LOT. I've experienced roughly 5-10 outages over the past few months. Most were minor, but they still interrupted our development workflows for several hours.

tester34 6y ago

If you're trying to blame MS, then I don't think so - as far as I've heard Github works independently from MS

dustinmoris 6y ago

I think they migrated the .NET code from Azure DevOps to GitHub Actions a while ago. It was open sourced somewhere but I cannot find the repo now. Not sure if they have made the repo private again but Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. EDIT: Here's the source: https://github.com/actions/runner https://github.com/actions/runner This code is identical to the runner code for Azure DevOps. Also the repo history only goes back a little while, so you know that this wasn't the original code and they migrated the code from Azure to GitHub and then made the C# code public. I highly doubt that GitHub and Azure have independently written the exact same code in C#, especially since GitHub didn't even use C# before Microsoft's acquisition. Seems things have hugely degraded since then. I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. Wait for the next big Microsoft conference when MSFT will shout through megaphones how .NET and C# has been powering millions of customers on GitHub and been a huge "success". So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P

cyberpunk 6y ago
The self hosted runner is definitely using some .net stuff, you get a load of .net related errors if you try and run it on a non-glibc container (e.g alpine w/musl). shudder
- qayxc 6y ago
  What's wrong with .net? musl also gives a lot of errors with other frameworks that aren't supported, yet no one bats an eye in those cases. What's special about .net? Is it just because it's MS or what?
Hawxy 6y ago
This has been public knowledge for years. Github Actions has been based on Azure Devops Pipelines since mid-2019 when it became a full CI/CD solution instead of just performing task automation. https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners https://docs.github.com/en/actions/reference/specifications-... > The GitHub-hosted runner application is a fork of the Azure Pipelines Agent. There was a related discussion on reddit at the time: https://www.reddit.com/r/programming/comments/cnoq0q/github_actions_now_supports_cicd_free_for_public/ewd8xyi/?context=3 https://www.reddit.com/r/programming/comments/cnoq0q/github_... It allowed Github to get spun up quickly with a competitor for Gitlab's CI/CD, and the community was asking for actions to become a CI/CD solution, so taking advantage of pre-existing infrastructure seems like a win-win for everyone. I don't see why they'd bother going to the effort of reinventing the wheel.
throwaway5842 6y ago
>Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure. Github has had availability issues for about a decade now, but I'm sure it's Microsoft's fault that their MySQL clusters have high replication times[0]. > I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#. This is not true, no code from GitHub has been "replaced" with .Net since Github never had a CI solution. Github was and still is a Ruby shop running MySQL. My understanding is that large parts of the actions codebase is still written in Ruby, but under the hood the CI jobs themselves run using the Azure Devops Pipelines infrastructure. In general blaming uptime on a programming language is the wrong way to think about it. Outside framework level failures like GC memory leaks it shouldn't really matter what programming language you use when it comes to uptime. Things like architecture, database availability, caching, capacity planning, and redundancy are far more operationally important. >So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P What? C# is consistently in the top 5 most used programming languages in the world in most surveys[1][2]. Why would they care about adding some marginal new project when it's used in almost every Microsoft product? [0]https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... [1]https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ [2]https://pypl.github.io/PYPL.html https://pypl.github.io/PYPL.html

dkarp 6y ago

Anecdotal, but where I worked ~5 years ago experienced so many github outages that they started maintaining a gitlab mirror. The outages had blocked enough deployments that it was worth setting up some backup.

akhong 6y ago

But I believe GitLab was down as well several hours ago. I think it’s because of free CI minutes resetting at the start of the month.

rvz 6y ago

> Is it just me or are these happening a lot more since Microsoft? The hard truth is yes. I have been saying it since the start of it all, Last time it happened was just 20 days ago. [0] > We unfortunately made the call to go all in in actions; and it really makes me miss gitlab... Going all in on something without a backup plan sounds like a dangerous decision to do for critical deployments or serious bug fixes that must be pushed immediately. I really wouldn't want to totally lose control by going all in on GitHub Actions and have no backup or secondary measure whatsoever to urgently fix such serious issues. Instead you have to wait for GitHub to go back online or worse contact the CEO of GitHub to sort it out. Again. No thanks and most certainly no deal. [0] https://news.ycombinator.com/item?id=26067426 https://news.ycombinator.com/item?id=26067426

jillesvangurp 6y ago

Actions were added after the acquisition. MS definitely has added a lot of features since then. So, that explains the complexity increase. That and code spaces is quite a complex thing to run for millions of developers. I think their scale is quite a bit more massive than what Gitlab is doing. And it seems a lot of OSS projects use it as well. I actually like Github actions. I also like that there's a rapidly growing number of pre-fab actions in the ecosystem. Makes creating builds a lot easier. If running in the same container is a problem for you, maybe breakup your workflows a bit into several jobs? I think you can actually chain them or run them concurrently. Also, I haven't played with this but you can plug in your own runner.

wdb 6y ago

Gitlab runners have been having degraded performance since Thursday