4 ms·

Nothing is pissing me off more than GitHub's stability going down the tubes RIGHT as work is migrating everything, and I mean everything, from CircleCI to GH.

Nothing is pissing me off more than GitHub's stability going down the tubes RIGHT as work is migrating everything, and I mean everything, from CircleCI to GH. The wildest thing is that Azure Repos/Pipelines was better than this. Their one caveat is also that they are still migrating it to Azure infra, so it's possible that's still in a one foot in one foot out kinda scenario, from what I've heard. But, this isn't inspiring confidence.

gonzo41 5mo ago
Mee too. We just did a very similar migration at work it's incredibly frustrating, I've got all my CI ported over and now this. MSFT should just create slophub.com they'd make money im sure.
- Mashimo 5mo ago
  Honest question, why are companies interested in hosting on github? As a private person I use it too as a free hoster, but from work I mainly know self hosted instances of jenkins and TeamCity.
  - ellisv 5mo ago
    Most developers have experience using GitHub. The UI and concepts are familiar. The friction for adopting features like Actions is relatively low.
    - not_ai 5mo ago
      At one point it was also used as signaling that a company was “modern.”
    - Mashimo 5mo ago
      > The UI and concepts are familiar. I guess, but it's not like you can't learn how to create a pullrequest on bitbucket or how to create an issue on jira as well within a work day? That seems like the smallest thing when switching to a new company. > The friction for adopting features like Actions is relatively low. Yeah, I know almost nothing about the CI integration and actions when it comes to Github. Will look into it. Thank you.
  - stephenlf 5mo ago
    Onboarding construct workers is super easy.
  - embedding-shape 5mo ago
    > Honest question, why are companies interested in hosting on github? Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance. With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves. But, I've literally heard those two arguments for "why GitHub" countless of times over the years.
    - Mashimo 5mo ago
      Oh FOSS projects I totally understand. It's where I go to too. But closed source companies surly don't need to establish a community?
      - embedding-shape 5mo ago
        Go with the flow, don't rock the boat and use what developers already know, are probably the most cited reasons I've heard. I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.
        Esophagus4 5mo ago
        I’ve argued the opposite most of the time in build vs buy. Buy in almost every case unless it’s a real competitive advantage to you. I know developers love to build, but do you think: 1) self-hosting git provides any competitive edge to the business over letting someone manage it? 2) it provides so much value that you’re willing to fund engineers to build, secure, support this on an ongoing basis? I’ve found the answer to those is No in both cases. The same reason you wouldn’t build your own internal chat tool, you’d use Slack. And you wouldn’t bother self-hosting your own Jira or documentation. Code hosting is code hosting, there’s no difference where it's hosted. There’s no slowdown in delivery with using GitHub - their March uptime was 99.5% which annoys some commenters but it’s fine. That’s 45 minutes downtime per month which is tolerable. You would spend way more effort and money building a jenky self-hosted solution to end up with a worse result.
  - ryandrake 5mo ago
    I don’t know why you would even really need hosted git or why you’d be affected by its downtime. Git is decentralized by design. One node going down should not stop development. You don’t need a “central hub” to keep working. I guess it’s all the other non-git stuff like issue tracking and other (unfortunately) centralized products on GitHub that causes disruption when they go down. Weird how GitHub built itself around a distributed VC system and then made all its other services centralized.
    - Mashimo 5mo ago
      > I guess it’s all the other non-git stuff Yes, you want to run automated builds, unit test, end to end test, UI tests, make it easy for testers to deploy specific versions / tags to internal server. Also kick off builds for iOS on mac computers. We use Teamcity for that. Tracking of issues, feature and epics. Maybe also knowledge base / wiki. We use Jira. And pull requests. Bitbucket.
  - xp84 5mo ago
    I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories and maybe for CI too. This is like asking “Why are companies interested in using AWS?” When one firm is so dominant for so long, the question is more like “Why shouldn’t we just use GitHub like 80% of software companies do?” The issues they’ve had are almost all very recent. Very few companies have reevaluated that decision, because moving a big and well-integrated part of infrastructure is a huge project that delivers no value to the business. Speculating that you’ll have fewer development-slowing outages is not the most convincing when asking for the budget to do this. Plus, self-hosted isn’t necessarily going to have better uptime - mistakes happen. I think before Actions, it would have been a lot easier to migrate off GH though. You’d just need to change a lot of repo URLs and find a way to set up webhooks from the new place to poke CI. Now with Actions, a lot lives in GH and in a proprietary flavor that doesn’t just ‘lift and shift.’
    - Mashimo 5mo ago
      > I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories Maybe, but I never heard about any company using github for internal projects in my real life. For me it was always to go to for open source projects. Then again it's not a topic that often comes up in my developer circles.
      - jcgrillo 5mo ago
        More than half the companies I have worked for use Github. The others used Atlassian tools which were at least as bad from a reliability perspective and much less nice to use (IMO).
        xp84 5mo ago
        For me it’s been every company since 2010.
      - taormina 5mo ago
        I think I worked at one company that used BitBucket instead of GitHub but GitHub has been the main place for the internal company repos everywhere else I’ve worked. GitHub is quite popular for any sort of git hosting.
- deleted 5mo ago
  [deleted]
sdevonoes 5mo ago
Why do you care about github? It’s Just another corporation doing what they know best: harvesting money. The software ecosystem can live without github just fine
biglyburrito 5mo ago
I'm on the other side of the fence. We're just about done migrating from GitHub to GitLab (self-hosted) and it's been refreshing to DGAF about any of the GH outages I read about.
- embedding-shape 5mo ago
  Similar boat myself too, finished moving all important stuff from GitHub to self-hosted Forgejo with cross-platform builds. Not only do I avoid all the downtime stuff, but E2E builds also takes ~20% of the completion time it used to take, since now my runners have dedicated hardware hosted at home.
  - rmaus 5mo ago
    To maintain a fair comparison, GitHub has supported self-hosted runners for several years (maybe that doesn’t work for your specific usage, for whatever reason).
    - embedding-shape 5mo ago
      > To maintain a fair comparison, GitHub has supported self-hosted runners for several years Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work. The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.
- anilakar 5mo ago
  We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.
  - motbus3 5mo ago
nicoburns 5mo ago
They're claiming a huge increase in traffic due to vibe coded projects. It might just be an excuse, but it certainly seems plausible to me.
- twoodfin 5mo ago
  I’d be shocked if this wasn’t the reason.
- AznHisoka 5mo ago
  Yep, definitely more traffic and also more new Github repos being created, with a pretty huge spike the last 2 months [1] [1] https://bloomberry.com/data/github/ https://bloomberry.com/data/github/
- motbus3 5mo ago
  Could be. But 99% of the repos are static garbage with no PR nor actions. They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos. It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch. This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened
  - parthdesai 5mo ago
    Serious question, have you been part of an org that had to scale orders of magnitude very quickly? Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.
    - owebmaster 5mo ago
      > you have to redesign it from the first principles And that start by layoffing your best engineers, I guess
Xmd5a 5mo ago
Artifacts - C'mon Wit Da Git Down https://www.youtube.com/watch?v=Js_Y_q-IkYo https://www.youtube.com/watch?v=Js_Y_q-IkYo
giwook 5mo ago
Github uptime down to 86% according to https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ (not my website)
Muromec 5mo ago
Azure repos are kinda fine. It's really basic and there is nothing to break. I actually really really like their ticketing thingy for the same reason. It has the necessary stuff and the management types can't add a million of fields to it and annoy me with reporting, burndown charts or what not.
- stackskipton 5mo ago
  Yea, I have Azure DevOps with free action minutes and I’ve started using it a ton more since it avoids all GH outages.
- Quothling 5mo ago
  It has an annoying bug where approving PR's from the cli won't delete branches when you squash commit, while clicking the button in the UI does it perfectly fine. It's been a bug for a while (as in several years), and if you find something like that, don't expect it to ever be fixed. As a whole it's not a bad tool though. As you say it's limited, but that can be both good and bad.
rufasterisco 5mo ago
i might be connecting unrelated dots, yet when i read "migration to Azure" this came back to me https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242 https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize...
Insanity 5mo ago
If you’re making this change now, I wonder how the technical leadership evaluated GitHub and its competitors.. and then still landed on GH. What made it better than e.g GitLab?
throwaway613746 5mo ago
[dead]
ori_b 5mo ago
You can cancel the migration, no need for sunk cost fallacy.
- bisby 5mo ago
  "You" can't necessarily do anything (you would be making a lot of assumptions about the influence this person has over the decision making process). "Someone" can cancel the migration. "Someone" just won't.
  - ori_b 5mo ago
    https://en.wikipedia.org/wiki/Generic_you https://en.wikipedia.org/wiki/Generic_you
dwedge 5mo ago
Two weeks ago I had a commission to explore migrating from selfhosted gitlab to github for better AI integration. Last night that project was cancelled due to github outages and we're going to upgrade the self hosted server instead. I'd be tempted to use something like forgejo but there are a dozen devs and honestly I've only ever used it solo.
- whstl 5mo ago
  I would try to sell it internally. The interface is not that different, and I had good experiences myself stability-wise.
- olafmol 5mo ago
  Out of interest: what does "better AI integration" mean? Any specific functional or non-functional requirements?
  - dwedge 5mo ago
    I didn't challenge them on this but it's because of Claude integrations with github. I'm not sure what that gives them over just running it against the codebase, but I didn't want to lose the opportunity to finally move them from that EoL server
jeffwask 5mo ago
We ended up an Azure Pipelines kinda by default because it was there and mostly paid for with the intention of later migrating, but it's been fine. Boring but stable and functional.
olafmol 5mo ago
Out of interest: what VCS where you using with CircleCI? (as CircleCI is VCS agnostic)
incomplete 5mo ago
i did the circleci --> github actions migration for my job 1.5 years ago, and things seemed great... at first. at the time, we'd been dealing w/circleci's semi-regular (but thankfully short) outages for over two years, and we were excited to move to a more stable system. now i'm considering deploying jenkins.
- mitchjj 5mo ago
  Apologies for shameless plug... have you considered Buildkite? Our statuspage is a sea of green even as we hit 1.3b minutes/wk (GHA sits at 2.1/wk now). Much less maintenance overhead than Jenkins, more dynamic featureset. The trial is all-access, unlocks the full product and can be extended past 30 days. Real human eng on standby throughout.
- ttrahan2 5mo ago
  From CircleCI here. A big effort and investment went into resolving those outage issues you're referring to. Results have been stellar for a while now. Here's the latest: https://status.circleci.com https://status.circleci.com
nunez 5mo ago
> The wildest thing is that Azure Repos/Pipelines was better than this. Wow; that's extremely damning because that service was garbage IME.

gonzo41 5mo ago

Mee too. We just did a very similar migration at work it's incredibly frustrating, I've got all my CI ported over and now this. MSFT should just create slophub.com they'd make money im sure.

Mashimo 5mo ago
Honest question, why are companies interested in hosting on github? As a private person I use it too as a free hoster, but from work I mainly know self hosted instances of jenkins and TeamCity.
- ellisv 5mo ago
  Most developers have experience using GitHub. The UI and concepts are familiar. The friction for adopting features like Actions is relatively low.
  - not_ai 5mo ago
    At one point it was also used as signaling that a company was “modern.”
  - Mashimo 5mo ago
    > The UI and concepts are familiar. I guess, but it's not like you can't learn how to create a pullrequest on bitbucket or how to create an issue on jira as well within a work day? That seems like the smallest thing when switching to a new company. > The friction for adopting features like Actions is relatively low. Yeah, I know almost nothing about the CI integration and actions when it comes to Github. Will look into it. Thank you.
- stephenlf 5mo ago
  Onboarding construct workers is super easy.
- embedding-shape 5mo ago
  > Honest question, why are companies interested in hosting on github? Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance. With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves. But, I've literally heard those two arguments for "why GitHub" countless of times over the years.
  - Mashimo 5mo ago
    Oh FOSS projects I totally understand. It's where I go to too. But closed source companies surly don't need to establish a community?
    - embedding-shape 5mo ago
      Go with the flow, don't rock the boat and use what developers already know, are probably the most cited reasons I've heard. I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.
      - Esophagus4 5mo ago
        I’ve argued the opposite most of the time in build vs buy. Buy in almost every case unless it’s a real competitive advantage to you. I know developers love to build, but do you think: 1) self-hosting git provides any competitive edge to the business over letting someone manage it? 2) it provides so much value that you’re willing to fund engineers to build, secure, support this on an ongoing basis? I’ve found the answer to those is No in both cases. The same reason you wouldn’t build your own internal chat tool, you’d use Slack. And you wouldn’t bother self-hosting your own Jira or documentation. Code hosting is code hosting, there’s no difference where it's hosted. There’s no slowdown in delivery with using GitHub - their March uptime was 99.5% which annoys some commenters but it’s fine. That’s 45 minutes downtime per month which is tolerable. You would spend way more effort and money building a jenky self-hosted solution to end up with a worse result.
- ryandrake 5mo ago
  I don’t know why you would even really need hosted git or why you’d be affected by its downtime. Git is decentralized by design. One node going down should not stop development. You don’t need a “central hub” to keep working. I guess it’s all the other non-git stuff like issue tracking and other (unfortunately) centralized products on GitHub that causes disruption when they go down. Weird how GitHub built itself around a distributed VC system and then made all its other services centralized.
  - Mashimo 5mo ago
    > I guess it’s all the other non-git stuff Yes, you want to run automated builds, unit test, end to end test, UI tests, make it easy for testers to deploy specific versions / tags to internal server. Also kick off builds for iOS on mac computers. We use Teamcity for that. Tracking of issues, feature and epics. Maybe also knowledge base / wiki. We use Jira. And pull requests. Bitbucket.
- xp84 5mo ago
  I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories and maybe for CI too. This is like asking “Why are companies interested in using AWS?” When one firm is so dominant for so long, the question is more like “Why shouldn’t we just use GitHub like 80% of software companies do?” The issues they’ve had are almost all very recent. Very few companies have reevaluated that decision, because moving a big and well-integrated part of infrastructure is a huge project that delivers no value to the business. Speculating that you’ll have fewer development-slowing outages is not the most convincing when asking for the budget to do this. Plus, self-hosted isn’t necessarily going to have better uptime - mistakes happen. I think before Actions, it would have been a lot easier to migrate off GH though. You’d just need to change a lot of repo URLs and find a way to set up webhooks from the new place to poke CI. Now with Actions, a lot lives in GH and in a proprietary flavor that doesn’t just ‘lift and shift.’
  - Mashimo 5mo ago
    > I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories Maybe, but I never heard about any company using github for internal projects in my real life. For me it was always to go to for open source projects. Then again it's not a topic that often comes up in my developer circles.
    - jcgrillo 5mo ago
      More than half the companies I have worked for use Github. The others used Atlassian tools which were at least as bad from a reliability perspective and much less nice to use (IMO).
      - xp84 5mo ago
        For me it’s been every company since 2010.
    - taormina 5mo ago
      I think I worked at one company that used BitBucket instead of GitHub but GitHub has been the main place for the internal company repos everywhere else I’ve worked. GitHub is quite popular for any sort of git hosting.
deleted 5mo ago
[deleted]

sdevonoes 5mo ago

Why do you care about github? It’s Just another corporation doing what they know best: harvesting money. The software ecosystem can live without github just fine

biglyburrito 5mo ago

I'm on the other side of the fence. We're just about done migrating from GitHub to GitLab (self-hosted) and it's been refreshing to DGAF about any of the GH outages I read about.

embedding-shape 5mo ago
Similar boat myself too, finished moving all important stuff from GitHub to self-hosted Forgejo with cross-platform builds. Not only do I avoid all the downtime stuff, but E2E builds also takes ~20% of the completion time it used to take, since now my runners have dedicated hardware hosted at home.
- rmaus 5mo ago
  To maintain a fair comparison, GitHub has supported self-hosted runners for several years (maybe that doesn’t work for your specific usage, for whatever reason).
  - embedding-shape 5mo ago
    > To maintain a fair comparison, GitHub has supported self-hosted runners for several years Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work. The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.
anilakar 5mo ago
We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.
- motbus3 5mo ago

nicoburns 5mo ago

They're claiming a huge increase in traffic due to vibe coded projects. It might just be an excuse, but it certainly seems plausible to me.

twoodfin 5mo ago
I’d be shocked if this wasn’t the reason.
AznHisoka 5mo ago
Yep, definitely more traffic and also more new Github repos being created, with a pretty huge spike the last 2 months [1] [1] https://bloomberry.com/data/github/ https://bloomberry.com/data/github/
motbus3 5mo ago
Could be. But 99% of the repos are static garbage with no PR nor actions. They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos. It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch. This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened
- parthdesai 5mo ago
  Serious question, have you been part of an org that had to scale orders of magnitude very quickly? Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.
  - owebmaster 5mo ago
    > you have to redesign it from the first principles And that start by layoffing your best engineers, I guess

Xmd5a 5mo ago

Artifacts - C'mon Wit Da Git Down https://www.youtube.com/watch?v=Js_Y_q-IkYo https://www.youtube.com/watch?v=Js_Y_q-IkYo

giwook 5mo ago

Github uptime down to 86% according to https://mrshu.github.io/github-statuses/ https://mrshu.github.io/github-statuses/ (not my website)

Muromec 5mo ago

Azure repos are kinda fine. It's really basic and there is nothing to break. I actually really really like their ticketing thingy for the same reason. It has the necessary stuff and the management types can't add a million of fields to it and annoy me with reporting, burndown charts or what not.

stackskipton 5mo ago
Yea, I have Azure DevOps with free action minutes and I’ve started using it a ton more since it avoids all GH outages.
Quothling 5mo ago
It has an annoying bug where approving PR's from the cli won't delete branches when you squash commit, while clicking the button in the UI does it perfectly fine. It's been a bug for a while (as in several years), and if you find something like that, don't expect it to ever be fixed. As a whole it's not a bad tool though. As you say it's limited, but that can be both good and bad.

rufasterisco 5mo ago

i might be connecting unrelated dots, yet when i read "migration to Azure" this came back to me https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242 https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Insanity 5mo ago

If you’re making this change now, I wonder how the technical leadership evaluated GitHub and its competitors.. and then still landed on GH. What made it better than e.g GitLab?

throwaway613746 5mo ago

[dead]

ori_b 5mo ago

You can cancel the migration, no need for sunk cost fallacy.

bisby 5mo ago
"You" can't necessarily do anything (you would be making a lot of assumptions about the influence this person has over the decision making process). "Someone" can cancel the migration. "Someone" just won't.
- ori_b 5mo ago
  https://en.wikipedia.org/wiki/Generic_you https://en.wikipedia.org/wiki/Generic_you

dwedge 5mo ago

Two weeks ago I had a commission to explore migrating from selfhosted gitlab to github for better AI integration. Last night that project was cancelled due to github outages and we're going to upgrade the self hosted server instead. I'd be tempted to use something like forgejo but there are a dozen devs and honestly I've only ever used it solo.

whstl 5mo ago
I would try to sell it internally. The interface is not that different, and I had good experiences myself stability-wise.
olafmol 5mo ago
Out of interest: what does "better AI integration" mean? Any specific functional or non-functional requirements?
- dwedge 5mo ago
  I didn't challenge them on this but it's because of Claude integrations with github. I'm not sure what that gives them over just running it against the codebase, but I didn't want to lose the opportunity to finally move them from that EoL server

jeffwask 5mo ago

We ended up an Azure Pipelines kinda by default because it was there and mostly paid for with the intention of later migrating, but it's been fine. Boring but stable and functional.

olafmol 5mo ago

Out of interest: what VCS where you using with CircleCI? (as CircleCI is VCS agnostic)

incomplete 5mo ago

i did the circleci --> github actions migration for my job 1.5 years ago, and things seemed great... at first. at the time, we'd been dealing w/circleci's semi-regular (but thankfully short) outages for over two years, and we were excited to move to a more stable system. now i'm considering deploying jenkins.

mitchjj 5mo ago
Apologies for shameless plug... have you considered Buildkite? Our statuspage is a sea of green even as we hit 1.3b minutes/wk (GHA sits at 2.1/wk now). Much less maintenance overhead than Jenkins, more dynamic featureset. The trial is all-access, unlocks the full product and can be extended past 30 days. Real human eng on standby throughout.
ttrahan2 5mo ago
From CircleCI here. A big effort and investment went into resolving those outage issues you're referring to. Results have been stellar for a while now. Here's the latest: https://status.circleci.com https://status.circleci.com

nunez 5mo ago

> The wildest thing is that Azure Repos/Pipelines was better than this. Wow; that's extremely damning because that service was garbage IME.