17 ms·

GitHub incident – now resolved

73 points by xvello 4y ago

benburwell 4y ago
I really want to like GitHub Actions. But it feels like every time I'm trying to get something done, they are broken.
- hatware 4y ago
  Wouldn't it be nice, 2 years into these hiccups, if GitHub could explain why the same problem keeps coming up? There's a point where it's funny, and we're way past that.
  - Queue29 4y ago
    They did shed some light into their internal struggles https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ https://github.blog/2022-03-23-an-update-on-recent-service-d...
    - boulos 4y ago
      That's not about Actions though, right? That's about the core mysql for repos.
      - HL33tibCe7 4y ago
        Actions relies on the main MySQL database, as do many other GitHub services
- hatf0 4y ago
  The problem that I've found is that no other CI/CD provider has feature parity to GitHub Actions & integration with Git. Sure, there's external providers (Travis-CI, Buildkite, etc) but none of them feel like they have the polish of GitHub Actions. GitLab & Azure DevOps also don't compare at all - I've migrated whole organizations off of both because they just don't feel polished / break rather frequently. So I'm personally stuck with GitHub, simply because no other company provides anything better.
  - cookiecaper 4y ago
    GitLab CI is definitely competitive with GitHub Actions. What's the specific feature set you're missing? "Polish" is pretty vague, especially in the context of a post that laments GitHub Actions' poor uptime record.
  - _joel 4y ago
    Gitlab IMHO is definitely in parity, if not better. I've had loads of success with it at a few places now, using basic stuff right up to full Autodevops with custom buildpacks for Elixir/Pheonix.
  - Tainnor 4y ago
    I have the exact opposite experience. GitHub CI is probably the worst CI I've used so far (except for custom homegrown messes), and Gitlab CI by far the best.
    - bastardoperator 4y ago
      GitLab CI isn't even on the radar. I'd take Actions or Buildkite anyday over the existing offerings. I'd argue the GitLab CI ecosystem is non-existent.
      - synu 4y ago
        Wow! I’ve heard GitLab CI is hard to learn because it’s so expansive, but never that it is non-existent. Do you mind sharing more?
        _joel 4y ago
        It's patently wrong there is no community, I've been part of it writing buildpacks.
        dnsmichi 4y ago
        GitLab team member here. I'd recommend starting with one of the workshops listed here [0] and maybe dive into more learning resources and the documentation. Our community platforms, such as the forum [2] and Stack Overflow, etc. [3] are also a great place to ask questions and collaborate on challenges. [0] https://about.gitlab.com/handbook/marketing/community-relations/developer-evangelism/projects/#workshops https://about.gitlab.com/handbook/marketing/community-relati... [1] https://about.gitlab.com/learn/ https://about.gitlab.com/learn/ [2] https://forum.gitlab.com/ https://forum.gitlab.com/ [3] https://about.gitlab.com/community/ https://about.gitlab.com/community/
rvz 4y ago
Again? Last time that happened was 9 days ago. [0] Just like I said before, at least twice a month GitHub Actions, Pages or something else goes down. Each time this happens, it makes no sense to go all in on GitHub. Perhaps companies like ARM, and projects like Wine [1], ReactOS [2], etc already went with self-hosting or have a failsafe solution to fall back on. [0] https://news.ycombinator.com/item?id=31815918 https://news.ycombinator.com/item?id=31815918 [1] https://www.phoronix.com/scan.php?page=news_item&px=Wine-GitLab-Main-Workflow https://www.phoronix.com/scan.php?page=news_item&px=Wine-Git... [2] https://github.com/reactos/reactos#code-mirrors https://github.com/reactos/reactos#code-mirrors
mfashby 4y ago
I don't really use github actions (like, ever), but it's the default setup for a terraform provider I'm working on and the very minute I queue a bunch of jobs; the system goes down. Interesting.
pilif 4y ago
When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us. Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero. I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.
- iso1631 4y ago
  > When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Oh you sweet summer child
- _joel 4y ago
  PSA: Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s). Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)
  - NhanH 4y ago
    Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).
debarshri 4y ago
I think it is not just github actions, github in general is experiencing degraded performance [1]. [1] https://www.githubstatus.com/ https://www.githubstatus.com/
eatonphil 4y ago
What's annoying about this is that the PR doesn't even say it's trying to run tests. It says everything is passing and just doesn't list the actions. For a second I thought someone must have deleted the actions yaml files. This is a dangerous failure mode. https://github.com/multiprocessio/dsq/pull/82 https://github.com/multiprocessio/dsq/pull/82 Screenshot here: https://twitter.com/phil_eaton/status/1542168020516216832 https://twitter.com/phil_eaton/status/1542168020516216832
- OJFord 4y ago
  As in you have it configured to prevent merge until they passed, they're not running at all, and it's allowing merges?
  - eatonphil 4y ago
    That's right. This button shouldn't be green. But it's not just that the actions aren't running but their service that reports that actions exist must be down too. That is a bad design. They should still report that the actions exist even if they can't run them. This PR button shouldn't be green/pressable.
    - JelteF 4y ago
      There's a way around this: Under "branch protection rules" in the settings for the repo, you can require certain "status checks" to be green before merging. This has as an added benefit that it works for allowing "auto-merge" (at the settings page of the repo). (I work at Microsoft, but not at Github)
      - eatonphil 4y ago
        Yes that is set already and that's why this is so concerning.
    - NoNameProvided 4y ago
      You can set Actions to required and then they will be displayed even when Actions is down and merge will be prevented.
synu 4y ago
I have been so happy to be on GitLab again after some time working at a company that used GitHub. The issues and epics in particular are so much better, and CI seems to be more reliable.
4khilles 4y ago
Highly recommend https://builds.sr.ht https://builds.sr.ht. I've been using for ~2 years, never had an issue.
- hatf0 4y ago
  LMAO, the entire front page of SourceHut is broken.. that's not a good look...
  - eatonphil 4y ago
    How is it broken? sr.ht? It seems to work for me...
  - gabrielgio 4y ago
    What is broken? Both https://sourcehut.org/ https://sourcehut.org/ and https://builds.sr.ht/ https://builds.sr.ht/ renders just fine to me.
    - hatf0 4y ago
      Looks like it just fixed itself. I wasn't able to see any featured projects & search for any projects for a good 5-10 minutes there.
      - deleted 4y ago
        [deleted]
  - mcint 4y ago
    Probably not the best use of archive resources, but you could store a snapshot of it, so hn commenters can believe you.
- gabrielgio 4y ago
  Just recently I found out about sourcehut. The project is refreshing. I don't know if there is something similar in the market but I enjoy how simple and straightforward to use it is.
deleted 4y ago
[deleted]
rglullis 4y ago
So, it seems like there is at least once a week a partial outage on github. For how long ate the CTOs and Engineering managers all around continue to accept this? Who shouldn't at the very least donate a bit to the various open source CI solutions, as a way to have some kind of hedge?
zippergz 4y ago
Title says "now resolved" but the link says the incident is ongoing and they "are actively working on a mitigation."
- xvello 4y ago
  Agreed, the admins re-titled it too early. Git operations and API requests are back to green, but other subsystems are still impacted.
mario_kart_snes 4y ago
My actions are simply not running.
deleted 4y ago
[deleted]
freedomben 4y ago
It seems a little early to call it "now resolved." I'm still seeing issues. If it's gotta munch through a queue or something, it would be helpful to announce that info. Edit: It's just the HN title that says "now resolved." This github status says: > We have identified the source of disruptions and are actively working on a mitigation. The systems are in recovery and services are returning to green.
fartcannon 4y ago
Not only is it not more reliable than your own server, using them launders your IP through copilot which they then sell to you and your competitors.
- oneepic 4y ago
  Source? In other places, it is stated they only use public data, such as from public repositories. See the faq at the bottom here: https://github.com/features/copilot/ https://github.com/features/copilot/
  - fartcannon 4y ago
    That's what I'm looking for. The source - but more so the exact words - that says they're legally allowed to do that. Because even if it's publicly viewable, it's still my intellectual property. I want to read the part that says, "even though it's your IP, we have the right to launder it through copilot and sell it back to you/others".
- notriddle 4y ago
  This is axe grinding. Please stop. HN has already had several threads related to Copilot. Take your grievances there, instead.
  - fartcannon 4y ago
    Hey, does your website accountkiller work for HN? They seem to be rather strict about their accounts.
zwilliamson 4y ago
Maybe they need to hire high end talent? Anyone have experience with their recruiting process?
jerryjerryjerry 4y ago
Emm, sounds not good... this kind of incidents can impact not just developers but also business insight applications being heavily analyzing github activities and projects in real-time, like this one (https://ossinsight.io/ https://ossinsight.io/). Anyway to minimize the impact of such github incident on everyone's daily projects and business?
brunojppb 4y ago
The company I currently work for is in the process of migrating out of Jenkins to Github Actions. With all problems that Github has, it has been, by far, a much better experience, even with all these issues and trade-offs, Github Actions in combination with the Github UI has been a net positive in all aspects. Jenkins is slow and a nightmare to maintain. It became a huge ball of mud that nobody wants to touch. Just keeping the lights on it's a large burden for the infrastructure team.
- rglullis 4y ago
  A bit a false dichotomy: there are other open source CI alternatives that are way more modern than Jenkins: Drone/Woodpecker, Gitlab, sourcehut all come to mind.

73 points by xvello 4y ago

benburwell 4y ago

I really want to like GitHub Actions. But it feels like every time I'm trying to get something done, they are broken.

hatware 4y ago
Wouldn't it be nice, 2 years into these hiccups, if GitHub could explain why the same problem keeps coming up? There's a point where it's funny, and we're way past that.
- Queue29 4y ago
  They did shed some light into their internal struggles https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ https://github.blog/2022-03-23-an-update-on-recent-service-d...
  - boulos 4y ago
    That's not about Actions though, right? That's about the core mysql for repos.
    - HL33tibCe7 4y ago
      Actions relies on the main MySQL database, as do many other GitHub services
hatf0 4y ago
The problem that I've found is that no other CI/CD provider has feature parity to GitHub Actions & integration with Git. Sure, there's external providers (Travis-CI, Buildkite, etc) but none of them feel like they have the polish of GitHub Actions. GitLab & Azure DevOps also don't compare at all - I've migrated whole organizations off of both because they just don't feel polished / break rather frequently. So I'm personally stuck with GitHub, simply because no other company provides anything better.
- cookiecaper 4y ago
  GitLab CI is definitely competitive with GitHub Actions. What's the specific feature set you're missing? "Polish" is pretty vague, especially in the context of a post that laments GitHub Actions' poor uptime record.
- _joel 4y ago
  Gitlab IMHO is definitely in parity, if not better. I've had loads of success with it at a few places now, using basic stuff right up to full Autodevops with custom buildpacks for Elixir/Pheonix.
- Tainnor 4y ago
  I have the exact opposite experience. GitHub CI is probably the worst CI I've used so far (except for custom homegrown messes), and Gitlab CI by far the best.
  - bastardoperator 4y ago
    GitLab CI isn't even on the radar. I'd take Actions or Buildkite anyday over the existing offerings. I'd argue the GitLab CI ecosystem is non-existent.
    - synu 4y ago
      Wow! I’ve heard GitLab CI is hard to learn because it’s so expansive, but never that it is non-existent. Do you mind sharing more?
      - _joel 4y ago
        It's patently wrong there is no community, I've been part of it writing buildpacks.
      - dnsmichi 4y ago
        GitLab team member here. I'd recommend starting with one of the workshops listed here [0] and maybe dive into more learning resources and the documentation. Our community platforms, such as the forum [2] and Stack Overflow, etc. [3] are also a great place to ask questions and collaborate on challenges. [0] https://about.gitlab.com/handbook/marketing/community-relations/developer-evangelism/projects/#workshops https://about.gitlab.com/handbook/marketing/community-relati... [1] https://about.gitlab.com/learn/ https://about.gitlab.com/learn/ [2] https://forum.gitlab.com/ https://forum.gitlab.com/ [3] https://about.gitlab.com/community/ https://about.gitlab.com/community/

rvz 4y ago

Again? Last time that happened was 9 days ago. [0] Just like I said before, at least twice a month GitHub Actions, Pages or something else goes down. Each time this happens, it makes no sense to go all in on GitHub. Perhaps companies like ARM, and projects like Wine [1], ReactOS [2], etc already went with self-hosting or have a failsafe solution to fall back on. [0] https://news.ycombinator.com/item?id=31815918 https://news.ycombinator.com/item?id=31815918 [1] https://www.phoronix.com/scan.php?page=news_item&px=Wine-GitLab-Main-Workflow https://www.phoronix.com/scan.php?page=news_item&px=Wine-Git... [2] https://github.com/reactos/reactos#code-mirrors https://github.com/reactos/reactos#code-mirrors

mfashby 4y ago

I don't really use github actions (like, ever), but it's the default setup for a terraform provider I'm working on and the very minute I queue a bunch of jobs; the system goes down. Interesting.

pilif 4y ago

When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us. Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero. I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.

iso1631 4y ago
> When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Oh you sweet summer child
_joel 4y ago
PSA: Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s). Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)
- NhanH 4y ago
  Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).

debarshri 4y ago

I think it is not just github actions, github in general is experiencing degraded performance [1]. [1] https://www.githubstatus.com/ https://www.githubstatus.com/

eatonphil 4y ago

What's annoying about this is that the PR doesn't even say it's trying to run tests. It says everything is passing and just doesn't list the actions. For a second I thought someone must have deleted the actions yaml files. This is a dangerous failure mode. https://github.com/multiprocessio/dsq/pull/82 https://github.com/multiprocessio/dsq/pull/82 Screenshot here: https://twitter.com/phil_eaton/status/1542168020516216832 https://twitter.com/phil_eaton/status/1542168020516216832

OJFord 4y ago
As in you have it configured to prevent merge until they passed, they're not running at all, and it's allowing merges?
- eatonphil 4y ago
  That's right. This button shouldn't be green. But it's not just that the actions aren't running but their service that reports that actions exist must be down too. That is a bad design. They should still report that the actions exist even if they can't run them. This PR button shouldn't be green/pressable.
  - JelteF 4y ago
    There's a way around this: Under "branch protection rules" in the settings for the repo, you can require certain "status checks" to be green before merging. This has as an added benefit that it works for allowing "auto-merge" (at the settings page of the repo). (I work at Microsoft, but not at Github)
    - eatonphil 4y ago
      Yes that is set already and that's why this is so concerning.
  - NoNameProvided 4y ago
    You can set Actions to required and then they will be displayed even when Actions is down and merge will be prevented.

synu 4y ago

I have been so happy to be on GitLab again after some time working at a company that used GitHub. The issues and epics in particular are so much better, and CI seems to be more reliable.

4khilles 4y ago

Highly recommend https://builds.sr.ht https://builds.sr.ht. I've been using for ~2 years, never had an issue.

hatf0 4y ago
LMAO, the entire front page of SourceHut is broken.. that's not a good look...
- eatonphil 4y ago
  How is it broken? sr.ht? It seems to work for me...
- gabrielgio 4y ago
  What is broken? Both https://sourcehut.org/ https://sourcehut.org/ and https://builds.sr.ht/ https://builds.sr.ht/ renders just fine to me.
  - hatf0 4y ago
    Looks like it just fixed itself. I wasn't able to see any featured projects & search for any projects for a good 5-10 minutes there.
    - deleted 4y ago
      [deleted]
- mcint 4y ago
  Probably not the best use of archive resources, but you could store a snapshot of it, so hn commenters can believe you.
gabrielgio 4y ago
Just recently I found out about sourcehut. The project is refreshing. I don't know if there is something similar in the market but I enjoy how simple and straightforward to use it is.

deleted 4y ago

[deleted]

rglullis 4y ago

So, it seems like there is at least once a week a partial outage on github. For how long ate the CTOs and Engineering managers all around continue to accept this? Who shouldn't at the very least donate a bit to the various open source CI solutions, as a way to have some kind of hedge?

zippergz 4y ago

Title says "now resolved" but the link says the incident is ongoing and they "are actively working on a mitigation."

xvello 4y ago
Agreed, the admins re-titled it too early. Git operations and API requests are back to green, but other subsystems are still impacted.

mario_kart_snes 4y ago

My actions are simply not running.

deleted 4y ago

[deleted]

freedomben 4y ago

It seems a little early to call it "now resolved." I'm still seeing issues. If it's gotta munch through a queue or something, it would be helpful to announce that info. Edit: It's just the HN title that says "now resolved." This github status says: > We have identified the source of disruptions and are actively working on a mitigation. The systems are in recovery and services are returning to green.

fartcannon 4y ago

Not only is it not more reliable than your own server, using them launders your IP through copilot which they then sell to you and your competitors.

oneepic 4y ago
Source? In other places, it is stated they only use public data, such as from public repositories. See the faq at the bottom here: https://github.com/features/copilot/ https://github.com/features/copilot/
- fartcannon 4y ago
  That's what I'm looking for. The source - but more so the exact words - that says they're legally allowed to do that. Because even if it's publicly viewable, it's still my intellectual property. I want to read the part that says, "even though it's your IP, we have the right to launder it through copilot and sell it back to you/others".
notriddle 4y ago
This is axe grinding. Please stop. HN has already had several threads related to Copilot. Take your grievances there, instead.
- fartcannon 4y ago
  Hey, does your website accountkiller work for HN? They seem to be rather strict about their accounts.

zwilliamson 4y ago

Maybe they need to hire high end talent? Anyone have experience with their recruiting process?

jerryjerryjerry 4y ago

Emm, sounds not good... this kind of incidents can impact not just developers but also business insight applications being heavily analyzing github activities and projects in real-time, like this one (https://ossinsight.io/ https://ossinsight.io/). Anyway to minimize the impact of such github incident on everyone's daily projects and business?

brunojppb 4y ago

The company I currently work for is in the process of migrating out of Jenkins to Github Actions. With all problems that Github has, it has been, by far, a much better experience, even with all these issues and trade-offs, Github Actions in combination with the Github UI has been a net positive in all aspects. Jenkins is slow and a nightmare to maintain. It became a huge ball of mud that nobody wants to touch. Just keeping the lights on it's a large burden for the infrastructure team.

rglullis 4y ago
A bit a false dichotomy: there are other open source CI alternatives that are way more modern than Jenkins: Drone/Woodpecker, Gitlab, sourcehut all come to mind.