9 ms·

GitHub degraded performance – resolved

101 points by mot2ba 5y ago

maherbeg 5y ago
I'm surprised they don't have a couple of separate clusters that they roll things out to and monitor. Seems like you could have a very stable "high paying customers" cluster that is at the very end of your deployment cycle after a ton of canary checks on the way get through.
- intunderflow 5y ago
  With Actions they do this, if you're on GitHub Enterprise and run an action it picks machines out of a special pool set aside just for enterprise customers.
contingencies 5y ago
They were down at least a solid hour last week and didn't even post to their status page. I put in a query and got no response. They then unilaterally closed the ticket asked me how my support experience was. Time to move to Gitlab. https://news.ycombinator.com/item?id=28874751 https://news.ycombinator.com/item?id=28874751
encryptluks2 5y ago
Just noticed GitHub is down for me. Can't access repos :(
judge2020 5y ago
GitHub Universe is only a week away, could be related to a bad deploy for some feature updates to-be-revealed then.
funOtter 5y ago
I was just doing some development on my GitHub Actions ... can only assume it was my fault.
- roland35 5y ago
  Should have probably left out that sudo rm -rf from your Action
mabbo 5y ago
In the middle of onboarding with a new company. We're at a critical point of training that requires GitHub. Oh boy, this is going to be a fun day.
- awestroke 5y ago
  The incident lasted less than an hour. How does this affect your whole day?
  - mabbo 5y ago
    It didn't. But at the time, the trainer was very, very worried.
- tyingq 5y ago
  Maybe you could sign up for a GitHub enterprise trial? At the least first few screens seem to be working.
bob1029 5y ago
This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically your entire business through the product. I do agree that it's ridiculous to assume that we can manage Github's software better than their own engineers, but at the same time our infrastructure has proven itself to be extremely reliable over the last 4-5 years. Even hosting GH enterprise on public AWS/Azure is more ideal in my eyes now, because I can control the physical region and tenancy. There is an Azure datacenter within 100 miles of many of our home offices and I can ensure that our Github stack spins up there. Minimizing the amount of internet you have to transit to get to your applications can sidestep a lot of this stormy public cloud/internet weather bullshit.
- andrewstuart2 5y ago
  I think you will quickly find if you're just deploying GH Enterprise on premises that it is not at all what you get from the GH Cloud offering. GHE has its own product roadmap that is quite a bit behind the cloud product, and in many cases (IMO) unacceptably so. It still doesn't support cache for runners, last I checked, though I've since moved on from the org that required me to work with GHE. I'm back to my happy place with self-hosted GitLab, and a little bit of GitHub cloud.
  - bob1029 5y ago
    > GHE has its own product roadmap that is quite a bit behind the cloud product This is exactly what we want though. We don't need the new fancy shit on a regular cadence. Issues, Code, PRs and 1 line checkbuild scripts are all we care about. Everything else is built into our software.
    - andrewstuart2 5y ago
      What I mean is that it's quite clearly not the same product as cloud. It does have roughly the same road map as cloud, just pretty far behind and/or cherry-picking some features in a different order. My experience with enterprise software is that it doesn't matter what the roadmap is as long as you get to choose if/when the benefits outweigh the risks for an update. And you usually want certain releases to get backported security updates for that same reason. You don't have to take new features and their associated bugs but you do want, and get, security fixes. That's a separate thing, because this would be the case if GHE was just "run your own copy of .com." What is really rough about GHE is that you can't choose a lot of the features or IMO baseline requirements like caching that you've probably come to expect from github.com, and may have been around for years. At least not until they can get GHE to parity with .com.
jacobrussell 5y ago
Good time for a lunch break I guess
mot2ba 5y ago
This incident has been marked as resolved. You guys could check how often Github notable incidents that really captures the audience's attention in HN [0] [1]: [0]https://news.ycombinator.com/from?site=githubstatus.com https://news.ycombinator.com/from?site=githubstatus.com [1]https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=false&query=githubstatus.com&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=fal...
- jrochkind1 5y ago
  Looks like about once every two months? Which is actually more than I expected, and seems like kind of too much.
intunderflow 5y ago
When GitHub is down we legally don't have to do any work, it's in the constitution I swear
- m_a_g 5y ago
  Best comment in this thread by far
jrochkind1 5y ago
Hmm, how do I get the Github Actions CI to run on all the already existing PR's for which it never ran? Anyone know?
- mdaniel 5y ago
  Would this do it? https://docs.github.com/en/rest/reference/actions#create-a-workflow-dispatch-event https://docs.github.com/en/rest/reference/actions#create-a-w... or, depending on the error, perhaps this? https://docs.github.com/en/rest/reference/actions#re-run-a-workflow https://docs.github.com/en/rest/reference/actions#re-run-a-w...
  - jrochkind1 5y ago
    Not sure. There was no error reported, just a "waiting" message, with no "run" created. i'm not sure I have any ID's to give to those API calls. Would have preferred something from the web UI. I just went ahead and created new commits for them all. (Create an empty commit, or --amend to re-commit the last commit with a new timestamp). Would prefer if there were an easier way to do it, but fighting with http API's I am not familiar with when they aren't immediately apparent and I'm not sure they'll work at all was not that easier way.
blondin 5y ago
i wonder if github ever considered a hybrid approach? an on-prem enterprise solution that syncs with github cloud.

101 points by mot2ba 5y ago

I'm surprised they don't have a couple of separate clusters that they roll things out to and monitor. Seems like you could have a very stable "high paying customers" cluster that is at the very end of your deployment cycle after a ton of canary checks on the way get through.

intunderflow 5y ago
With Actions they do this, if you're on GitHub Enterprise and run an action it picks machines out of a special pool set aside just for enterprise customers.

contingencies 5y ago

They were down at least a solid hour last week and didn't even post to their status page. I put in a query and got no response. They then unilaterally closed the ticket asked me how my support experience was. Time to move to Gitlab. https://news.ycombinator.com/item?id=28874751 https://news.ycombinator.com/item?id=28874751

encryptluks2 5y ago

Just noticed GitHub is down for me. Can't access repos :(

judge2020 5y ago

GitHub Universe is only a week away, could be related to a bad deploy for some feature updates to-be-revealed then.

funOtter 5y ago

I was just doing some development on my GitHub Actions ... can only assume it was my fault.

roland35 5y ago
Should have probably left out that sudo rm -rf from your Action

mabbo 5y ago

In the middle of onboarding with a new company. We're at a critical point of training that requires GitHub. Oh boy, this is going to be a fun day.

awestroke 5y ago
The incident lasted less than an hour. How does this affect your whole day?
- mabbo 5y ago
  It didn't. But at the time, the trainer was very, very worried.
tyingq 5y ago
Maybe you could sign up for a GitHub enterprise trial? At the least first few screens seem to be working.

bob1029 5y ago

This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically your entire business through the product. I do agree that it's ridiculous to assume that we can manage Github's software better than their own engineers, but at the same time our infrastructure has proven itself to be extremely reliable over the last 4-5 years. Even hosting GH enterprise on public AWS/Azure is more ideal in my eyes now, because I can control the physical region and tenancy. There is an Azure datacenter within 100 miles of many of our home offices and I can ensure that our Github stack spins up there. Minimizing the amount of internet you have to transit to get to your applications can sidestep a lot of this stormy public cloud/internet weather bullshit.

andrewstuart2 5y ago
I think you will quickly find if you're just deploying GH Enterprise on premises that it is not at all what you get from the GH Cloud offering. GHE has its own product roadmap that is quite a bit behind the cloud product, and in many cases (IMO) unacceptably so. It still doesn't support cache for runners, last I checked, though I've since moved on from the org that required me to work with GHE. I'm back to my happy place with self-hosted GitLab, and a little bit of GitHub cloud.
- bob1029 5y ago
  > GHE has its own product roadmap that is quite a bit behind the cloud product This is exactly what we want though. We don't need the new fancy shit on a regular cadence. Issues, Code, PRs and 1 line checkbuild scripts are all we care about. Everything else is built into our software.
  - andrewstuart2 5y ago
    What I mean is that it's quite clearly not the same product as cloud. It does have roughly the same road map as cloud, just pretty far behind and/or cherry-picking some features in a different order. My experience with enterprise software is that it doesn't matter what the roadmap is as long as you get to choose if/when the benefits outweigh the risks for an update. And you usually want certain releases to get backported security updates for that same reason. You don't have to take new features and their associated bugs but you do want, and get, security fixes. That's a separate thing, because this would be the case if GHE was just "run your own copy of .com." What is really rough about GHE is that you can't choose a lot of the features or IMO baseline requirements like caching that you've probably come to expect from github.com, and may have been around for years. At least not until they can get GHE to parity with .com.

jacobrussell 5y ago

Good time for a lunch break I guess

mot2ba 5y ago

This incident has been marked as resolved. You guys could check how often Github notable incidents that really captures the audience's attention in HN [0] [1]: [0]https://news.ycombinator.com/from?site=githubstatus.com https://news.ycombinator.com/from?site=githubstatus.com [1]https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=false&query=githubstatus.com&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=fal...

jrochkind1 5y ago
Looks like about once every two months? Which is actually more than I expected, and seems like kind of too much.

intunderflow 5y ago

When GitHub is down we legally don't have to do any work, it's in the constitution I swear

m_a_g 5y ago
Best comment in this thread by far

jrochkind1 5y ago

Hmm, how do I get the Github Actions CI to run on all the already existing PR's for which it never ran? Anyone know?

mdaniel 5y ago
Would this do it? https://docs.github.com/en/rest/reference/actions#create-a-workflow-dispatch-event https://docs.github.com/en/rest/reference/actions#create-a-w... or, depending on the error, perhaps this? https://docs.github.com/en/rest/reference/actions#re-run-a-workflow https://docs.github.com/en/rest/reference/actions#re-run-a-w...
- jrochkind1 5y ago
  Not sure. There was no error reported, just a "waiting" message, with no "run" created. i'm not sure I have any ID's to give to those API calls. Would have preferred something from the web UI. I just went ahead and created new commits for them all. (Create an empty commit, or --amend to re-commit the last commit with a new timestamp). Would prefer if there were an easier way to do it, but fighting with http API's I am not familiar with when they aren't immediately apparent and I'm not sure they'll work at all was not that easier way.

blondin 5y ago

i wonder if github ever considered a hybrid approach? an on-prem enterprise solution that syncs with github cloud.