8 ms·

GitHub Major Service Outage

204 points by DeepWinter 9y ago

See https://status.github.com

detaro 9y ago
"discussion": https://news.ycombinator.com/item?id=14451924 https://news.ycombinator.com/item?id=14451924
runn1ng 9y ago
Git access to repos seems to be working for me (pull, push).
runeks 9y ago
Good thinking on GitHub's part not using github.com/github/status to host the content of status.github.com. Amazon, take notice.
- brian_herman 9y ago
  Were they doing this before amazon had that major outage?
  - aneutron 9y ago
    DUN DUN DUUUUUN !
  - deleted 9y ago
    [deleted]
  - deleted 9y ago
    [deleted]
  - imbriaco 9y ago
    Yes. They've been hosting it outside of their production infrastructure for several years.
- mhils 9y ago
  Nitpick: They would still fail for DNS issues with *.github.com, so a domain like githubstatus.com would be even more resilient.
  - peterwwillis 9y ago
    Nitpick: www.githubstatus.com is more flexible, potentially more resilient.
    - aerovistae 9y ago
      Nitpick: www.statushub.com is less obviously related, and therefore won't be attacked in tandem. If they want to go all the way, maybe just something like www.wesellnikesdiscount.com.
citrusui 9y ago
The outage seems to be resolved as of 8:58 EDT
- DeepWinter 9y ago
  Yep. Now wonder what the issue was. Ghost in the shell?
tyingq 9y ago
Seeing an interesting thing where my github issue comments just posted are apparently posted a short time in the future. http://imgur.com/a/eQSc9 http://imgur.com/a/eQSc9 Doesn't seem to break anything, but it is a bit curious. May not be new though...I just happened to notice it today.
- i336_ 9y ago
  I've noticed this behavior with a lot of services. I can only chalk it up to something like clock drift between the processing node and the database server. Irritatingly I can't remember which site it was but I posted something somewhere a couple days ago and immediately after hitting enter the site marked what I'd said as submitted "a few seconds from now". I never fail to be amused that the fuzzy time library being used has code specifically designed to handle this edge case scenario. :D
  - toomuchtodo 9y ago
    Message queues and eventual consistency. Unless your request requires something "atomicy", 200/201 response should be a sign of "got the message, will get to work on it when we can".
    - peterwwillis 9y ago
      201 is "request has been fulfilled, resource has been created". It is explicitly not "we will get to it when we can". You are thinking of 202, "request has been accepted for processing", for asynchronous request processing.
      - toomuchtodo 9y ago
        I assure you, 201 is used all the time to say something has been done when it's only been queued, regardless of what the RFC says. This is based on real world integration experience. Http response status codes rarely align with RFC guidelines.
i336_ 9y ago
As of right now... On the one hand, I see "Everything operating normally." at the top in green, and no flags or alerts. On the other hand, the charts look good, but "App server availability" looks interesting, the right edge of the chart is pretty much at 0%.
drinchev 9y ago
A bit weird. GitHub says they fixed it, but on the other hand CircleCI still considers it as an outage : > Monitoring > May 31, 2017 3:08 PM > GitHub have declared the outage resolved and we are starting to see incoming GitHub hooks. Builds are being triggered again. However we are still seeing failures with the GitHub API. This continues to prevent our webapp > from fetching data from GitHub. We are monitoring the situation and will ensure sufficient capacity for when their service resumes normal operations.
- amorphid 9y ago
  Maybe fixing the problem is different than recovering? Like stopping blood loss vs slowly replacing the blood.
thejosh 9y ago
Looks like their cdn is having problems as well now, seeing timeouts when trying to download archives.
- r3bl 9y ago
  Was receiving random downtimes when I've tried opening a certain project and its wiki ~1 hour ago. Nothing big and a couple of refreshes fixed it, just minor annoyance.
gionn 9y ago
web looks fine but repository are not so responsive
Jayakumark 9y ago
A little funny like Silicon Valley episode , saw the news from GitHub CEO yesterday saying our goal is zero downtime and now it's down
- IanCal 9y ago
  Perhaps an issue with punctuation? Goal: zero downtime. vs Goal zero: downtime.
  - pavement 9y ago
    Works on contingency? No, money down!
- Xylakant 9y ago
  Zero downtime is always a goal and never achieved for any complex service. They literally all go down sooner or later.
- yeukhon 9y ago
  You sure it isn't zero downtime deployment? But I thought Github runs infrastructure globally? I remember some outage were caused by DoDS, and some were software bugs / bad config. Probably good idea to do rolling deployment. I will be surprised if they haven't for the kind of top engineering team they are running.
nadim 9y ago
MEAN WEB RESPONSE TIME - 262ms 98TH PERC. WEB RESPONSE TIME - 1134ms 4.3x?
- maxyme 9y ago
  And the 98th percentile is still faster than the 50th percentile of GitLab...
samgranieri 9y ago
Apparently it's resolved. I'd like to read their postmortem on it. They write those extremely well
apeace 9y ago
As others have said, Github's postmortems are always great. But frankly, I'd rather they have better uptime. Every couple months is too much. I pay them. My work pays them. If their CEO is serious about zero downtime, how about he offers his paying customers a credit for time they cannot access the service?
- richardwhiuk 9y ago
  They are? https://status.github.com/messages/2017-01-18 https://status.github.com/messages/2017-01-18 has a bunch of major service outages and no link to any post-mortem. The vague rumour always seems to be 'DDoS attack I guess' but there's very little in the way of formal reporting as far as I can tell...
- Thaxll 9y ago
  Use on prem. https://enterprise.github.com/faq https://enterprise.github.com/faq
- Xylakant 9y ago
  > I pay them. My work pays them. Hmm. You pay them to uphold a contract. What does that contract say about SLAs and availability? Probably the same as the TOS that I agreed to when paying and those specifically say: GitHub does not warrant that the Service will meet your requirements; that the Service will be uninterrupted, timely, secure, or error-free; that the information provided through the Service is accurate, reliable or correct; that any defects or errors will be corrected; that the Service will be available at any particular time or location; or that the Service is free of viruses or other harmful components. You assume full responsibility and risk of loss resulting from your downloading and/or use of files, information, content or other material obtained from the Service. If you negotiate, you might get better terms and guarantees, for example with github enterprise. You might also have to pay substantially more for those. I understand, it sucks when github is down. But we all get what we pay for and we all don't want to pay for more. And yes, I do have clients that meticulously mirror all their dependencies from outside sources and spend significant money on this - money that pays off in exactly these situations.
rodionos 9y ago
github daily availability history: https://apps.axibase.com/chartlab/25f38b08/2/ https://apps.axibase.com/chartlab/25f38b08/2/
JohnHaugeland 9y ago
feel free to make a better one
- edoceo 9y ago
  It's called GitLab. Not 100% uptime but better (and constantly improving)
  - goralph 9y ago
    GitLab, really ?
    - edoceo 9y ago
      I'm very happy I switched. I even posted here on their last outage. That was a stressful 15 minutes. Not perfect, just working loads better for me and my teams
  - xrjn 9y ago
    Gitlab does not have features parity with Github. I personally also stumbled upon a bizzare bug that doesn't allow a friend to add me to any of his private repo's. Asked on IRC and nothing much came of it unfortunately.
    - cmatija 9y ago
      Hiya, Which features would you like to see in GitLab? We'd love to talk about it. You could also open a feature proposal issues in https://gitlab.com/gitlab-org/gitlab-ce/issues https://gitlab.com/gitlab-org/gitlab-ce/issues.
      - justinclift 9y ago
        Err... you're responding to someone who very clearly said they hit a (serious for them) bug which needs fixing. That's definitely not a feature proposal. :D
  - 9y ago
saosebastiao 9y ago
Business idea: github hosting failover. You'd probably need a modified git client, but if you can't push/pull/whatever from github, it transparently fails over to your service which will sync up with github once they've recovered. Even better idea: github should stop failing.
- prh8 9y ago
  Gitlab has both push/pull mirroring, I wonder if it would be possible to use them together to accomplish this.
- peterjlee 9y ago
  Why not just use Github Enterprise then?
- emars 9y ago
  When expanded to see the monthly trend it shows 99.6% availability. Serious Question: is there enough people that would pay for that 0.4% to support a business?
- tedchs 9y ago
  A couple times I have configured multiple "remotes" for my local git repo and pushed to both, e.g. GitHub + Google Cloud Source Repository, or even just a bare repo on a VPS.
peterjlee 9y ago
Funny thing is Github sometimes makes more sales after an outage because clients want to upgrade to the enterprise edition to host on their own servers.
JensRantil 9y ago
I guess they just rolled out their new DNS infrastructure (https://githubengineering.com/dns-infrastructure-at-github/ https://githubengineering.com/dns-infrastructure-at-github/) :-P

204 points by DeepWinter 9y ago

See https://status.github.com

detaro 9y ago

"discussion": https://news.ycombinator.com/item?id=14451924 https://news.ycombinator.com/item?id=14451924

runn1ng 9y ago

Git access to repos seems to be working for me (pull, push).

runeks 9y ago

Good thinking on GitHub's part not using github.com/github/status to host the content of status.github.com. Amazon, take notice.

brian_herman 9y ago
Were they doing this before amazon had that major outage?
- aneutron 9y ago
  DUN DUN DUUUUUN !
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- imbriaco 9y ago
  Yes. They've been hosting it outside of their production infrastructure for several years.
mhils 9y ago
Nitpick: They would still fail for DNS issues with *.github.com, so a domain like githubstatus.com would be even more resilient.
- peterwwillis 9y ago
  Nitpick: www.githubstatus.com is more flexible, potentially more resilient.
  - aerovistae 9y ago
    Nitpick: www.statushub.com is less obviously related, and therefore won't be attacked in tandem. If they want to go all the way, maybe just something like www.wesellnikesdiscount.com.

citrusui 9y ago

The outage seems to be resolved as of 8:58 EDT

DeepWinter 9y ago
Yep. Now wonder what the issue was. Ghost in the shell?

tyingq 9y ago

Seeing an interesting thing where my github issue comments just posted are apparently posted a short time in the future. http://imgur.com/a/eQSc9 http://imgur.com/a/eQSc9 Doesn't seem to break anything, but it is a bit curious. May not be new though...I just happened to notice it today.

i336_ 9y ago
I've noticed this behavior with a lot of services. I can only chalk it up to something like clock drift between the processing node and the database server. Irritatingly I can't remember which site it was but I posted something somewhere a couple days ago and immediately after hitting enter the site marked what I'd said as submitted "a few seconds from now". I never fail to be amused that the fuzzy time library being used has code specifically designed to handle this edge case scenario. :D
- toomuchtodo 9y ago
  Message queues and eventual consistency. Unless your request requires something "atomicy", 200/201 response should be a sign of "got the message, will get to work on it when we can".
  - peterwwillis 9y ago
    201 is "request has been fulfilled, resource has been created". It is explicitly not "we will get to it when we can". You are thinking of 202, "request has been accepted for processing", for asynchronous request processing.
    - toomuchtodo 9y ago
      I assure you, 201 is used all the time to say something has been done when it's only been queued, regardless of what the RFC says. This is based on real world integration experience. Http response status codes rarely align with RFC guidelines.

i336_ 9y ago

As of right now... On the one hand, I see "Everything operating normally." at the top in green, and no flags or alerts. On the other hand, the charts look good, but "App server availability" looks interesting, the right edge of the chart is pretty much at 0%.

drinchev 9y ago

A bit weird. GitHub says they fixed it, but on the other hand CircleCI still considers it as an outage : > Monitoring > May 31, 2017 3:08 PM > GitHub have declared the outage resolved and we are starting to see incoming GitHub hooks. Builds are being triggered again. However we are still seeing failures with the GitHub API. This continues to prevent our webapp > from fetching data from GitHub. We are monitoring the situation and will ensure sufficient capacity for when their service resumes normal operations.

amorphid 9y ago
Maybe fixing the problem is different than recovering? Like stopping blood loss vs slowly replacing the blood.

thejosh 9y ago

Looks like their cdn is having problems as well now, seeing timeouts when trying to download archives.

r3bl 9y ago
Was receiving random downtimes when I've tried opening a certain project and its wiki ~1 hour ago. Nothing big and a couple of refreshes fixed it, just minor annoyance.

gionn 9y ago

web looks fine but repository are not so responsive

Jayakumark 9y ago

A little funny like Silicon Valley episode , saw the news from GitHub CEO yesterday saying our goal is zero downtime and now it's down

IanCal 9y ago
Perhaps an issue with punctuation? Goal: zero downtime. vs Goal zero: downtime.
- pavement 9y ago
  Works on contingency? No, money down!
Xylakant 9y ago
Zero downtime is always a goal and never achieved for any complex service. They literally all go down sooner or later.
yeukhon 9y ago
You sure it isn't zero downtime deployment? But I thought Github runs infrastructure globally? I remember some outage were caused by DoDS, and some were software bugs / bad config. Probably good idea to do rolling deployment. I will be surprised if they haven't for the kind of top engineering team they are running.

nadim 9y ago

MEAN WEB RESPONSE TIME - 262ms 98TH PERC. WEB RESPONSE TIME - 1134ms 4.3x?

maxyme 9y ago
And the 98th percentile is still faster than the 50th percentile of GitLab...

samgranieri 9y ago

Apparently it's resolved. I'd like to read their postmortem on it. They write those extremely well

apeace 9y ago

As others have said, Github's postmortems are always great. But frankly, I'd rather they have better uptime. Every couple months is too much. I pay them. My work pays them. If their CEO is serious about zero downtime, how about he offers his paying customers a credit for time they cannot access the service?

richardwhiuk 9y ago
They are? https://status.github.com/messages/2017-01-18 https://status.github.com/messages/2017-01-18 has a bunch of major service outages and no link to any post-mortem. The vague rumour always seems to be 'DDoS attack I guess' but there's very little in the way of formal reporting as far as I can tell...
Thaxll 9y ago
Use on prem. https://enterprise.github.com/faq https://enterprise.github.com/faq
Xylakant 9y ago
> I pay them. My work pays them. Hmm. You pay them to uphold a contract. What does that contract say about SLAs and availability? Probably the same as the TOS that I agreed to when paying and those specifically say: GitHub does not warrant that the Service will meet your requirements; that the Service will be uninterrupted, timely, secure, or error-free; that the information provided through the Service is accurate, reliable or correct; that any defects or errors will be corrected; that the Service will be available at any particular time or location; or that the Service is free of viruses or other harmful components. You assume full responsibility and risk of loss resulting from your downloading and/or use of files, information, content or other material obtained from the Service. If you negotiate, you might get better terms and guarantees, for example with github enterprise. You might also have to pay substantially more for those. I understand, it sucks when github is down. But we all get what we pay for and we all don't want to pay for more. And yes, I do have clients that meticulously mirror all their dependencies from outside sources and spend significant money on this - money that pays off in exactly these situations.

rodionos 9y ago

github daily availability history: https://apps.axibase.com/chartlab/25f38b08/2/ https://apps.axibase.com/chartlab/25f38b08/2/

JohnHaugeland 9y ago

feel free to make a better one

edoceo 9y ago
It's called GitLab. Not 100% uptime but better (and constantly improving)
- goralph 9y ago
  GitLab, really ?
  - edoceo 9y ago
    I'm very happy I switched. I even posted here on their last outage. That was a stressful 15 minutes. Not perfect, just working loads better for me and my teams
- xrjn 9y ago
  Gitlab does not have features parity with Github. I personally also stumbled upon a bizzare bug that doesn't allow a friend to add me to any of his private repo's. Asked on IRC and nothing much came of it unfortunately.
  - cmatija 9y ago
    Hiya, Which features would you like to see in GitLab? We'd love to talk about it. You could also open a feature proposal issues in https://gitlab.com/gitlab-org/gitlab-ce/issues https://gitlab.com/gitlab-org/gitlab-ce/issues.
    - justinclift 9y ago
      Err... you're responding to someone who very clearly said they hit a (serious for them) bug which needs fixing. That's definitely not a feature proposal. :D
- 9y ago

saosebastiao 9y ago

Business idea: github hosting failover. You'd probably need a modified git client, but if you can't push/pull/whatever from github, it transparently fails over to your service which will sync up with github once they've recovered. Even better idea: github should stop failing.

prh8 9y ago
Gitlab has both push/pull mirroring, I wonder if it would be possible to use them together to accomplish this.
peterjlee 9y ago
Why not just use Github Enterprise then?
emars 9y ago
When expanded to see the monthly trend it shows 99.6% availability. Serious Question: is there enough people that would pay for that 0.4% to support a business?
tedchs 9y ago
A couple times I have configured multiple "remotes" for my local git repo and pushed to both, e.g. GitHub + Google Cloud Source Repository, or even just a bare repo on a VPS.

peterjlee 9y ago

Funny thing is Github sometimes makes more sales after an outage because clients want to upgrade to the enterprise edition to host on their own servers.

JensRantil 9y ago

I guess they just rolled out their new DNS infrastructure (https://githubengineering.com/dns-infrastructure-at-github/ https://githubengineering.com/dns-infrastructure-at-github/) :-P