5 ms·

Major Service Outage at GitHub

155 points by acconrad 10y ago

the_duke 10y ago
Again. Affects both the website and git.
- joatmon-snoo 10y ago
  Affects push to GitHub* I really hope that your copy of git isn't broken because GitHub went down :p
  - the_duke 10y ago
    Haha got me. :P No I just tried both the website and a `git pull`.
    - mbaha 10y ago
      It's back :).
synthecypher 10y ago
How am I supposed to do any work if I can't copy and paste code.
- PerryCox 10y ago
  StackOverflow is still up.
- mkaziz 10y ago
  My builds are broken because they have dependencies on github repos.
  - mjolk 10y ago
    This is a teaching moment for you then. Minimize external dependencies when you've moved from "thank god it works" to "this is production ready."
  - dsp1234 10y ago
    One solution to that problem is to run your own local git server, with an automatic process for pulling github repositories of the dependencies you need. Then you can always know that you will have the access to our dependencies even with a third party is unavailable.
Cafey 10y ago
Has anyone heard what might be the cause?
- the_duke 10y ago
  It just went down about 10 minutes ago, you'll have to wait a bit for an explanation...
- andrewstuart2 10y ago
  > 09:14 MST Major service outage. It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe one guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.
  - joatmon-snoo 10y ago
    The last outage they were back up in 4 minutes.
  - alfalfasprout 10y ago
    Pretty sure devops at a service like Github have slack alerts and/or something like pagerduty. They'll be drowning in alerts.
    - andrewstuart2 10y ago
      Knows the cause, that is. :-)
- TheRealWatson 10y ago
  Rumors are that they edited some JSON files. Not a good day to touch JSON.
pella 10y ago
Hacker News Time! :)
- joatmon-snoo 10y ago
  I count 4 threads on /new referencing GitHub down :D
  - messutied 10y ago
    but we are keeping this one :) I like it
- madlynormal 10y ago
  It's always Hacker News Time!
sahin-boydas 10y ago
Long live Gitlab...
- hackerboos 10y ago
  Except. Pushes of new repos are still broken in the CE. No fix for over 3 weeks. https://gitlab.com/gitlab-com/support-forum/issues/207 https://gitlab.com/gitlab-com/support-forum/issues/207 https://gitlab.com/gitlab-org/gitlab-ce/issues/26369 https://gitlab.com/gitlab-org/gitlab-ce/issues/26369 https://gitlab.com/gitlab-org/gitlab-ce/issues/21502 https://gitlab.com/gitlab-org/gitlab-ce/issues/21502 Only work around is to create a README on empty repos via the GUI and which then opens up the Protected Branches page. That said the rules around who is able to commit on the repos are broken.
  - Perihelion 10y ago
    Hey there, thanks for bringing this up. First, I want to apologize for the inconvenience you've experienced as a result of this issue. Second, I'd like to apologize for the delay in getting back to you on this as well. You're right, 3+ weeks is a pretty long time for something like this to be affecting people. Unfortunately I don't have an ETA on when this issue will be resolved, but I'll be keeping tabs on it going forward. Please feel free to add additional comments to the issues if you haven't already, or you can reach out to me directly. Again, I'd like to apologize for any trouble this caused you and others affected by this issue.
    - sytse 10y ago
      This issue should be fixed as of today in 8.16.3
- afshinmeh 10y ago
  Gitlab can go down. Gitlab might have outages as well unless you are using the self-hosted version (although, in this case there is a self-hosted version of Github as well). Just because Gitlab is not as famous as Github and hackers don't run their botnets on it, it doesn't mean it is safe or stable.
dagenleg 10y ago
Github is down --> check out the unicorn --> refresh hacker news --> refresh again --> shitpost in comments
- atomical 10y ago
  This submission is a shitpost.
  - deleted 10y ago
    [deleted]
- karlmdavis 10y ago
  The coding equivalent of accident rubbernecking.
euph0ria 10y ago
I wonder how many companies are unable to deploy new updates now because they rely on Github?
- steve1011 10y ago
  mine...
  - holtalanm 10y ago
    mine as well
    - mojitoking 10y ago
      And mine
- samueloph 10y ago
  mine. Try to deploy -> bad key -> check github website -> go to hackernews
  - FloatingGhost 10y ago
    Yep, exactly the same process here. :( Now my travis build might be red all weekend
- dbg31415 10y ago
  Friends don't let friends deploy on Fridays... it's always a horrible idea.
  - smonff 10y ago
    Especially in France where it's 5:27.
    - er0l 10y ago
      excellent point sir.
mkurz 10y ago
Someone was faster: https://news.ycombinator.com/item?id=13391883 https://news.ycombinator.com/item?id=13391883
mariusbutuc 10y ago
…and this is how we remember how much we rely on GitHub, and how much value it provides. Every day.
- yeukhon 10y ago
  It sort of becomes your Internet and computer. No computer no work. No Internet no VPN no work. But in this case I guess investing in local mirror is a great investment.
deleted 10y ago
[deleted]
ComputerGuru 10y ago
Obligatory Gogs plug: https://gogs.io https://gogs.io It also has an awesome automatic mirror mode that can remain sync'd to a remkte repository. I don't care what your company does or how you run it, but please don't just rely on Github to host your code without a mirror somewhere. There are drawbacks to this incessant centralization.
- VMG 10y ago
  does it mirror issues, PRs, wiki pages etc?
  - ComputerGuru 10y ago
    I don't think so (we use other apps for those features) but Github's wiki is just a git repo, isn't it?
- jonathanatx 10y ago
  I'll have to check out the source code for their github replacement when github is back up.
  - mbertschler 10y ago
    https://try.gitea.io/gitea/gitea https://try.gitea.io/gitea/gitea
- johnnycarcin 10y ago
  Just this week I setup gogs on my VPS to mirror all of my repos on github. It's part of my new year goal to avoid my dependence on 3rd party apps. Of course I still am depending on my VPS provider so...
  - the_duke 10y ago
    Just set up a Banana Pi or whatever at home to mirror your mirror.
almonj 10y ago
Friday the 13th...
- Rooster61 10y ago
  The day following a full moon...
  - ff10 10y ago
    The day github had a major outage...
deleted 10y ago
[deleted]
holtalanm 10y ago
looks like it is back up now
- deleted 10y ago
  [deleted]
mbaha 10y ago
It's back. [END OF THREAD ?]
- dajohnson89 10y ago
  Since I can't push to my repo, I'm gonna say no, it's not back.
turshija 10y ago
Site came back online, but ssh/git still down.
philipDS 10y ago
Github has been down (or very slow) a lot the last few months. They used to have a good engineering reputation (at least with me), but it's been tough times over there.
deleted 10y ago
[deleted]
analogmemory 10y ago
"Github is down" is the new compiling. http://xkcd.com/303/ http://xkcd.com/303/
deleted 10y ago
[deleted]
yeukhon 10y ago
So there was an outage yesterday which required failover to another data center. Wonder if both are related. Either way I want to read their postmortem. GitHub's postmortem is usually very educational.
mariusbutuc 10y ago
Currently 98.9212% app server availability in the last month. That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last month. And the trend of that percentage doesn't look that rosy. [1] https://uptime.is/98.9212 https://uptime.is/98.9212
deleted 10y ago
[deleted]
mkaziz 10y ago
I use JSPM on my projects and I have several dependencies on github repos that aren't in NPM. Anyone have any ideas on a fallback solution for when github goes out like this?
- shepbook 10y ago
  Is there a way to be able to point JSPM to a different, non-github repo? A remote clone of the repo might work then.
- dsp1234 10y ago
  Have a local git server, with an automatic process for pulling in the repositories that are your dependencies. Then use the repos from that local git server instead of github directly.
vitro 10y ago
I've configured one remote with two pushurls and set this remote to all my branches. This way I always push to BitBucket and GitHub at the same time. Just wondering, how this would work if I was not the single user contributing.
- bisby 10y ago
  You'd have to either coordinate with the other users to have them do the same, or officially declare which one is "official" and which one is "backup" and set up something so the backup tracks the official (and then you only need to push to the official as well)
webmaven 10y ago
GitHub's Twitter status post at the start of the outage: https://twitter.com/githubstatus/status/819940738452615168 https://twitter.com/githubstatus/status/819940738452615168 Currently 442 retweets, 266 likes. ⋯ OK, they say it's back up: https://twitter.com/githubstatus/status/819947685503442945 https://twitter.com/githubstatus/status/819947685503442945
- etimberg 10y ago
  Still getting 503 First Byte Timeouts when trying to view content on raw.githubusercontent.com so definitely not all back yet
royka118 10y ago
Been working on a down detector type app, its like half baked at the mo, hopefully I can build up a data set that allows people to fairly compare the reliability of different SASS products such as github and bitbucket https://young-forest-39406.herokuapp.com/sites https://young-forest-39406.herokuapp.com/sites
escapologybb 10y ago
I hope everybody on HN is doing okay with this recent outage, I'm here if anybody wants to talk. Seriously, Github Outage Syndrome (GOS) can be horrible. If you don't talk about your outage feelings, they will eat you from the inside out. Obviously anything you tell me will be kept in the strictest confidence, all without judgement. Stay strong HN, Github will be back soon! Edited to Add: Why they hate guys? Didn't mean to offend, thought the post said it all and could probably do with a little levity. My bad. Maybe.
srik 10y ago
Problems at quite a few places actually - http://downdetector.com/ http://downdetector.com/
dajohnson89 10y ago
OAuth API is still down.
kumarm 10y ago
Seems back up now.
achairapart 10y ago
A little bit OT, but having colored favicons based on the status is a nice touch! https://status.github.com/images/status-icon-green.png https://status.github.com/images/status-icon-green.png https://status.github.com/images/status-icon-orange.png https://status.github.com/images/status-icon-orange.png https://status.github.com/images/status-icon-red.png https://status.github.com/images/status-icon-red.png
mwpmaybe 10y ago
I'm envisioning some kind of caching forward proxy for git. Point your clones/fetches at it and it will proxy the connection to the source fork. If the source fork is down, it will serve its cached copy of the fork. I found this[0] but it's prerelease and looks like it hasn't been touched in five years. Anyone know of any other solutions like this? Good idea? Bad idea? 0. https://github.com/rohanpm/ngitcached https://github.com/rohanpm/ngitcached
mojitoking 10y ago
GitHub adds value. They have a few engineering issues to solve given the negative attention such a popular code versioning system attracts. Although there are some outages due to this condition, the value added to our organization far outweighs these minor inconveniences.

155 points by acconrad 10y ago

the_duke 10y ago

Again. Affects both the website and git.

joatmon-snoo 10y ago
Affects push to GitHub* I really hope that your copy of git isn't broken because GitHub went down :p
- the_duke 10y ago
  Haha got me. :P No I just tried both the website and a `git pull`.
  - mbaha 10y ago
    It's back :).

synthecypher 10y ago

How am I supposed to do any work if I can't copy and paste code.

PerryCox 10y ago
StackOverflow is still up.
mkaziz 10y ago
My builds are broken because they have dependencies on github repos.
- mjolk 10y ago
  This is a teaching moment for you then. Minimize external dependencies when you've moved from "thank god it works" to "this is production ready."
- dsp1234 10y ago
  One solution to that problem is to run your own local git server, with an automatic process for pulling github repositories of the dependencies you need. Then you can always know that you will have the access to our dependencies even with a third party is unavailable.

Cafey 10y ago

Has anyone heard what might be the cause?

the_duke 10y ago
It just went down about 10 minutes ago, you'll have to wait a bit for an explanation...
andrewstuart2 10y ago
> 09:14 MST Major service outage. It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe one guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.
- joatmon-snoo 10y ago
  The last outage they were back up in 4 minutes.
- alfalfasprout 10y ago
  Pretty sure devops at a service like Github have slack alerts and/or something like pagerduty. They'll be drowning in alerts.
  - andrewstuart2 10y ago
    Knows the cause, that is. :-)
TheRealWatson 10y ago
Rumors are that they edited some JSON files. Not a good day to touch JSON.

pella 10y ago

Hacker News Time! :)

joatmon-snoo 10y ago
I count 4 threads on /new referencing GitHub down :D
- messutied 10y ago
  but we are keeping this one :) I like it
madlynormal 10y ago
It's always Hacker News Time!

sahin-boydas 10y ago

Long live Gitlab...

hackerboos 10y ago
Except. Pushes of new repos are still broken in the CE. No fix for over 3 weeks. https://gitlab.com/gitlab-com/support-forum/issues/207 https://gitlab.com/gitlab-com/support-forum/issues/207 https://gitlab.com/gitlab-org/gitlab-ce/issues/26369 https://gitlab.com/gitlab-org/gitlab-ce/issues/26369 https://gitlab.com/gitlab-org/gitlab-ce/issues/21502 https://gitlab.com/gitlab-org/gitlab-ce/issues/21502 Only work around is to create a README on empty repos via the GUI and which then opens up the Protected Branches page. That said the rules around who is able to commit on the repos are broken.
- Perihelion 10y ago
  Hey there, thanks for bringing this up. First, I want to apologize for the inconvenience you've experienced as a result of this issue. Second, I'd like to apologize for the delay in getting back to you on this as well. You're right, 3+ weeks is a pretty long time for something like this to be affecting people. Unfortunately I don't have an ETA on when this issue will be resolved, but I'll be keeping tabs on it going forward. Please feel free to add additional comments to the issues if you haven't already, or you can reach out to me directly. Again, I'd like to apologize for any trouble this caused you and others affected by this issue.
  - sytse 10y ago
    This issue should be fixed as of today in 8.16.3
afshinmeh 10y ago
Gitlab can go down. Gitlab might have outages as well unless you are using the self-hosted version (although, in this case there is a self-hosted version of Github as well). Just because Gitlab is not as famous as Github and hackers don't run their botnets on it, it doesn't mean it is safe or stable.

dagenleg 10y ago

Github is down --> check out the unicorn --> refresh hacker news --> refresh again --> shitpost in comments

atomical 10y ago
This submission is a shitpost.
- deleted 10y ago
  [deleted]
karlmdavis 10y ago
The coding equivalent of accident rubbernecking.

euph0ria 10y ago

I wonder how many companies are unable to deploy new updates now because they rely on Github?

steve1011 10y ago
mine...
- holtalanm 10y ago
  mine as well
  - mojitoking 10y ago
    And mine
samueloph 10y ago
mine. Try to deploy -> bad key -> check github website -> go to hackernews
- FloatingGhost 10y ago
  Yep, exactly the same process here. :( Now my travis build might be red all weekend
dbg31415 10y ago
Friends don't let friends deploy on Fridays... it's always a horrible idea.
- smonff 10y ago
  Especially in France where it's 5:27.
  - er0l 10y ago
    excellent point sir.

mkurz 10y ago

Someone was faster: https://news.ycombinator.com/item?id=13391883 https://news.ycombinator.com/item?id=13391883

mariusbutuc 10y ago

…and this is how we remember how much we rely on GitHub, and how much value it provides. Every day.

yeukhon 10y ago
It sort of becomes your Internet and computer. No computer no work. No Internet no VPN no work. But in this case I guess investing in local mirror is a great investment.

deleted 10y ago

[deleted]

ComputerGuru 10y ago

Obligatory Gogs plug: https://gogs.io https://gogs.io It also has an awesome automatic mirror mode that can remain sync'd to a remkte repository. I don't care what your company does or how you run it, but please don't just rely on Github to host your code without a mirror somewhere. There are drawbacks to this incessant centralization.

VMG 10y ago
does it mirror issues, PRs, wiki pages etc?
- ComputerGuru 10y ago
  I don't think so (we use other apps for those features) but Github's wiki is just a git repo, isn't it?
jonathanatx 10y ago
I'll have to check out the source code for their github replacement when github is back up.
- mbertschler 10y ago
  https://try.gitea.io/gitea/gitea https://try.gitea.io/gitea/gitea
johnnycarcin 10y ago
Just this week I setup gogs on my VPS to mirror all of my repos on github. It's part of my new year goal to avoid my dependence on 3rd party apps. Of course I still am depending on my VPS provider so...
- the_duke 10y ago
  Just set up a Banana Pi or whatever at home to mirror your mirror.

almonj 10y ago

Friday the 13th...

Rooster61 10y ago
The day following a full moon...
- ff10 10y ago
  The day github had a major outage...

deleted 10y ago

[deleted]

holtalanm 10y ago

looks like it is back up now

deleted 10y ago
[deleted]

mbaha 10y ago

It's back. [END OF THREAD ?]

dajohnson89 10y ago
Since I can't push to my repo, I'm gonna say no, it's not back.

turshija 10y ago

Site came back online, but ssh/git still down.

philipDS 10y ago

Github has been down (or very slow) a lot the last few months. They used to have a good engineering reputation (at least with me), but it's been tough times over there.

deleted 10y ago

[deleted]

analogmemory 10y ago

"Github is down" is the new compiling. http://xkcd.com/303/ http://xkcd.com/303/

deleted 10y ago

[deleted]

yeukhon 10y ago

So there was an outage yesterday which required failover to another data center. Wonder if both are related. Either way I want to read their postmortem. GitHub's postmortem is usually very educational.

mariusbutuc 10y ago

Currently 98.9212% app server availability in the last month. That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last month. And the trend of that percentage doesn't look that rosy. [1] https://uptime.is/98.9212 https://uptime.is/98.9212

deleted 10y ago

[deleted]

mkaziz 10y ago

I use JSPM on my projects and I have several dependencies on github repos that aren't in NPM. Anyone have any ideas on a fallback solution for when github goes out like this?

shepbook 10y ago
Is there a way to be able to point JSPM to a different, non-github repo? A remote clone of the repo might work then.
dsp1234 10y ago
Have a local git server, with an automatic process for pulling in the repositories that are your dependencies. Then use the repos from that local git server instead of github directly.

vitro 10y ago

I've configured one remote with two pushurls and set this remote to all my branches. This way I always push to BitBucket and GitHub at the same time. Just wondering, how this would work if I was not the single user contributing.

bisby 10y ago
You'd have to either coordinate with the other users to have them do the same, or officially declare which one is "official" and which one is "backup" and set up something so the backup tracks the official (and then you only need to push to the official as well)

webmaven 10y ago

GitHub's Twitter status post at the start of the outage: https://twitter.com/githubstatus/status/819940738452615168 https://twitter.com/githubstatus/status/819940738452615168 Currently 442 retweets, 266 likes. ⋯ OK, they say it's back up: https://twitter.com/githubstatus/status/819947685503442945 https://twitter.com/githubstatus/status/819947685503442945

etimberg 10y ago
Still getting 503 First Byte Timeouts when trying to view content on raw.githubusercontent.com so definitely not all back yet

royka118 10y ago

Been working on a down detector type app, its like half baked at the mo, hopefully I can build up a data set that allows people to fairly compare the reliability of different SASS products such as github and bitbucket https://young-forest-39406.herokuapp.com/sites https://young-forest-39406.herokuapp.com/sites

escapologybb 10y ago

I hope everybody on HN is doing okay with this recent outage, I'm here if anybody wants to talk. Seriously, Github Outage Syndrome (GOS) can be horrible. If you don't talk about your outage feelings, they will eat you from the inside out. Obviously anything you tell me will be kept in the strictest confidence, all without judgement. Stay strong HN, Github will be back soon! Edited to Add: Why they hate guys? Didn't mean to offend, thought the post said it all and could probably do with a little levity. My bad. Maybe.

srik 10y ago

Problems at quite a few places actually - http://downdetector.com/ http://downdetector.com/

dajohnson89 10y ago

OAuth API is still down.

kumarm 10y ago

Seems back up now.

achairapart 10y ago

A little bit OT, but having colored favicons based on the status is a nice touch! https://status.github.com/images/status-icon-green.png https://status.github.com/images/status-icon-green.png https://status.github.com/images/status-icon-orange.png https://status.github.com/images/status-icon-orange.png https://status.github.com/images/status-icon-red.png https://status.github.com/images/status-icon-red.png

mwpmaybe 10y ago

I'm envisioning some kind of caching forward proxy for git. Point your clones/fetches at it and it will proxy the connection to the source fork. If the source fork is down, it will serve its cached copy of the fork. I found this[0] but it's prerelease and looks like it hasn't been touched in five years. Anyone know of any other solutions like this? Good idea? Bad idea? 0. https://github.com/rohanpm/ngitcached https://github.com/rohanpm/ngitcached

mojitoking 10y ago

GitHub adds value. They have a few engineering issues to solve given the negative attention such a popular code versioning system attracts. Although there are some outages due to this condition, the value added to our organization far outweighs these minor inconveniences.