13 ms·

Github major service outage

123 points by forlorn 13y ago

https://status.github.com/

jstreebin 13y ago
http://www.isitdownrightnow.com/github.com.html http://www.isitdownrightnow.com/github.com.html https://status.github.com/messages https://status.github.com/messages
- samweinberg 13y ago
  Or for the HN friends without Javascript, http://isup.me/github.com http://isup.me/github.com.
misiti3780 13y ago
im getting a 500 (from NYC) on the main page + i cant push to remote repos
- forlorn 13y ago
  It's on any page, try visiting any repository, wiki etc - same story.
- noir_lord 13y ago
  From here (NE UK) everything is down. Typically I was in the middle of setting up a development environment on a new machine which has a lot of composer dependencies.... Oh well will have to go drink coffee in the sporadic sunshine :).
- gkrishnan 13y ago
  Checked from India, everything seems to be down...
valokafor 13y ago
Same for me from San Diego, CA. I am unable to push and unable to access the main page. Getting "This page is taking way too long to load."
oaksagelew 13y ago
The status page says that "Code Downloads" is "Normal". How do you get to the downloads page for a given repo? Is there some canonical form for a URL for repos? Maybe while they're fixing it we can still download code...
- vjt 13y ago
  try: git://github.com/{user}/{repo}.git or https://github.com/{user}/{repo}.git anyway, not working right now for me (Italy) EDIT: working, give it 30 seconds to start.
  - oaksagelew 13y ago
    Nope, no joy - 30 seconds or otherwise.
dpweb 13y ago
Why Unicorn Why!
- revskill 13y ago
  So, what's better than Unicorn ?
  - terhechte 13y ago
    Two unicorns.
  - glitchdout 13y ago
    Puma on JRuby, apparently. http://ylan.segal-family.com/blog/2013/05/20/unicorn-vs-puma-redux/ http://ylan.segal-family.com/blog/2013/05/20/unicorn-vs-puma...
  - deleted 13y ago
    [deleted]
  - tinco 13y ago
    Passenger :)
  - FooBarWidget 13y ago
    Phusion Passenger Enterprise: https://news.ycombinator.com/item?id=5662569 https://news.ycombinator.com/item?id=5662569
PommeDeTerre 13y ago
I find it somewhat odd that, given git's emphasis on distributed version control, we see so many git users have centralized themselves quite heavily on GitHub. Whenever GitHub runs into problems like this, it reminds me of when a team's CVS or Subversion servers used to go down. It could be a pretty disruptive occurrence, if it wasn't resolved quickly. While git can theoretically handle this better, in practice the use of GitHub, for instance, renders git just as susceptible.
- 0x006A 13y ago
  Thats not true, with CVS/SVN you can not commit, log or do anything while the server is down. With git you can still commit to your local tree, push it to another server or send patches. While it might slow down workflows using github.com for push/pull its not that complicated to temporarily push/pull via another server.
  - k3n 13y ago
    Playing devil's advocate here... but if you were so inclined you could accomplish the same with SVN. Creating and hosting a repo locally (or offsite) is trivially easy, and though it might not be as friendly or natural as it would be with git on a daily basis, it's not impossible either.
    - zwily 13y ago
      So if four people set up local svn servers, and then push to those servers during the outage, what happens when the outage is over?
      - k3n 13y ago
        I've already said it wouldn't be easy, so I'm not sure what your point is.
- knowtheory 13y ago
  How is it at all odd? Github offers a convenient platform for using git. People use it. If Github were to explode forever tomorrow, active projects would just take their locally cloned repositories, and put them online somewhere else and carry on committing (albeit sans github's awesome social tools). That's the real power git offers us. It's just a fact of reality that most projects centrally organize through a few bottlenecks. It's easier for people to remember where to go, if there's only one place to go to. But if you're really concerned about it, i suppose you could build a tool to automatically sync a github repo w/ another 3rd party host if you wanted (or visaversa).
chris_wot 13y ago
Database outage causing everything to go down?
- Lord_DeathMatch 13y ago
  That and their wording suggests a single db server :/
  - noir_lord 13y ago
    They have a typical master/slave setup with memcached on the front. http://www.slideshare.net/err/inside-github http://www.slideshare.net/err/inside-github
    - nknighthb 13y ago
      Since they're using MySQL, is there a technical reason (as opposed to historical/lack of time reason) they're not using Galera Cluster? I'm in the process of migrating an existing datastore to MariaDB+Galera, and so far it seems like everything I could hope for in a clustered RDBMS.
      - leedo__ 13y ago
        Last I investigated Galera it lacked support for query caching. Over 50% of our queries are cache hits, so it made it hard to justify using Galera over a normal master+slave setup. However I could see it being useful for setups where a single server can't handle the load (we average 300 queries/sec on a single server with lots of room to spare.)
        nknighthb 13y ago
        They still disable the query cache, but MySQL's query cache generally isn't considered all that great a thing anyway, so few people care. You're better off making judicious use of Redis or memcached. The biggest win for Galera is high-availability that actually works with minimal effort. (I've never experienced a high-availability solution not based on multi-master/all-nodes-hot principles that didn't cause more problems than it solved.) They also claim some scalability wins at the front end, but I haven't really tested that, and am content with the performance not being terrible.
Queue29 13y ago
Least reliable service that I pay for by a long shot.
- knowtheory 13y ago
  Must not pay for very many services? https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month
  - driverdan 13y ago
    To be fair 99.77% uptime isn't very good.
    - jcape 13y ago
      That isn't fair at all. It's 99.770% for a single month, immediately following a major event. If you sampled yesterday (or tomorrow, assuming no further issues) it would be higher. If you just look at today, it's at the much lower at 95.871%. If you assume no availability issues for the last 12 months (not true, but the point remains) then it's 99.981%. During an actual outage, availability was at the unacceptable 0%. Unfortunately they don't provide 12mo stats, which is what you typically want if you're going to start calculating nines of availability.
      - josh2600 13y ago
        Hey, are you seriously defending 3 9's of uptime? That's abysmal. Github, if they're honest about their 12 month uptime levels would be lucky to be a single 9 service. Their uptime is Terrible with a capital T. But you know what? Until there's something better everyone is going to keep using them, right? Great services with values that are hard to find become damn near irreplaceable even with terrible uptime. This is an obvious place to compete; if you made a github clone that simply stayed online you could win market share during every downtime. However cloning github would not be trivial. And therein lies the problem and the answer to why we accept their terrible uptime levels. They give us something we can't get elsewhere: social coding and easy centralization.
mratzloff 13y ago
It's back up for me.
bennyg 13y ago
I didn't even know it was down haha. I've been pushing to a private GitHub repo all morning with no problems.
bdcravens 13y ago
A number of ticketing systems, like Unfuddle and Assembla, include free git repos. Personally I prefer Github, but if you're already using such a system, might as well utilize its git hosting (for a backup if nothing else)
mfonda 13y ago
Looking through the past few months of messages on http://status.github.com/messages http://status.github.com/messages, I'm really quite surprised at how many service outages, major or minor, are reported. It seems something goes wrong more days than not.
qhoc 13y ago
We mainly used github and one of their last updates, our entire repository turned to blank. Nada, no file, no wiki, nothing... We emailed support and tweeted their account. After one week, someone replied asking "is there problem still there?" I thought that was funny in term of how immature this company is. They eventually fixed it after like 2 weeks by restoring the backup database. Anyways, Github is nice. Use it with caution. It cannot be full blown Enterprise ready grade service yet.

123 points by forlorn 13y ago

https://status.github.com/

http://www.isitdownrightnow.com/github.com.html http://www.isitdownrightnow.com/github.com.html https://status.github.com/messages https://status.github.com/messages

samweinberg 13y ago
Or for the HN friends without Javascript, http://isup.me/github.com http://isup.me/github.com.

misiti3780 13y ago

im getting a 500 (from NYC) on the main page + i cant push to remote repos

forlorn 13y ago
It's on any page, try visiting any repository, wiki etc - same story.
noir_lord 13y ago
From here (NE UK) everything is down. Typically I was in the middle of setting up a development environment on a new machine which has a lot of composer dependencies.... Oh well will have to go drink coffee in the sporadic sunshine :).
gkrishnan 13y ago
Checked from India, everything seems to be down...

valokafor 13y ago

Same for me from San Diego, CA. I am unable to push and unable to access the main page. Getting "This page is taking way too long to load."

oaksagelew 13y ago

The status page says that "Code Downloads" is "Normal". How do you get to the downloads page for a given repo? Is there some canonical form for a URL for repos? Maybe while they're fixing it we can still download code...

vjt 13y ago
try: git://github.com/{user}/{repo}.git or https://github.com/{user}/{repo}.git anyway, not working right now for me (Italy) EDIT: working, give it 30 seconds to start.
- oaksagelew 13y ago
  Nope, no joy - 30 seconds or otherwise.

dpweb 13y ago

Why Unicorn Why!

revskill 13y ago
So, what's better than Unicorn ?
- terhechte 13y ago
  Two unicorns.
- glitchdout 13y ago
  Puma on JRuby, apparently. http://ylan.segal-family.com/blog/2013/05/20/unicorn-vs-puma-redux/ http://ylan.segal-family.com/blog/2013/05/20/unicorn-vs-puma...
- deleted 13y ago
  [deleted]
- tinco 13y ago
  Passenger :)
- FooBarWidget 13y ago
  Phusion Passenger Enterprise: https://news.ycombinator.com/item?id=5662569 https://news.ycombinator.com/item?id=5662569

PommeDeTerre 13y ago

I find it somewhat odd that, given git's emphasis on distributed version control, we see so many git users have centralized themselves quite heavily on GitHub. Whenever GitHub runs into problems like this, it reminds me of when a team's CVS or Subversion servers used to go down. It could be a pretty disruptive occurrence, if it wasn't resolved quickly. While git can theoretically handle this better, in practice the use of GitHub, for instance, renders git just as susceptible.

0x006A 13y ago
Thats not true, with CVS/SVN you can not commit, log or do anything while the server is down. With git you can still commit to your local tree, push it to another server or send patches. While it might slow down workflows using github.com for push/pull its not that complicated to temporarily push/pull via another server.
- k3n 13y ago
  Playing devil's advocate here... but if you were so inclined you could accomplish the same with SVN. Creating and hosting a repo locally (or offsite) is trivially easy, and though it might not be as friendly or natural as it would be with git on a daily basis, it's not impossible either.
  - zwily 13y ago
    So if four people set up local svn servers, and then push to those servers during the outage, what happens when the outage is over?
    - k3n 13y ago
      I've already said it wouldn't be easy, so I'm not sure what your point is.
knowtheory 13y ago
How is it at all odd? Github offers a convenient platform for using git. People use it. If Github were to explode forever tomorrow, active projects would just take their locally cloned repositories, and put them online somewhere else and carry on committing (albeit sans github's awesome social tools). That's the real power git offers us. It's just a fact of reality that most projects centrally organize through a few bottlenecks. It's easier for people to remember where to go, if there's only one place to go to. But if you're really concerned about it, i suppose you could build a tool to automatically sync a github repo w/ another 3rd party host if you wanted (or visaversa).

chris_wot 13y ago

Database outage causing everything to go down?

Lord_DeathMatch 13y ago
That and their wording suggests a single db server :/
- noir_lord 13y ago
  They have a typical master/slave setup with memcached on the front. http://www.slideshare.net/err/inside-github http://www.slideshare.net/err/inside-github
  - nknighthb 13y ago
    Since they're using MySQL, is there a technical reason (as opposed to historical/lack of time reason) they're not using Galera Cluster? I'm in the process of migrating an existing datastore to MariaDB+Galera, and so far it seems like everything I could hope for in a clustered RDBMS.
    - leedo__ 13y ago
      Last I investigated Galera it lacked support for query caching. Over 50% of our queries are cache hits, so it made it hard to justify using Galera over a normal master+slave setup. However I could see it being useful for setups where a single server can't handle the load (we average 300 queries/sec on a single server with lots of room to spare.)
      - nknighthb 13y ago
        They still disable the query cache, but MySQL's query cache generally isn't considered all that great a thing anyway, so few people care. You're better off making judicious use of Redis or memcached. The biggest win for Galera is high-availability that actually works with minimal effort. (I've never experienced a high-availability solution not based on multi-master/all-nodes-hot principles that didn't cause more problems than it solved.) They also claim some scalability wins at the front end, but I haven't really tested that, and am content with the performance not being terrible.

Queue29 13y ago

Least reliable service that I pay for by a long shot.

knowtheory 13y ago
Must not pay for very many services? https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month
- driverdan 13y ago
  To be fair 99.77% uptime isn't very good.
  - jcape 13y ago
    That isn't fair at all. It's 99.770% for a single month, immediately following a major event. If you sampled yesterday (or tomorrow, assuming no further issues) it would be higher. If you just look at today, it's at the much lower at 95.871%. If you assume no availability issues for the last 12 months (not true, but the point remains) then it's 99.981%. During an actual outage, availability was at the unacceptable 0%. Unfortunately they don't provide 12mo stats, which is what you typically want if you're going to start calculating nines of availability.
    - josh2600 13y ago
      Hey, are you seriously defending 3 9's of uptime? That's abysmal. Github, if they're honest about their 12 month uptime levels would be lucky to be a single 9 service. Their uptime is Terrible with a capital T. But you know what? Until there's something better everyone is going to keep using them, right? Great services with values that are hard to find become damn near irreplaceable even with terrible uptime. This is an obvious place to compete; if you made a github clone that simply stayed online you could win market share during every downtime. However cloning github would not be trivial. And therein lies the problem and the answer to why we accept their terrible uptime levels. They give us something we can't get elsewhere: social coding and easy centralization.

mratzloff 13y ago

It's back up for me.

bennyg 13y ago

I didn't even know it was down haha. I've been pushing to a private GitHub repo all morning with no problems.

bdcravens 13y ago

A number of ticketing systems, like Unfuddle and Assembla, include free git repos. Personally I prefer Github, but if you're already using such a system, might as well utilize its git hosting (for a backup if nothing else)

mfonda 13y ago

Looking through the past few months of messages on http://status.github.com/messages http://status.github.com/messages, I'm really quite surprised at how many service outages, major or minor, are reported. It seems something goes wrong more days than not.

qhoc 13y ago

We mainly used github and one of their last updates, our entire repository turned to blank. Nada, no file, no wiki, nothing... We emailed support and tweeted their account. After one week, someone replied asking "is there problem still there?" I thought that was funny in term of how immature this company is. They eventually fixed it after like 2 weeks by restoring the backup database. Anyways, Github is nice. Use it with caution. It cannot be full blown Enterprise ready grade service yet.