8 ms·
Github: Major Service Outage
- jamesjguthrie 14y agoPfft, always down these days. BitBucket for me.
- joshmlewis 14y agoNice try, Bitbucket marketing team. ;)
- piggity 14y agoNice try, LaunchPad marketing team trying to deflect users from Bitbucket!
- jamesjguthrie 14y agoLol, downvoted? Really? I'm not BitBucket marketing team, check my posts. This is just an honest view on GitHub.
- spleeder 14y agoThe internet is collapsing folks. YouTube is down as well: http://news.ycombinator.com/item?id=4670859 http://news.ycombinator.com/item?id=4670859
- theevocater 14y agoI saw that and was confused. YouTube seems to be working for me
- deleted 14y ago[deleted]
- nbakshi 14y agoit was down for a little time. http://www.nbcnews.com/technology/technolog/uh-oh-youtube-sporadically-down-some-1C6549957 http://www.nbcnews.com/technology/technolog/uh-oh-youtube-sp...
- Jgrubb 14y agoI think it's probably my fault. Github seems to go down every time I start poking around in someone else's projects.
- aes256 14y agoI just signed up last night and I don't really know what I'm doing yet. Sorry everyone!
- zzleeper 14y agoBy any chance, did you 'poke' with a cluster of LOICs?
- thebigshane 14y ago(Never heard that term before...) Low Orbit Ion Cannon (LOIC) is an open source network stress testing and denial-of-service attack application [...] LOIC performs a denial-of-service (DoS) attack (or when used by multiple individuals, a DDoS attack) on a target site by flooding the server with TCP packets or UDP packets with the intention of disrupting the service of a particular host. People have used LOIC to join voluntary botnets. -- http://en.wikipedia.org/wiki/Low_Orbit_Ion_Cannon http://en.wikipedia.org/wiki/Low_Orbit_Ion_Cannon
- kmfrk 14y agoYou damn chaos monkey ruining it for the rest of us.
- trebor 14y agoGitHub says it's a DDOS attack that's hampering their services.
- creativityhurts 14y agoThey said they're under a DDoS attack https://twitter.com/github/statuses/259029493669310464 https://twitter.com/github/statuses/259029493669310464
- alook 14y agoWho DDOS'es twitter? Mercurial committers? the Bitbucket people?
- Stefan_H 14y agodid you mean "who DDOS'es Github"?
- danso 14y agoAssuming you mean github: people who want to cause chaos.
- fusiongyro 14y agoI heard once about a tire shop that drummed up business by strewing nails along the highway around it. Somewhere outside America; India or Thailand or somewhere else. Anyway, that kind of business practice wouldn't get you far in the Western world, so it seems unlikely. OTOH, if I had a botnet and I wanted to see how powerfully it could DDOS without drawing a lot of mainstream media attention, github might be good for target practice. They have better infrastructure than most, and they always do detailed write-ups afterwards.
- riffraff 14y agoOT: the urban legend about tire shops putting nails in the neighborhoods' is very widespread. I have heard about it in at least three countries, and I recall my school teacher telling me about it in '91. I sort of expect it wouldn't be an effective practice anywhere.
- huhtenberg 14y agoIt's an urban legend until you actually catch a nail and there is a tire shop conveniently located not 100 yards away. True story, Indian reservation in BC, Canada. Who cares if it's a tacky business model or an urban legend. It works.
- NelsonMinar 14y agoGithub status is hosted on Github, d'oh. Their Twitter feed posted at 2037Z: "We are experiencing issues due to a DDOS attack, working hard to restore service..." https://twitter.com/github/status/259029493669310464 https://twitter.com/github/status/259029493669310464
- jbredeche 14y agoLooks like github status is on heroku. It's loading fine for me: > host status.github.com status.github.com is an alias for appid129905herokucom-760859479.us-east-1.elb.amazonaws.com. appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 50.19.101.240 appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 50.19.101.216 appid129905herokucom-760859479.us-east-1.elb.amazonaws.com has address 107.22.241.165
- danhunter 14y agoYep after the last major outage they switched to Heroku
- bgentry 14y agoThey were already running their status site on Heroku prior to the last major service outage.
- adgar2 14y agoThis is correct. They needed ~90 dynos to keep up with about 500 qps on their status page in the last outage. Source: http://news.ycombinator.net/item?id=4524489 http://news.ycombinator.net/item?id=4524489
- gojomo 14y agoShould probably throw a Fastly or CloudFlare cache in front of a single dyno.
- songgao 14y agoHN is much slower here as well. Is HN also targeted?
- FiloSottile 14y agoProbably it's only all of us swarming here as GH and YT are down ;)
- hkdobrev 14y agoGithub status: "We've temporarily disabled service on port 80 while we investigate the source of a connection flood. HTTPS, GIT, and SSH service are unaffected." Great way to keep the git push/pull workflow unaffected.
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- FiloSottile 14y ago01:41 PM PST We've temporarily disabled service on port 80 while we investigate the source of a connection flood. HTTPS, GIT, and SSH service are unaffected. 01:33 PM PST "We are experiencing issues due to a DDOS attack, working hard to restore service"
- 8ig8 14y agoI'm reading over the status updates a few hours after the fact, but for some reason the 1:33PM stands out because it is wrapped in quotes. Probably minor and meaningless. Just noted because it's different from the others.
- imbriaco 14y agoIt's because we update the status site from our chat room via Hubot and whoever posted that particular update didn't realize that they didn't need to put quotes around the body of the message.
- theevocater 14y agoIt appears that their http endpoints are down, but their git endpoints are working fine EDIT: or not anymore. I was able to push but now I am not.
- ca136 14y agoWas just able to push/pull okay, overall seems like they're dealing with the outage really well. Their status page is really helpful, they were on twitter quickly, and they tried to restore service to people who might need to push/pull to their repos.
- theevocater 14y agoyeah watched their status site off and on for the last few minutes and it seems their git endpoints are intermittent while the site is out.
- xxpor 14y agoDoes anyone know where GitHub is hosted? I assume not AWS because status.github is on heroku (and therefore AWS).
- csarva 14y agoThe status site is on heroku/ec2 while the main services are on rackspace. Appears both are being attacked by one very unhappy camper.
- shadowmatter 14y agoThey're on Rackspace: http://techcrunch.com/2012/09/18/patent-complaint-filed-against-rackspace-for-hosting-github/ http://techcrunch.com/2012/09/18/patent-complaint-filed-agai... https://github.com/blog/493-github-is-moving-to-rackspace https://github.com/blog/493-github-is-moving-to-rackspace
- deleted 14y ago[deleted]
- thechut 14y agoVery slow but I was able to pull at 4:55EST
- FuzzyDunlop 14y agoNothing against Github but this probably highlights the real benefit of DVCS: setting up multiple remotes for your repo. Manage it probably and when one service goes down, fall back to Bitbucket or another service. It would limit the potential damage these attacks could cause, given the reliance dev teams have on pushing code to a central repo. Taking down a site like Github has a fairly clear effect on the productivity of a lot of their users.
- bluesaunders 14y agoIt'd be glorious if one service (say GitHub) had multiple remote links that dynamically switched for you when service faced an outage.
- dasil003 14y agoIf it's one service though there is still always a single point of failure outside of your control. If you want true redundancy you need to use multiple versions with no interconnections and develop your own solution to mitigate partial failure, of course your own solution is also a single point of failure, but at least it's under your control. That said, Github has never been down long enough for me to justify a fallback procedure and policy for my team. The complexity of the solution outweighs the benefit. We'll cross that bridge when we come to it.
- nivloc 14y ago
- zheng 14y agoFirst off, as others have said, disabling port 80 is a great way to handle this, I don't really care that much if I can see/use github the website for a few hours, but I'd be much more upset if I couldn't pull my code. Secondly, I kind of like when big sites go down when I'm not in desperate need because it means a really nice aftermath write up is on the way. Can't wait to hear more about this one.
- adambard 14y agoThey didn't even disable https, so the site was still fully usable.
- mst 14y ago> "disabling port 80 is a great way to handle this" It's been a disaster for a bunch of people I know. git is distributed by nature so they all had extra remotes they could use. github's issues and other project metadata wasn't distributed, since that's github's alone. So all of the friends of mine who are corporate github users who were using git in a distributed style (a minority of their overall customer base, I would suspect) are more screwed by the web app's absence than they would have been by a repository problem. I suspect github made the right choice for their customer base overall, but I still find the anecdata interesting.
- kijin 14y agoThe web app is also available on port 443, which every "corporate github user" should have been using anyway.
- wfarr 14y agoIn fact, all HTTP access redirects to HTTPS for just about everything. And most modern browsers (recent versions of Chrome and Safari) that have accessed a website over HTTPS once happen to _prefer_ HTTPS by default for that site.
- huhtenberg 14y ago> disabling port 80 is a great way to handle this They got lucky. What would be a great way to handle it if next time it's port 21 or 443?
- brokentone 14y agoGitHub, YouTube, I'm having trouble on Wordpress.org, anything else down for people? Anyone noticing any correlations?
- longdivision 14y agohttps://twitter.com/AnonymousOwn3r https://twitter.com/AnonymousOwn3r is taking credit for attacks against YouTube and GitHub amongst others.
- veeti 14y agoStop giving some idiot script kiddie the attention he wants. It's probably not him anyway.
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- longdivision 14y agoIt's Likely, but this is someone with a history of claims to this sort of attack and a large following. I was interested in who was running the large attacks and what their motivations might be and no sources had been suggested in the discussion at that point.
- lsb 14y agoI can take credit for winning my city's marathon, but only investigations will reveal the facts.
- hkdobrev 14y agoGithub.com is now up https://github.com/ https://github.com/ https://status.github.com/ https://status.github.com/
- desbest 14y agoWhy do people always post on Hacker News whenever Github is offline? It's not going to make Github go online any quicker. They have a Twitter account, you know.
- tedivm 14y agoIt's a site most people here use, and it's interesting. It's not about getting things back up faster, but letting people know what's going on and giving them the chance to witness how a large site fails and recovers.
- autotravis 14y agocan we please try not to become slashdot, with the "this doesn't belong here!!" comments on every post.
- wamatt 14y ago/enqueue jokes about decentralized DVCS systems being hosted centrally... Srsly tho, the relationship between Git and GitHub, might be somewhat analogous to that of BitTorrent and TPB
- wmf 14y agoI call it recentralized.
- tlrobinson 14y agoFortunately since you have an entire copy of the repository it doesn't really matter. Your team can just push the latest copy to another remote like Bitbucket and start using that. Well, except for Issues, Pull Requests, Wiki, etc, but those aren't part of Git.
- acdha 14y agoIt's both a joke and educational: I didn't know they were down until I caught up on my RSS feeds last night
- augustl 14y agoCheatsheet for continuing work when Github is down: 1) ssh myserver.com 2) adduser git 3) sudo su git 4) cd $HOME 5) mkdir .ssh 6) Add people's public keys to .ssh/authorized_keys (in /home/git) 7) git init --bare myrepo.git 8) Push and pull to git@myserver.com:myrepo.git Voila!