11 ms·
GitHub experiencing a large DDoS attack
- iclelland 13y ago15:50 UTC: We are working to mitigate a large DDoS. We will provide an update once we have more information.
- wavee 13y agoSo what?
- briandear 13y agoThe so what is that the majority of people on HN are developers and the majority of them use Github either personally or with their teams. So a major outage is costing potentially millions of dollars in potentially lost developer time. If you calculate an average hourly of just $50 and a developer loses 20 minutes. Then that outage costs $17 x the number of affected developers. That's a pretty big loss. I'm reading HN right now because I can't manage pull requests and do code reviews of my distributed team's code. So I'm just spinning in my chair. That's the so-what.
- rblstr 13y agoYou can't pull code from another node, i.e another developer? Git is distributed, if you want centralised source control use Perforce or SVN or something.
- briandear 13y agosure, and I can build a commenting and pull request system while I'm at it. It's highly inefficient to change workflow for an outage that might not last an hour.
- sukaka 13y agoyes you can pull from another node. you can also push to another node.
- bdcravens 13y agoYou may want to reread the comment. It's more than being able to commit and pull via the command line.
- wavee 13y agoGithub is under ddos at least once a month, how is this still a news worth upvoting here?
- ErikAugust 13y agoWhat's with DDoSing GitHub? Special place in Hell...
- sargun 13y agoGithub.com is a profitable entity that makes a fairly large amount of money. Additionally, Github.com hosts a lot of content. Content that other people may not like. Possibilities are: 1) Extortion of some sort i.e. We want your money, or we'll DDoS you every week 2) Content i.e. Remove repositories a, b, and c with content we don't like, or we'll DDoS you every week 3) Weapons demonstration i.e. We want to show off what l33t hax0rs we are, so we'll DDoS Github.com every week
- gavingmiller 13y agoI ask because I have no plausible answer: What does someone stand to gain by DDoSing github?
- look_lookatme 13y agoCould be an extortion scheme.
- leishulang 13y agomaybe some hackers who believes git repos should be, well, distributed?
- briandear 13y agoSo maybe they can build a distributed system themselves instead of making a point the f*c%s everyone while they have their childish tantrum. This is like nuking a city because you don't like the mayor.
- camus 13y agoblackmailing ? "if you dont pay we'll DDOS you" or whatever. Given all these infected Windows computers all over the world ,it's not going to stop soon. That's the Microsoft legacy.
- gtaylor 13y agoAm I crazy in thinking that there's zero chance Github would pay a "ransom"? Seems like they'd just work with their vendors/providers and mitigate it to avoid setting that precedent (making them even more vulnerable to future attempts).
- bradleyland 13y agoThe attacker's bet is that they can apply pressure beyond the victim's ability to mitigate. If they win that bet, the victim may reach a point where paying out is preferable to the lost business they'd suffer because of the attack.
- jbrooksuk 13y agoWhy is this happening to them almost every day? Is someone jealous? Proving a point? If so, what point? Surely by now, GitHub must know who's responsible and putting more in place to mitigate as much as possible before this happens. Right?
- ripter 13y agoEveryday? The last attack was August 4th. https://status.github.com/messages https://status.github.com/messages
- jbrooksuk 13y agoI said "almost".
- KurtMueller 13y agoYou said 'almost every day'. Seems more like 'almost every week'. I'm just nitpicking though.
- cmer 13y agoThis is getting really problematic. It stops our whole team every time. Anybody has experience with Gitlab and Gitlab CI? How's the flow compared to Github, especially for pull requests, commenting and collaboration?
- jbrooksuk 13y agoI used GitLab by myself, it works really well and since V5 is super easy to get installed.
- nonchalance 13y agoIs there a VM image like Github enterprise?
- druiid 13y agoThere is at least one Vagrant+puppet installer I know of but, well, uhhh... it's on Github and won't help you much at this point :P https://github.com/sbadia/puppet-gitlab https://github.com/sbadia/puppet-gitlab
- kxu 13y agoYes, kind of deadlock.. Nice try anyway :)
- manacit 13y agoBitNami keeps up-to-date Gitlab VMs and Installers: http://bitnami.com/stack/gitlab http://bitnami.com/stack/gitlab
- taylorbuley 13y agoStarted timing out for us mid-deploy. Not a good situation.
- druiid 13y agoI've been very happy with Gitlab. The CI server seems a bit simplistic at this point and I haven't tried it, but I'm slowly moving all of the stuff I need to have more consistent access to, from Github.
- pulleasy 13y agohttp://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-commit-locally.png http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-commit-loca...
- deleted 13y ago[deleted]
- cheeseprocedure 13y agoCan someone with experience mitigating an attack like this describe how it's done? A known set of hosts/address spaces is fine, but it's the "distributed" part I don't understand how to deal with.
- timdorr 13y agoUsually there is some pattern to the traffic that can be identified and filtered out. There are manual ways to do this (iptables, basic firewalls) or more sophisticated "anti-DDOS" boxes you can buy to automate or simplify the process (e.g. Cisco Anomaly Guard or Juniper Junos DDoS Secure).
- pktgen 13y agoWe don't know anything about the attack, but in general: 1. All of this requires that you have more bandwidth than the attacker. 2. ACL drop everything but the ports/protocols you use on your frontend IP. For github.com that would be TCP 80 and 443. (ACLs are cheap, can be done at wire-rate on any capable edge router, so they're a good "first step." In this case it would have the bonus of dropping DNS amplification attacks which are increasingly common.) 3. If it's a TCP SYN flood (probably from spoofed IPs) enable SYN cookies. (Of course now the bottleneck is their load balancer.) 4. If it's an L7 attack, they'll need to identify patterns in the requests to identify and drop the traffic. For example, the attacker may be requesting a single URL only. If it's an L7 attack on a TCP service, they can also automatically add a drop rule for that source address, because at this point the client IP will have been validated (via the SYN challenge). They may also be able to identify certain patterns in the traffic that can be ACL'd at the edge. For example, IP TTL being identical.
- lsh123 13y agoThere are services that specialize in handling DDOS attacks with distributed data centers and large pipes. It is very hard to do it on a shared infrastructure (e.g. AWS) w/o affecting other customers thus hosting providers will be more interested in protecting other clients.
- 300bps 13y agoCan someone with experience mitigating an attack like this describe how it's done? I believe Step 1 is to submit the fact that GitHub is being DDoS'd to Hacker News, reddit, Slashdot and other social networking sites with a direct link to GitHub's web site. This will cause a flood of real people to load the web site, thereby crowding out the DDoS. The site will still be down, but at least it will be due to real people and not the DDoS.
- nonchalance 13y agowhile git is distributed, Github managed to centralize the open source culture
- joemaller1 13y agoLet me get this straight, the command-line interface for googling code in a locally cloned GitHub repo is called grep?
- oalders 13y agoOr "git grep".
- alfg 13y agoSeems to be a recurring problem for them now.
- gunmetal 13y agoGithub enterprise works great.
- izolate 13y agoLet's all take a watercooler break here at HN
- Terretta 13y agoYou git what you pay for. (Actually, there are long established yet less high profile alternatives with decent features and better pricing for commercial teams, like beanstalk or unfuddle or codebasehq. No hip cred, but for example supporting archived projects that you can read w/o using up your repo count license.)
- astrodust 13y agoUnfuddle, while good for its day, is not even close to GitHub. It's woefully out of date, slow, and temperamental in ways that are frustrating. The closest thing going is Bitbucket which is a pretty good competitor for GitHub...in 2010.
- gavingmiller 13y agoThis argument also works the other way. We payed less for beanstalk and got half the features. Sure it was a remote git repo, but our use case for GitHub is more than just external storage, and we were able to remove other tools that we were paying for.
- Terretta 13y agoWe've been developing custom software for over a decade. GitHub's pricing model counts every private repo, active or not, while others let you put repos into R/W or archive mode, or don't count repos at all and only count against your storage quota. The difference in annual cost is significant when you have 20 active independent projects, and hundreds of archived projects, not to mention all the customers needing accounts and access.
- briandear 13y agoCurious though -- yesterday it was the NYTimes, today github, Apple a few weeks before that. Pretty high profile stuff in my opinion.
- AnSavvides 13y agoI do not understand why GitHub get attacked so often - honestly it's not like they are doing anything wrong, they are a fabulous service and the go-to place for most (if not all) developers for all sorts of projects, be it open source or in an enterprise/corporate context.
- manojlds 13y agoDidn't you just answer your question? It is the "go-to" place for most developers.
- Cakez0r 13y agoThat doesn't really answer his question. Why would somebody want to inconvenience developers?
- thenonsequitur 13y agoThe goal may be to potentially open up a new attack vector to steal code, some of which may contain secrets. This is pure speculation, but a definite possibility.
- abeger 13y agoMaybe a developer killed their parents?
- jofo25 13y agoMaybe they want to disrupt developers working for a big bad company.
- rickyc091 13y agoProbably to disrupt businesses that rely on github or cause a service to function improperly causing a security loophole they can exploit.
- viraptor 13y ago- Raising awareness of self-hosted alternatives - Getting people to use BB instead - Fake DDOS to get more companies to pay for github:enterprise - Preventing a system upgrade in a company which has deployment relying on github There are lots of potential ways to use it...
- deleted 13y ago[deleted]
- daigoba66 13y agoI wonder if there is a way to work with git offline, and then synchronize later? /sarcasm
- thecodemonkey 13y agoWhat a coincidence. We actually just finished migrating all our projects over to GitLab about an hour before this outage. I still very much love GitHub, it just ended up not scaling for us (we have a lot of repositories that seldom needs to be touched, which results in a $20 / month Linode + Backup being a much better solution) GitLab also allows us to group repositories and gives a little bit more flexibility in regards of git server-hooks. Also, server-side branch locking! (Does anybody know how to lock branches server-side with GitHub?)
- briandear 13y agoJust a thought, but it would be interesting if there was a DDoS attack on the github status page. Then what would we do?
- troygoode 13y agofind out from @github
- tlongren 13y agoAppears to be loading fine now.
- xnxn 13y agoTime to read up on `git send-email`, I suppose.
- danielsju6 13y agoJust a PSA Git is distributed, fire up a quick instance with git daemon. git daemon --verbose --export-all --base-path=.git --reuseaddr --strict-paths .git/
- daemon13 13y ago>> Just a PSA Git is distributed Can you please elaborate?
- swede 13y agoPublic Service Announcement You can have multiple remotes for your Git repositories, one of them (Github) being down should not affect your work since you can coordinate a new one with your team in the manner described above.
- vhost- 13y agoYep! I usually setup a mirror at bitbucket and set my local to push to both github and bitbucket and only pull from github. It worked well when github was down and I needed a branch from a remote developer.
- anemator 13y agoNice. I didn't know you could push to multiple remotes (in the same command). Could you give me an example? Thanks!
- vhost- 13y agoThe reply right under mine seems to be a reply to you. It has a stackoverflow link that shows you how to setup two URLs in your origin remote.
- holyjaw 13y agoCan't reply to the child to this comment, so I'll add this as a new child: Here's how to set multiple remotes. (See answer 2). http://stackoverflow.com/questions/849308/pull-push-from-multiple-remote-locations http://stackoverflow.com/questions/849308/pull-push-from-mul...
- joemaller1 13y agoSnarky comments about distributed version control aside, the bigger problem is the ecosystems which revolve around GitHub. Two which are immediately screwing my day up are Composer and Homebrew. Currently, both are totally dependent on GitHub.
- postit 13y agoWhy in hell is anyone so pissed with GitHub?