25 ms·
Denial of Service Attacks
- robgering 13y agoI'm not sure why someone would attack GitHub. Extortion? But aren't there more valuable targets? Showing off their botnet, perhaps? These attacks seem frequent.
- Zikes 13y agoUnfortunately there may not be a "good" reason. It's not hard for me to imagine that someone could do it for kicks, or perhaps to test some new attack vectors against a reasonably hardened target.
- ansible 13y agoIt's not hard for me to imagine that someone could do it for kicks, or perhaps to test some new attack vectors against a reasonably hardened target. Though when you try out a new attack vector, the community (hopefully) publishes enough details about the attack to help the next target more effectively deal with that particular attack. If someone is attacking whitehouse.gov or a similar target, I somewhat understand their reasons why (though I don't agree with them). github.com, on the other hand, while a for-profit corporation, is also a valuable bit of Internet infrastructure that makes the world a better place. Attacking targets like github.com, Wikipedia, and others helps no one, and forwards no coherent political agenda. So I'm going with the "for kicks" / jerkwad theory.
- bestdayever 13y agoA lot of times, amateur "hacking" groups will require new members to prove their worth in some way and I'm sure github is a great target to test their "skills"
- nenolod 13y agoUsually it's some kid who wants to show off their botnet (or, more likely their $5 booting service investment) to their friends on hackforums. I deal with this crap all the time.
- leakybucket 13y agoPerhaps the attackers are drawn to sites they believe will later publicly document the attack, to learn whatever they can on how the operations team responded.
- natch 13y agoJust as anti-virus providers are sometimes suspected of creating viruses to help drum up business, I wonder if this could be a case of anti-DDOS service providers either doing some nasty marketing, or, looked at another way, running a protection racket.
- AYBABTME 13y agoLots of businesses heavily relies on Github for their operations. Sure, Git is distributed, but there's more to Github than just Git; think of continuous integration/deployment tools that integrate with the service, PRs. Attacking Github means you deny many more than only Github as a target. Which I guess, makes it more valuable than another target.
- trentmb 13y agoIt surprises me that GitHub doesn't sell/license a 'GitHub Appliance' that can get installed locally, but mirrors with something on their side too.
- eli 13y agoIsn't that Github Enterprise? You install it fully on your own servers.
- nathan_f77 13y agoThey do [1] [1] https://enterprise.github.com/ https://enterprise.github.com/
- VeejayRampay 13y agoI always wonder this myself. Were I skilled enough to be able to execute such big scale attacks against someone or something, I wouldn't think of Github as a potential target. Github is just some code, it doesn't bear any political or financial weight. But then again, some people get a kick out of ruining stuff so why not.
- noja 13y agoIt just shows that we need some kind of distributed version control system.
- sanderjd 13y agoHa! This is good satire, but for me personally, github going down isn't a version control problem nearly as much as it is a project collaboration problem. I can't go review pull requests and discuss issues when github is down, but I can still do all the traditional version control activities. Github is so much more than distributed version control. If somebody started doing the non-version-control things that github does in a distributed fashion, I would be very interested in taking a look.
- j-hernandez 13y agoI am still laughing at this comment.
- ilaksh 13y agoActually there are quite a large number of researchers who have concluded that the underlying architecture of the internet itself needs to be more distributed. They call it data oriented or name-based networking, content centric etc. The hardest part about switching over to a fully distributed internet is to maintain existing business models. Either someone will figure out and successfully market a business friendly universal data/computing distribution distribution system/network, or we will see free ones pop up that start to supplant more and more centralized services that are associated with particular domain names on the traditional internet. We are also eventually likely to see very strong push back against ISPs that overcharge for business internet. Well, at least in a sane and just world all of this would come true shortly.
- IgorPartola 13y agoI honestly feel bad for the engineers at GitHub for having to deal with stuff like this. GitHub is large, so they are a target, and the specifics of what they do means that caching is not a straightforward task. I imagine there are a lot more vectors of attack that have not been used yet and guarding against them is always going to be on a case-by-case basis. In the meantime, when GitHub is having downtime or even badtime it impacts its users pretty significantly. The private repo's I work on are a source of income for GitHub, but if this gets common enough the people in charge might just move away from it to a smaller competitor that doesn't have these problems just so that my time is not wasted on waiting on GitHub to come back up.
- anaphor 13y agoIsn't the whole point of git that you don't need to even have internet access to get work done?
- valarauca1 13y agoI thought the error message was "Commit locally github is down"
- cytzol 13y agoYou still have your local source code, but GitHub gives you issue tracking, comments, pull requests, and other things you could still use for work - none of which can be cloned locally.
- anaphor 13y agoOkay yes, but it probably isn't going to be down for more than a few hours at a time. I'm sure pull requests and issues can wait a few hours vs. actually fixing bugs or writing new code. Right?
- cytzol 13y agoYou'd have to hope so! I wanted to use the site, but having it down didn't bother me. But if I depended on it for my day job, I'd want more than just my source code available.
- api 13y agoIs there any way to mitigate DDOS attacks systematically without sacrificing network neutrality?
- nenolod 13y agoDirect peering with the eyeball networks helps a lot, as you can use smaller links (and thus, smaller scrubbing devices) on them. I do not believe that it sacrifices network neutrality, really. Comcast is really in a class of it's own regarding one-sided peering policies, but the other providers like Cox for example are fairly easy to peer with.
- jsmthrowaway 13y agoIf the majority of ASNs in the world followed BCP 38, these attacks would be more difficult because the origin would be easily identifiable. Today you can't tell where it's coming from because backwater networks see value in letting their customers emit forged packets however they like. So all you can do is mitigate and wait for them to move on. BCP 38/RFC 2827 would change the DoS game, but it's been a best practice for longer than most of this audience has been alive and nobody yet gives a shit and/or they are too lazy to automate the implementation. So operators waste their lives cleaning up after bad actor ASNs that they can't even identify. I shouldn't be mitigating 65 Gbps destined for a controversial customer, the attacker should be removed from the Internet before I even notice. You can tell from my tone that attacks are part of life for me. I'd venture that denials are the second largest problem facing the Internet today, behind the organizational structure of critical systems like DNS and ahead of spam and surveillance. However, there is now a sizable DoS prevention industry so I wouldn't be surprised if BCP 38 drifts into even more obscurity, but that's the cynic typing.
- kbuck 13y agoActually, since this attack wasn't volumetric and was instead attacking GitHub's (TCP-based) applications, they have the rare ability to identify the attacker's drones and possibly hand the list off to someone that can get them shut down. Hopefully GitHub does the right thing here.
- caio1982 13y agoKudos to the folks at Github for such summary of the attack! Clear, with a decent amount of info and honest.
- deleted 13y ago[deleted]
- muaddirac 13y agoI'd be interested to know who their "DDoS mitigation service provider" is.
- dangerlibrary 13y agoCloudflare is a very popular choice.
- pronoiac 13y agoYes, but if you largely deal with non-web protocols - like git - Cloudflare's much less effective.
- xxdesmus 13y agoWe have solutions that would definitely work for GitHub. :)
- jgrahamc 13y agoIt is not CloudFlare, but I would be fascinated to learn some more technical detail of these attacks as I work on systems to block this type of Layer 7 attack.
- eli 13y agoVery popular among consumers. I think large organizations are more likely to pay someone like http://www.prolexic.com/ http://www.prolexic.com/
- Fomite 13y agoI read this as 'prilosec.com' and was like "Yeah, if I worked for a really visible organization, I'd probably have an anti-heartburn medication habit."
- gandalfu 13y agoI have seen the folks from prolexic at work and they service/platform is impressive. On their spare time they take down botnets: http://www.prolexic.com/knowledge-center-ddos-vulnerability-disclosure-dirt-jumper.html http://www.prolexic.com/knowledge-center-ddos-vulnerability-... http://arstechnica.com/security/2012/08/ddos-take-down-manual/ http://arstechnica.com/security/2012/08/ddos-take-down-manua... Shameless plug for hackmiami, if anyone is interested in learning how its done up and close they run frequent talks/meetups locally: http://hackmiami.org/ http://hackmiami.org/
- eik3_de 13y agoTo GitHub and everyone: please use UTC timestamps when there are potential readers outside of your timezone. Since every technical person should know their current UTC difference, calculating the local time is easy.
- noja 13y agoIf anyone wants to convert from a given time zone to your local time zone, you can use: date -d '2014-03-14 14:25 PDT' Google does this conversion automatically on their own outage pages.
- agwa 13y ago3 letter timezones are ambiguous though. For example, Australia's east coast timezone is "EST"/"EDT" just like in the US. If you want to present time in a non-UTC timezone, you really need to use syntax like "UTC-0700".
- slug 13y agoFrom the 'date' command manual, there's more readable alternatives: show New York time 14:25 in your local time zone: ~$ date --date 'TZ="America/New_York" 2014-03-14 14:25' show New York time 14:25 in Alberta time zone: ~$ TZ="Australia/Alberta" date --date 'TZ="America/New_York" 2014-03-14 14:25' On a GNU/Linux system, for more timezone strings see ls -Ral /usr/share/zoneinfo/
- vacri 13y agoIt's bad customer service to require a user to crack open a terminal in order to understand your announcements. UTC is coordinated universal time. It's a standard. Just use it.
- lstamour 13y agoBetter still would be to use some kind of JavaScript that converts the time into the local browser's time zone. Or a link to such a service.
- crashandburn4 13y agoAm I the only person that gets slightly annoyed whenever I read "an order of magnitude" and the article doesn't mention whether it's binary or decimal. What do you people think they're talking about, I'm guessing decimal order of magnitude?
- valarauca1 13y agoA binary order of magnitude is only doubling, which doubling traffic isn't exactly a DDoS just a busy day and likely within the spec of their current network to handle. A decimal order of magnitude is a factor of 10. And would likely represent a problem. Really not hard, you don't use the term 'order of magnitude' in binary. Instead opting for 'bit shift', or doubling.
- pessimizer 13y agoDoesn't matter to me, 2^X and 10^X are close enough to each other for pretty small to large values of X. When I say it, I usually mean something between x5 and x15. I think avoiding the temptation to false precision is more important. Inconsistent units and the retention of insignificant digits in order to make numbers look bigger (or smaller) drive me up the wall, though.
- marcosdumay 13y ago2^10 = 1000 10^10 = 10000000000 Not that close, if you ask me. If you mean something between 5 and 15, you are using base 10. *16 are 4 orders of magnitude in binary, but 1 in decimal. That said, everybody asking that same question, please, do not use binary orders of magnitude. Our language suffers every time somebody does that.
- valarauca1 13y ago3-5 orders of magnitude is a decent margin of error for things like astrophysics.
- stcredzero 13y agoI said something like that to a hot redhead geologist once, and she laughed in my face.
- ozh 13y agoCall me naive but I fail at imagining why would someone want to DOS Github. I mean, if you're into this, it's certainly fun to launch DOS attacks against large "evil" things such as government services, large corps and Micro$oft becoz w1ndoz sux0rz, but... Github? Why?
- mentos 13y agoI was wondering the same thing? Maybe Github competitors?
- sytse 13y agoI'm running GitLab and the first thing I think when I read about their DDOS is 'how long before they attack us'. DDOS attacks just cause everyone a lot of pain and it would be great if nobody needed extensive countermeasures, DDOS operators waste everyone's time and resources.
- bsamuels 13y ago-criminals demanding ransom -"our agency has received intel that this site is a spawning ground for computer hackers" see: freenode ddos
- 72deluxe 13y agoPerhaps you have a botnet and want to test its capability on a "small scale"/unimportant target before unleashing it on a larger target? Sort of like testing without getting massive amounts of repercussions.
- rplnt 13y agoCompetition would be my first guess. Or an active employee somewhere that doesn't want to use github: "See, it's offline, we should roll out our own solution/keep using what we have". Or just testing out botnet on an appropriately sized target. Or anything else really. There are dozens of valid reasons.
- todd3834 13y ago
- kclay 13y agoI find it odd that github can even be subjected to DOS attacks, but it seems its only HTTP traffic. I also wonder why or if it is even possible to DOS the raw tcp layer of the git protocol.
- marcosdumay 13y agoYou can DOS anything that has a network interface.
- larrys 13y agoWondering if, for a service like github, it would be possible to setup a whitelist of allowable ip addresses. If an attack was launched only that whitelist would be allowed until the attack was mitigated. So while certain legitimate traffic would be blocked for sure, people who connect through fixed ip addresses that were whitelisted would get through and be able to do what they needed to do. Thoughts?
- aroman 13y agoSeems like it'd be pretty trivial to circumvent that. Just have your botnet do a few regular old requests to the network a few days before launching. That way the IPs of the botnet members get white listed. For a website of GitHub's scale, I don't think it would be very effective, though maybe it could be helpful in combination with other measures.
- larrys 13y agoNo, I'm specifically talking about clients who have signed up entering in their ip address that they access from. "Just have your botnet do a few regular old requests to the network a few days before launching." Not talking about "whitelist sites that have made access in the last x days". For example on HN it would be easy to create a white list. They do it now recognizing new people who signed up and keeping track of activity as well (by points). You could either have people identify the ip address that they accessed from and further limit the whitelist to a certain period of time and activity additionally. The idea is not to be 100% perfect but enough so that if you are a regular user of github from an IP address at your office (as opposed to wifi cafe) you will be able to get through. This is, by the way, how registries limit access to their system. It's all whitelist you have to pre identify the ip addresses that you will access the system from. The whitelist only comes into play when under attack. And for sure yes if you are connecting from a new place you will be blocked. But others will not be blocked and there will be some access for some people.
- randunel 13y agoThere are ISPs who change your ip address every 10 seconds, not just on reconnection. This would complicate github too much, I'd rather have 2 hrs downtime every now and then, than have to input my ip address classes from all the locations :D
- julesbond007 13y agoI'm quite surprised this happened to github...Sometimes I'm trying to look at some repos, but I apparently click too fast and have to wait before I can do other things. I thought they had ddos attacks under control.
- xedarius 13y agoYou ever sit there and wonder who the person is on the other end of the attack? Someone sitting there, I guess with not much on that day, decides to command their army of infected bots to attack github. Why github I wonder? Perhaps it provides a challenging target. Perhaps github is used as a testing ground for a more profitable future attack. We often get technical writeups after a DDoS attack, however we very rarely get a writeup sumising the motive behind the attack. I can't believe every attack is simply driven by 'because they can'.
- chrsm 13y agoI met some folks from GitHub last year, and this is what they postulated as well.
- baldfat 13y agoIt just isn't even possible to say anything specific in terms of anonymous person's motivation. If they had a name and face that is the only time you can talk with some certainty. Perhaps some developer was not going to make his deadline or wanted to take down the network to give him more time? Who knows...
- erichurkman 13y agoGithub is a valuable target if attackers are trying to get access to private repositories. A lot of organizations have their entire code base on Github.
- stcredzero 13y agoHow is a DDOS attack going to help with that?
- doktrin 13y agoI'm not entirely sure why this question was downvoted. How exactly would a DDoS attack help attackers compromise the target system?
- jacquesm 13y agoThe smaller a service is the easier it is to mitigate such attacks. All kinds of tools that smaller services can use (whitelists, software based filters such as iptables, location based filters and so on) are not available once you cross a certain level of scale. So any simplistic solutions that you might think of for a smaller service will likely simply not be applicable.
- deleted 13y ago[deleted]
- lauradhamilton 13y agoWTF is wrong with people attacking github and meetup. DDoSing a government site I can understand, sure. (Aaaand now I'm on a list.)
- xwowsersx 13y ago> In addition to managing the capacity of our own network, we've contracted with a leading DDoS mitigation service provider. A simple Hubot command can reroute our traffic to their network which can handle terabits per second. They're able to absorb the attack, filter out the malicious traffic, and forward the legitimate traffic on to us for normal processing. That's kind of awesome
- gandalfu 13y agoThis is what we use at our company. Recently bought by Akamai. http://www.prolexic.com/why-prolexic-best-dos-and-ddos-scrubbing-centers.html http://www.prolexic.com/why-prolexic-best-dos-and-ddos-scrub...
- geovizer 13y agoGitHub has been targeted by the Chinese government hackers before, with a man-in-the-middle attack, and blocking GitHub with the Great Firewall. Maybe they are at it again? http://www.theregister.co.uk/2013/01/31/github_ssl_man_in_the_middle_attack/ http://www.theregister.co.uk/2013/01/31/github_ssl_man_in_th... https://en.greatfire.org/blog/2013/jan/github-blocked-china-how-it-happened-how-get-around-it-and-where-it-will-take-us https://en.greatfire.org/blog/2013/jan/github-blocked-china-...
- coops 13y ago"A simple Hubot command can reroute our traffic to their network which can handle terabits per second." Really? You have to round-trip through Campfire to control your network?
- devicenull 13y agoWhy wouldn't they do this? They presumably want someone to look at the attack before engaging the protection, and I'm sure not all of their staff is able to make network changes. If they've got it automated to the point where a single command can do it, what does it matter via what method they use? If they're all in Campfire anyway, there's no overhead here.
- imbriaco 13y agoIt's just the most efficient and visible way for us to do it, it's not the only way. Here's a couple of reasons why we like it: 1. It's scripted so you don't have to think about it at 3am. 2. The rest of the team can see it happening in realtime so you don't have to explain what you're doing via a side channel. They can see it happening. 3. It doesn't require specialized knowledge of routing to enable it. If the on-call engineer sees an attack and calls someone for guidance, it's super easy to tell them "type /mitigation enable" for instance. 4. Of course we can run the exact same script or login to our routers and manually change our BGP announcements if we need to.
- zaroth 13y agoIf the attacks against Github are mostly proving grounds for fledgling DDoSaaS, I would assume write-ups like these only serve to elevate their status as a good proving ground. Did this article contain anything particularly useful for anyone thinking about DDoS hardening? I didn't find anything. I guess it's not really supposed to be a technical article, just a smattering of buzzwords to let you know how hard they try. The postmortem-half-apology has become quite an art form; as getting it right can actually draw a lot of positive publicity, and getting it wrong can be brutal. But I can definitely see how this post would feel like a pat on the back to whoever launched the attack.
- asolove 13y agoI was going to disagree with you, but then I realized I didn't understand what you were saying: What do you suggest they should have done?
- zaroth 13y agoGithub downtime (and subsequent postmortems) are a regular feature of the HN front page. The postmortems have come to command their own audience, similar to the CloudFlare reports. It's actually a pretty bad position Github's being put in. They sit at the crossroads of playing defense against DDoS and trying to dispel or at least ameliorate any blame for the downtime. My point was, if they have indeed become the internet's DDoS proving ground (as several others were speculating), then while you can see how much effort they're putting into these postmortems, I can see it becoming a vicious cycle. Then the challenge is, how does Github placate their users without basically pinning a ribbon on the attacker? The funny thing is how the "best practice" checklist for a postmortem (say what happened, say how you thought you were safe, say how something unexpected broke your assumptions, apologize, say what you're doing differently in the future) basically ties their hands. A pretty bad position for Github all around.
- jshen 13y agoThere are likely many people who have not experienced this form of attack. Many who may not have even been aware of it explicitly, and this article may have made this aware of it, and what the common strategy is for dealing with them. That's good, and so many developers user github, so the article likely reaches a larger audience than those "thinking about DDoS hardening". What isn't useful is the narcissism of your comment, and the assumption is that everything should be targeted at you and people like you.
- Aloisius 13y agoIt is too bad ICMP Source Quench couldn't have been repurposed to help deal with these kinds of attacks. It would be extremely nice to be able to simply send a packet to each host involved in an attack and have them (and optimally routers in between) slow their rate to the target host.
- scurvy 13y agotl;dr We're bad at detecting and handling layer 7 attacks. We're better now. Dear github dudes, netflow is your friend.
- csense 13y agoWhat motive does the attacker have? There are lots of articles on HN about DDoS attacks on various websites or online services. Most of the discussion is about the bandwidth used and the technical mechanics of the attack and defense. This is interesting, but there's little discussion of the economic motivation. I assume the kind of infrastructure used to launch this attack is not free. I understand people or groups might be using this as a way to further various political agendas or simply for bragging rights. I also understand DDoS attacks might be an extortion tool. In the former case, wouldn't the attacker try to loudly and publicly claim responsibility? In the latter case, wouldn't the defenders take pride in their "we don't negotiate with extortionists" stance while they're in disclosure mode? Or maybe this is just some rich guy's private hobby, and he does it for the amusement he gets out of reading about people's reactions when they can't figure out who's responsible? It seems like the set of rich guys who have the technical skills to do this kind of thing without getting caught would be kinda small. And if they hire people, the bigger their organization gets, the likelier they'll hire a law enforcement plant -- or simply someone with a conscience -- and the game will be up. Organized crime might be a possibility, but I assume those guys are interested in making money, not just committing crimes and wreaking havoc. So what's the business model that motivates these attacks? If it's extortion, why do the targets feel comfortable revealing the attack, but uncomfortable revealing they're being squeezed for money?