5 ms·
GitHub: About This Week's Availability
- robbiet480 15y agoWho would DDOS GitHub and for that matter, why?
- 1123581321 15y agoI'm not saying 37signals did it, but I bet someone who read this post did: http://37signals.com/svn/posts/3099-benchmarking-basecamps-uptime-against-five-other-web-apps http://37signals.com/svn/posts/3099-benchmarking-basecamps-u... I'm sure it was tempting more than a few people with the resources. Edit: just to be clear I meant any reader of signal vs. noise could have seen that (and Jesse's #1 comment and the jokes about DDOS in the comments) and decided to pull a prank. 37signals would never do that to anybody.
- dhconnelly 15y agoThis seems a bit far fetched.
- jnewland 15y agoMy wife is pretty convinced I brought this upon myself with my comment on that post. Lol, you never know.
- 1123581321 15y agoI guess I didn't display my 37signals fanboy card prominently enough.
- imajes 15y agoextortionists. There's a growing band of pirates (the web kind) who hold sites to ransom via their massive botnets, in exchange for payment -- 'protection money'.
- pavel_lishin 15y agoBut why Github? Wouldn't a less tech-savvy business be much more likely to cave in, and have a better cash flow? (e.g., porn and gambling sites?)
- wahnfrieden 15y agoUptime is very valuable to github. For example, I can't easily deploy to my site when githubs down. I suspect many others have bought into its availability too.
- cdelsolar 15y agoI had a very slow deploy earlier this week; if it had gotten canceled in the middle of it that would have been pretty bad. Maybe the deployment script should copy the repo head to one of my own servers and continue from there? What do people do?
- thibaut_barrere 15y agoRandom thoughts but: - Github are obviously very successful and may have money - their clients are reacting quickly on twitter so attackers might think they have an edge in terms of pressuring
- reinhardt 15y agoNot in that line of business but I'd expect any porn and gambling site with half-decent cash flow to be quite tech-savvy when it comes to securing it from all the DDOSers, fraudsters and other hostile entities they have to deal with daily.
- ketralnis 15y agoSure, which is why you attack them too.
- baddox 15y agoDo any of these extortionists have a reputation for actually stopping the attack once they get paid? I certainly wouldn't trust some random group that's currently DDOSing me enough to pay them.
- rplnt 15y agoBesides reasons mentioned, it could benefit their competitors. Remember that sourceforge was under heavy ddos not even a year ago.
- frisco 15y agoI remember reading a comment a long time ago -- maybe here or Quora, I think -- talking about how people don't usually realize how much trouble organized crime gives small and midsize businesses. You'll start getting DDOSed and then an email appears a day later: pay us and we'll stop bringing down your site. I have no idea if extortion is what's happening here versus simple lulzseekers or a grudge someone holds against them, but I wouldn't be surprised.
- jrockway 15y ago"Organized crime" or just regular crime?
- pavel_lishin 15y agoAs I understand it, eastern european criminal organizations (I hesitate to use the term "mafia") control pretty big botnets, and use them to do exactly this. Although github is sort of an odd target for them to be going after. Maybe their hired geeks are just testing their own software - github seems like the sort of place that would figure out how to defeat a DDoS attack, and by learning how, the black hats could improve their tools.
- bdg 15y agoIn some countries that are absolutely poor, it is entirely common to have the local mob recruiter look for young talented kids that they can bring into their organization. They make them an offer when the kids are out begging for change or something like "we'll take care of your mom and make sure she doesn't break her legs if you come and get an education in our school, we might even pay you." The youngster ends up in basically a run-down warehouse full of other kids each taught how to look for basic exploits like mysql injection or use root kits, and they sit around and do this all day. This is why some countries are typically COEs for hacking, such as Russia with crimeware kits, and until recently Brazil for bank hacking. This is also why you need coordinated efforts of special agents armed with assault rifles to clean out some spam-mills sometimes (they're taking them away from some mob). What I've described is a bit outmoded, typically this behavior is turning into other things such as this: http://abh-news.com/cybercrime-china-hackers-training-camp-closed-866.html http://abh-news.com/cybercrime-china-hackers-training-camp-c... as well as the makers of exploit scripts are just getting better and recruiting these kids doesn't make sense any more... you can intelligently scan for millions of sites using phpmyadmin and automatically run your exploit with it. As with any crime you find the trends all over the map: not all robberies are mob related, not all of them aren't, some of them are more elaborate than others, etc.
- robbyt 15y agoGithub is really great, but I'm a cynic. Indeed, why _would_ anyone want to DDoS github? How can we believe Github that the outages were due to a DDoS? They're smart people, aren't they using solid load balancesrs that can mitigate DDoS attacks? Why haven't they issued an actual statement describing the supposed attacks in better detail?
- mike-cardwell 15y agoThat's not just cynical; that's down right paranoid.
- dhconnelly 15y agoAs another poster mentioned, DDOS attacks on successful companies are not uncommon. And there isn't an easy fix for this.
- jtchang 15y agoLoad balancers generally don't offer as much mitigation against DDoS as you think. The load balancer will probably fall over if the backend servers don't do so first. The sheer volume of seemingly valid traffic is what kills you. Which is why effective anti-DDoS means working with your upstream provider to figure out how to differentiate between real traffic and fake traffic. The best solution to a DDoS is the phone # of a tech that can implement firewall rules upstream and a good traffic analyzer to tell them exactly how to filter it.
- swalberg 15y agoOften you can't pick apart the good traffic from the bad. Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect. After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule: 1. Check for the IAMNOTABOT cookie 2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access 3. Set IAMNOTABOT=true cookie 4. Redirect to the old page
- AznHisoka 15y agoCouldn't they just keep adding IP's to some blacklist (not htaccess, but something more efficient), and the attackers will eventually run out of IPs to attack from? There's only a finite # of computers you can really have control of. So you end up blocking some innocents, but you take care of that after the attack is over.
- rcthompson 15y agoI thought that botnet operators had methods of rapidly changing bots' IPs so as to defeat IP blocks.
- abraham 15y agoIf it is a smart DDoS the requests won't look any different than typical users visiting your site.
- Tobu 15y agoMaybe a whitelist could work. My IP has visited in the past week / I have a legit session cookie, now gimme bandwidth.
- brown9-2 15y agoThis only works if you decide you want to never accept a new customer again.
- AznHisoka 15y agoThis is only to block the attack. The thinking is when the attackers run out of PC's they will give up. You lose some customers in the short-term. If you don't do any blocking, they will just keep attacking and attacking.
- AznHisoka 15y agoThis is only to block the attack. The thinking is when the attackers run out of PC's they will give up. You lose some customers in the short-term. If you don't do any blocking, they will just keep attacking and attacking.
- IanDrake 15y agoI wonder if something like http://www.cloudflare.com http://www.cloudflare.com could help.
- dbecker 15y agoIf I were a DDoS attacker, I wouldn't want to make enemies with the sort of people that rely on github. Though it's easy to imagine that those guys think differently than I do.
- ellie42 15y agoI think if you build your app properly the only problem with DDoS is traffic. Github partnerships with Rackspace so their cloud traffic can't be expensive. Correct me if I'm wrong.
- newobj 15y agoI suspect you may be oversimplifying the solution or undersimplifying the nature of these kinds of attacks. What would constitute a 'properly' infallible defense to you?
- tworats 15y agoAre there any good guides on how to combat a DDOS? I'm curious how a team of good technical people without past experience in the area could react.
- zxoq 15y agoFighting a DDOS is very hard if you do not have the resources to purchase more bandwidth. The DDOS attacks I've dealt with have been almost exclusively UDP-floods in the 500-2000 mbit range. No amount of server-side configuration will fix that unfortunately. If the attack is some more creative form, but with less bandwidth (SYN flood etc.) a cisco router can help out a lot. Your best bet is to move to a protected hosting provider (dragonara or ddoshostingsolutions are quite good for Europe / US). Failing that you can purchase a bunch of VPS servers, round-robin DNS to them and reverse proxy to your actual server (and keep that IP off the DNS system entirely). This way only the proxy servers will go down if you are attacked. Edit: If they are not spoofing they sender address, you can achieve some success by mailing the hosting provider of the IP performing the attack. In 90% of cases the server has been hijacked and they will shut it down, this only works for US / European companies though since Chinese / Russian hosting providers never ever reply in my experience. For maximum success you can attach a tcpdump of the attack traffic.
- newhouseb 15y agoI was poking around a botnet today and saw that the particular one I was looking at was just DDOSing DNS (port 53). Could you just use an /etc/hosts file and turn off UDP entirely? My understanding is that part of the problem is that the reason 53 is so vulnerable is because the OS sits there waiting for any UDP replies from DNS on it.
- zxoq 15y agoIf your server is running slow because of the load, and not because of the bandwidth of the attack that might be a viable route. It is trivial to use iptables to block all UDP traffic except from your configured DNS server. Most of the cases though it's as simple as attack is 200 mbit and connection is 100mbit -> no amount of dropping packets once they reach the server will do anything, as the connection is entirely saturated.
- jvoorhis 15y agoI've used Github frequently all week,personally and professionally, and if they temporarily lost a nine or three, I haven't noticed.
- pron 15y agoVery nice. Just one thing: I already know I'm awesome and I don't need my source revision control provider to boost my self confidence (I'm referring to the line "...you, our awesome GitHub users"). OTOH, it's clear that it's GitHub's employees, rather than its users that need constant reassurance of their coolness. GitHub is a nice company that provides a good, solid and necessary service - hosted SCM. Then why is it that their company blog focuses on their drinking habits? Dear GitHub, Although your product may sound boring to laymen, you provide a good service to a very important industry. Your "boring" day job reflects nothing on your personal and very exciting lives. Your customers, however, like you for what you provide them with, not for your companionship, and I'm sure your friends like you for the opposite reasons, as they very well should. In fact, everybody loves you for many reasons, and you are all very lovable. So please, keep your extra-curricular activities to your friends, and your work activities to your customers. You can call your friends "awesome" if that's your kind of thing, but for various reasons it is better to treat your customers with proper decorum. If you're unable to keep the two separate, you're in for some bitter disappointment later in life. Love, everyone.