17 ms·
We are under attack
- abfan1127 12y agoAre you using Cloudflare?
- dewey 12y agoA lot of CloudFlare IPs are blocked by the chinese firewall, for a site that's primarily aimed at chinese users probably not an option. Edit: I don't actually know if this is still true and on what scale, I just know that's it's true for a website I use according to chinese users.
- mirashii 12y agoEven if it is true, getting an enterprise account, which is still very reasonably priced, gives you dedicated IP addresses to your site, so this shouldn't be a huge concern. Generally when under fire, Cloudflare is also happy to get you up and running and talk finer details on billing for the long term later. The bigger problem is their stance on Latern mentioned above.
- mirashii 12y agoHonestly a bit confused on the downvotes, was just trying to provide additional information for people who aren't familiar with working with CF. Anyone care to enlighten me?
- d0ugie 12y ago> Because of the number of requests we are receiving, our bandwidth costs have shot up to USD $30,000 per day. Perhaps the US State Department might be inclined to help?
- xnull6guest 12y agoThis is the correct answer. Given that this website is part of the 'civil society' sphere, it likely already gets taxpayer money from the State Department. They should go to their funders - be they public or private - and ask for help with their mission - I'm not sure it's appropriate to ask individuals for help.
- NegatioN 12y agoBut the funders won't solve this problem though, only mitigate it. (though I agree it's something they definitely should do ASAP) Putting it out there like this, seems more likely to overcome the technical difficulties of it.
- e12e 12y agoI nice anti-China spin might be part of the mission...?
- revelation 12y agoIf you're paying $30000 for a site that is pretty much all static in bandwidth costs a day, you're probably paying about $29990 too much.
- random_rr 12y ago2.6 billion requests per hour will do that to you
- brador 12y agoRoute them through a cached captcha page. Or just call cloudflare.
- tibbon 12y agoHaven't sites been DDoS'd that were using Cloudflare? Its good, but not a magic bullet.
- david_shaw 12y agoYeah, but my understanding is that it helps a lot. CloudFlare masks the actual IP address of the web site, and distributes the load through the CloudFlare platform. It's still possible to overload that capacity, but it's a lot more than what any standalone web server can take. Furthermore, the web server itself will stay online, as it isn't actually getting hit by the flood of requests.
- eli 12y agoServing a captcha page is more work than serving a static page.
- monort 12y agoYou can block with firewall IPs of users, who didn't solve captcha. You will get only SYNs from incoming connection requests then.
- stevecalifornia 12y agoContact Akamai who recently bought the DDOS mitigation service Prolexic. They may be able to mitigate the attack and save you bandwidth costs. Alternatively, call CloudFlare. Don't just absorb this through Amazon.
- moe 12y agoDon't call Akamai. Never call Akamai. They are horrible. Try Incapsula, GigENet, Blacklotus, Cloudflare first.
- falconed 12y agoCan you elaborate on why you think Akamai is horrible?
- moe 12y agoI've dealt with them in their CDN role on multiple occasions. They're the most abhorrent combination of incompetence and arrogance that I've ever met in the tech industry. They're the Oracle of the network world. Just don't waste your time on them. There's plenty better and cheaper CDNs nowadays. Leave Akamai to the Governments and MegaCorps, they deserve each other.
- stevejones 12y agoThis is the exact opposite of my interactions with Akamai.
- toomuchtodo 12y agoMy experience with them was exactly the same as OP. Wanted us to bring buckets of cash for 10-100TB of bandwidth. Went with CacheFly, super satisfied.
- e12e 12y ago
- nerdy 12y agoFirst a 2.6bn request/hour DDoS and then making the front page of HN... talk about getting flooded with requests. Hopefully making the front page will at least get them the attention of Amazon or enough donations to cover the temporarily (absurdly) high operating cost.
- stanmancan 12y agoThey claim they're able to handle the 2.5bn req/hour right now, and if thats true, the HN traffic wouldn't even be noticable
- nerdy 12y agoYeah there might've been a smidge of sarcasm in there
- deleted 12y ago[deleted]
- e40 12y agoWith a mission such as yours, I would think Cloudflare (or similar) protection is a must.
- WhitneyLand 12y agoI hate that this is happening, but isn't this something you have to expect/prepare for when your business involves controversy? Not to mention when the people who don't like you have the resources of nation states.
- cpncrunch 12y agoMove to OVH -- they offer free DDoS protection as standard, and unlimited bandwidth. I just moved to OVH after getting DDoSed. I'm paying $109/month for a quad core 3.7Ghz Xeon, 64GB RAM, dual 2TB software RAID. It's a pretty sweet deal, and I haven't had any problems so far.
- bovermyer 12y agoI assume you mean 64GB of RAM.
- cpncrunch 12y agoLOL yes, fixed. It's pretty nice...it basically means that our entire disk is always cached.
- arm 12y agoTo be exact, he probably means 64 GiB¹ of RAM (see IEEE 1541-2002²). ―――――― ¹ — http://www.wolframalpha.com/input/?i=1%20GiB%20to%20MiB%20and%201%20GB%20to%20MB http://www.wolframalpha.com/input/?i=1%20GiB%20to%20MiB%20an... ² — https://en.wikipedia.org/wiki/IEEE_1541-2002 https://en.wikipedia.org/wiki/IEEE_1541-2002
- MichaelGG 12y agoNo, GB is perfectly correct when referring to memory. Just because some people standardized on Gibibyte and redefined gigabyte doesn't invalidate what memory makers have been doing for ever. JEDEC still uses GB, as they should.
- moe 12y agoSitting down and writing a blog-post seems a pretty laid back reaction to $30k/day in Amazon bills... First thing I'd have done is take the site offline and call the various DDoS mitigation services (Incapsula, Cloudflare, etc.). Pretty surely most of them would gladly pick up the slack here, given the free PR (possibly even in mainstream press) they get in return.
- pi-err 12y agoNot sure that turning the light off is the first thing you want to do when you're bullied. And pretty certain that there's zero PR value in those stories.
- ikeboy 12y agoCompany under attack turns to this service to recover!
- johnpowell 12y agoTurning off the light is a great first step when wasted money is flying out of your wallet.
- chralieboy 12y agoThey wouldn't just be giving in, it would take less than a day to get up and running on CloudFlare. That is probably worth avoiding $30k bill, especially for a non-profit. As for PR, there absolutely is good will towards organizations that take on censorship. This is the #1 story on HN, so a company stepping in and saying "we got this covered, free of charge" not only shows they are good people, but also gives would be customers an idea of the quick implementation cycle for their CDN + the load it can handle. Real world use case + helping to stop censorship = great PR
- loganu 12y agoThis is exactly what I was thinking. I read the blog post, then came to the comments. I completely expected something along the lines of ... "Hey, I'm the VP @ SomethingTech, we like what you're doing and will help you mitigate the attacks for free if you get in touch with us." ... to be the top comment here.
- tlrobinson 12y agoIt would be interesting to see which IP space the bulk of the traffic is coming from. Seems like it would be trivial for the Chinese government to spoof traffic from any IP within China...
- Tossrock 12y agoFor a DDoS, you can spoof your IP to anything, because you don't care about actually receiving response packets. This is standard in a SYN flood.
- ramigb 12y agoI think they need to use something similar to this ... http://en.wikipedia.org/wiki/Coral_Content_Distribution_Network http://en.wikipedia.org/wiki/Coral_Content_Distribution_Netw...
- deleted 12y ago[deleted]
- ramigb 12y agoI would appreciate it if people who down voted my comment explained to me if it's because i'm wrong or because they don't understand what i am trying to say or just for the heck of it :).
- lucaspiller 12y agoThe issue is your comment doesn't really add any value. Anyone can paste a random link saying "you should use this" but it takes effort to explain why it would be useful to them. HN comments are about fostering discussion, so say something to be discussed :-)
- ramigb 12y agoThank you for your input, but 1. this is not a random link, 2. the explanation is all inside the link i posted, so why to be redundant? anyways, if i didn't care for a discussion i wouldn't post it at the first place and i wouldn't later ask why was i downvoted, thank you again for your opinion lucaspiller.
- ArtDev 12y agoThis is interesting. Though Hacker News appears to not be blocked, it has been flagged as "Contradictory" on certain days. Is the Chinese government blocking certain news items? Take a look here: https://en.greatfire.org/news.ycombinator.com https://en.greatfire.org/news.ycombinator.com
- superobserver 12y ago> Is the Chinese government blocking certain news items? That would certainly be my guess.
- Kronopath 12y agoIf you click on the day in the calendar to look at the details, the "Contradictory" status is when some of their test servers work and others don't. For this site in particular, there are several servers showing a timeout and no data received. So it's possible that HN is being partially blocked.
- dsl 12y agoCensorship in China isn't just an all or nothing thing. There are content filters at the city, providence, and country wide level. Separate filters for traffic leaving the country vs internal. etc.
- nitrogen 12y agoIt looks specifically like cURL's exit value and the downloaded page size varied on some requests. It would be interesting to know what the contradictory download size was, and what the curl exit value was.
- Kronopath 12y agoAll that is there when you click on the date, if you scroll rightwards. The exit values in the blocked sites are timeouts (CURLE_OPERATION_TIMEDOUT), and the broken download sizes are 0 bytes.
- jgroszko 12y agoIsn't this title a little sensationalist without specifying who's under attack? I assumed it was a royal we and after clicking the link realized it was just this one site.
- rcthompson 12y agoIt's the title of the blog post that's being linked to. With that title being transplanted to HN, you need to consciously think about what the context is supposed to be, which is why HN lists the source domain.
- deleted 12y ago[deleted]
- pjc50 12y agoThat's not how international internet peering works though; it's entirely a set of private agreements. It's not even a given that the traffic has either Chinese IP source addresses or is actually from China.
- toaskaquestion 12y agoOut of curiosity, who pays more -- the attacker or the victim? Purely from a monetary perspective. Edit: never mind, figured it's obvious. Aren't DDoS requests pretty much simple GET requests? Is it not possible to determine which requests to serve and which ones to ignore?
- SeoxyS 12y agoThe attackers typically have a botnet at their disposal. The victim has to pay their own costs.
- twunde 12y agoThe requests can vary. Sometimes they are simple GET requests, sometimes they're exploiting a cpu|memory|io-intensive process in the application and sometimes they can be reflected DNS attacks. The problem is separating the legitimate requests from the bad requests. Sure I can see there are 5 Million requests to the main page of the app. But which are from poeple legitimately trying to use the application and which are from the botnet? You can't just do it by IP without running the chance you're going to cause problems for legitimate users. There are ways to mitigate this but it requires being able to analyse current traffic and past traffic quickly, and at scale while having the expertise to set up firewalls and other filtering correctly
- Sanddancer 12y agoThese days, DDoSes are not just lots of GET requests, because, as you said, they're fairly easy to mitigate. These days, the most common attacks are various UDP-based attacks, like NTP reflection [1]. You send a spoofed header to a server that speaks over UDP, and they send a huge amount of traffic to the victim. https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks/ https://blog.cloudflare.com/understanding-and-mitigating-ntp...
- lucaspiller 12y agoCouldn't something like that be blocked at a firewall level with AWS though, i.e. drop everything except TCP port 80?
- deleted 12y ago[deleted]
- Animats 12y agoPost the IP addresses from which you're getting attacked. Others can analyze them by ISP, and public pressure on the worst ISPs might help.
- Consultant32452 12y agoThe bad ISP in question is the People's Republic of China.
- Animats 12y agoI mean post the whole IP address list for public analysis.
- Consultant32452 12y agoHonestly most companies I've worked for have blocked the entire Chinese IP block. Obviously Chinese hackers can use proxies but it honestly does cut out a TON of problems. The downside of course is you will never get a Chinese customer/viewer. In preparing to respond to this post I googled "China IP block" and pretty much every result was about how to configure .htaccess or iptables to block the entire country.
- stevejones 12y agoThis thread brought to you by the CloudFare PR Agency.
- dataker 12y agoThey couldn't wait a day? http://gizmodo.com/china-finally-admits-it-has-an-army-of-hackers-for-cybe-1692188006 http://gizmodo.com/china-finally-admits-it-has-an-army-of-ha...
- LLWM 12y agoPerhaps next week they'll admit they have a navy.
- nitinics 12y agoYou should trace the attackers by tracing back. Work with your upstream providers and mailing lists (NANOG) and publicly shame these attackers. Likely, they are spoofing addresses - validate that and make sure you let the network know where the spoofed traffic is sourcing from to follow BCP38 and BCP84, defined by RFCs 2827 and 3704.
- Nyr 12y agoAssuming it is direct spoofed traffic and not a reflection, naming and shaming will accomplish nothing. Names of the big ISPs allowing this are not a secret.
- MichaelGG 12y agoTransit providers do not care. They make money on it, some people are using it legitimately, and they just don't care, for the most part. It's a well known problem. It might not hurt to mention it, but they know what they're doing.
- calbear81 12y agoState sponsored cyber warfare is something that happens at a scale that normal private companies and organizations are not well equipped to deal with. I wonder if there's someone you can alert in the government who can consider coordinating a counterattack or pass a backchannel note to the right people to cut it out.
- LLWM 12y agoCalling a simple DDoS cyber warfare is a bit hyperbolic when we know there are actual sophisticated attacks being carried out against far more important targets.
- loganu 12y agoIs there someone? Of course. There always is. Finding out whoever that person is and getting into contact with them is another story.
- morlockhq 12y agoHow effective would this solution be: http://www.linuxjournal.com/content/back-dead-simple-bash-complex-ddos http://www.linuxjournal.com/content/back-dead-simple-bash-co...
- blingojames 12y agoNice, thanks. How about running this script all the time, just in case there will be a DDOS?
- sah2ed 12y agoA similar approach is linked to in the article's comments in http://www.inetbase.com/scripts/ddos/ddos.sh http://www.inetbase.com/scripts/ddos/ddos.sh from http://deflate.medialayer.com/ http://deflate.medialayer.com/
- jbrun 12y agoCall http://equalit.ie/ http://equalit.ie/ - they have a free open source tool for exactly this!
- xixixao 12y agoGoogle's Project Shield could help? http://www.google.com/ideas/projects/project-shield/ http://www.google.com/ideas/projects/project-shield/
- formatjam 12y agoYes, Project Shield is the best solution.
- dreamins 12y agoNo one likes DDOSes from China. One can plead Amazon as much as one wants. Pay or get booted, there are probably 2 engineers paid 6 figures a year by Amazon getting paged for this DDOS, someone must pay for the time they spend tuning DDOS protection instead of their primary project to make attacked website accessible for everyone else. Source: worked for AWS, was oncall during similar attacks. Nasty things with those they tend to start around 6-7PM (guess when does working day start in China).
- simonebrunozzi 12y agoI used to work at AWS too. Where were you? Seattle / support?
- dreamins 12y agoSoftware engineer, I am pretty easy trackable in internets too :]
- toomuchtodo 12y agoI chuckled, because when everyone tells me "AWS is practically the internet" I can point out "The Internet is resilient at a far lower cost than Amazon".
- dreamins 12y agoInternet as a whole yes, making a single attacked web service resilient to DDOSes at a low cost is quite a challenge.
- powertower 12y agoWhy don't providers just set up a system that creates a country-level null route for a given destination IP? And have a UI with a checkbox for the user to do it, for any selected country. It would mitigate the issue, and once it's over, the user can un-restrict traffic / or just keep blocking if it's a non-valuable source. I know you can do this on the server, using many different techniques. But this does not help as the traffic still reaches you (that you have to pay for). You can also do this with Geo DNS (and get much less of a bill). And the ISPs, datacenters, and anyone with a router can block ASIA or China allocated IP ranges. Especially if it's not the type of a flood that's designed to attack the routers (instead of the web-server). So what's stopping Amazon?
- Xeoncross 12y ago- Nginx instead of Apache - Use memcached+nginx to load the drupal content instead of calling up PHP each request (PHP saves the page in memcached, nginx reads it from there). - put it all behind CloudFlare This works for Wordpress too.
- nathanb 12y agoShould the Chinese government have the power to shield their citizens from information and monitor them electronically? Should a group of people in democratic, Western countries be able to subvert the will of a world superpower with impunity? Of the two scary worlds, I guess I'd rather choose the latter. But I don't even like having to choose. (I doubt Amazon like being asked to choose even less, and I would be surprised if they cut you any slack. Sedition is not looked upon favorably, and abetting those perpetrating it is not either.)
- 1ris 12y ago>Should a group of people in democratic, Western countries be able to subvert the will of a world superpower with impunity? Crazy time we live it that this is even possible.
- logfromblammo 12y agoI'd suggest that the ability to perform a subversion of the will of a foreign state is a necessary adaptation, preventing a nominally democratic state from sliding towards aristocratic, oligarchic, or plutocratic governance. The ability to increase freedom in a foreign state is related to the ability to prevent a decrease of freedom in your own. Inconvenient websites help uphold the duties of the fourth estate when mainstream media outlets have seemingly abandoned--or at least heavily de-prioritized--those duties. It is important for all governments, not just those with sketchy human rights records, to know that even the mightiest machine can be taken offline by a single wooden sandal.
- gnarbarian 12y agowhen stuff like this happens it would be nice if we could simply refuse to route all traffic originating in the offending countries until the attack is over.
- me1010 12y agoyawnnnn... wake me when the clickbate is over.
- richieb 12y agoMaybe http://www.google.com/ideas/projects/project-shield/ http://www.google.com/ideas/projects/project-shield/ can help?
- ddgcd123 12y agoI think one of the companies which are responsible for the attack is :("Qihoo 360" or the "Company") (NYSE: QIHU) Most of the PC in China have install Qihoo 360. It is actually a spy software. It collect user information, I think it must be used to launch a DDOS attack. DON'T BUY THEIR STOCK!!!!
- rtpg 12y agoI wonder if the US gov't wouldn't mind donating $11m/year to this org to deal with the increased costs...
- datashovel 12y agoIt seems a little hypocritical to me that AWS will create a service for every technology known to man, but will not create a service to help companies who rely on their infrastructure to deal with DDoS.
- haosdent 12y agoCould we block this in ISP?
- EGreg 12y agoIsn't this what CloudFlare has built a reputation for defending against?
- methou 12y agoA number of the DDoSes from China are involuntary induced by DNS Poisoning. When users query a block dns name, they may receive an IP other than the website they want to visit. It may used to redirect traffic to make DDoS without those 'drones' even know about they're involved. Just like for a week long, whenever I try to access Facebook, I got redirected to some german IP, and receive a TLS CN mismatch error.
- disjointrevelry 12y agoI hope you keep us up to date. I live in the US and I've had accounts suddenly locked out after their passwords were changed even for criticizing some Chinese issue. Several email accounts, among other sites. With the amount of money the Chinese are spending, I think it's not going to be easy in the US. Maybe you're an American and things will be easier for you, but I've learned not to trust the natives. Not even their law enforcement. Maybe the corporations might be more stable and have more integrity, but you have to realize that China is the largest economy in the world now, and they can flick Amazon out of their market with their pinky and not even blink.
- NextPerception 12y ago"We need help to manage this. If you have expertise in this area, please contact Charlie Smith or ping us via Twitter." Step 1 : Unleash DDos Attack against target Step 2 : Wait for the the target to become overwhelmed and ask for public assistance Step 3 : Contact the target under the guise of being able to help Step 4 : Win trust of Target after "mitigating" the attack you are actually in control of Step 5 : Repeat until enough access has been gained
- anonbanker 12y agoThe more I watch the security theater, the more this all looks to be set-up. I really don't fear the chinese, as the NSA is far more competent and organized, and is much more dangerous to the world at large, not just the citizens of the United States. anyone remember that story a few days ago where China said "Hey, we want NSA-level backdoors in hardware, too!"? can someone help me with a link?
- secfirstmd 12y agoFor human rights, civil society and media working in difficult DDoS threat environments check out these guys, they are awesome ---> Equalit.ie https://equalit.ie/ https://equalit.ie/ who run a DDoS project called Deflect - which I think is free for people in those categories of people. https://equalit.ie/portfolio/deflect/ https://equalit.ie/portfolio/deflect/
- infinitnet 12y agohttps://news.ycombinator.com/item?id=9242710 https://news.ycombinator.com/item?id=9242710