8 ms·
What happens when DNS in China is poisoned and pointed at your server
- ChrisArchitect 12y agofor those wondering, extended title could be along the lines of "Experiencing the effects of a DNS poisoning attack"
- acdha 12y agoThis is why I really like having Varnish in front of anything Internet-facing: for something like returning a 400 for an un-approved Host header, you're going to be effectively bottlenecked by the network interface.
- thaumaturgy 12y agoThat's cute, the IP address for his demo curl command is ... daringfireball's web server.
- pronoiac 12y agoAnd if you run it, you get the Daring Fireball homepage, which surprised me - it's not configured for this problem.
- Anthony-G 12y agoI noticed that as well. However, they seem to be friends / colleagues / associates. A web search for "Craig Hockenberry daringfireball" shows that Craig Hockenberry is (or has been) a contributor to John Gruber's show and John Gruber has recommended one of Craig's apps [1]. [1] http://safetylightapp.com/ http://safetylightapp.com/
- coldcode 12y agoI wonder if this could really be harnessed in China as a weapon.
- brazzledazzle 12y agoThat's the only reason I can imagine they'd respond with anything else besides 127.0.0.1 or some internal web server with warnings to obey their overlords.
- beachstartup 12y agowhy is 'out' inbound, and 'in' outbound?
- faster 12y agoBecause it's from the perspective of the router; traffic out of the router is TO his network.
- beachstartup 12y agoif he owns/operates the router, it still doesn't make any sense. in our network, 'out' means outbound from the edge, and 'in' means in from the edge, in all cases.
- faster 12y agoThe bandwidth chart I get from my colo (where I don't own the router that measures my bandwidth usage) works the way I described. The article mentions a single server, and nothing about owning the router also. If they did, that's probably where they'd null-route the traffic they don't want. It seemed backwards to me too, when I first saw it.
- fallous 12y agoWe were the lucky targets of China's DNS poisoning on Monday the 19th. Any requests from clients in China to Edgecast's CDN was instead thrown at our public IP. After deducing what was going on, we ended up blacklisting large (class A in some cases) APNIC address blocks assigned to China. Interestingly, it wasn't our webservers that were overwhelmed but instead the Cisco firewall that sits in front of them. 25K concurrent connections made it decidedly unhappy.
- meowface 12y agoMy company's web server saw a similar unusual surge in traffic with host headers and URLs of Chinese sites for an 8 hour window a few months ago. All the IPs were from a single Chinese ISP, though the volume of the traffic was a lot lower than what OP observed. Seems like these events aren't uncommon.
- jlmorton 12y agoSame here. We received well over 150mbit/s, also on Monday the 19th, also from China. We also saw occassional spikes starting on January 9th. We saw the same BitTorrent /announce traffic, and lots of other random traffic. We ended up mitigating it by moving our IP address on that host, and blocking all input on the old address.
- themoogle 12y agoyou are lucky you only got 150mbit/s Here is what I had to deal with on the 9nth Ramming my server... http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png Yes that is almost 1000mbit/s All coming from china...
- broknbottle 12y agoasa 5505 with upgraded license?
- fallous 12y agoYeah, rackspace-provided unit.
- smilekzs 12y agoThey (the GFW) intend to have every server owner ban China IPs so that they could claim "you guys are also banning us, what position are you in to blame us for GFW?"
- pavel_lishin 12y agoBig ol' [citation needed] on that one.
- smilekzs 12y ago/* That's what he said */ Well, do you know what the Communist Party thinks?
- bm212 12y agothis seems like a clever way of censoring internet access from within China - get the site owners to do the censorship for you.
- emiunet 12y agoThanks a lot for the post. What suprised me the most was the fact that Google is serving 40K (now 47K in live statistic) requests per second. That was way below what I thought. Each Apache server without real optimization running on a typical physical machine would be able to serve 40-50 requests/second; so they only need a thousand of servers to do 50K requests/second. Or am I missing something?
- Narkov 12y agoYeah, the actual search bit. Serving a static file is pretty simple at 40-50 req/sec but searching an entire index of the Internet @ 40-50 req/sec is harder.
- intr1nsic 12y agoYour missing factor is "the fact that Google is serving 40K (now 47K in live statistic) requests per second" is wrong. "The number of requests peaked out at 52 Mbps. Let’s put that number in perspective". Perspective is 52 Mbps isn't a small number, but not massive either.
- noonespecial 12y agoNot particularly feasible but it would be awesome to simply proxy all of the requests to their intended destination.
- IgorPartola 12y agoOh man. That would be pretty cool. Somebody like CloudFlare could do that since they proxy for lots of smaller sites. Then again, HTTPS will add even more complexity.
- hobs 12y agoEven if it likely impossible for anyone smaller than CF, it might even mean they would move the attack off sooner since it wasnt doing anything :)
- throwawayaway 12y agothat would be good, or proof of something not widely known in china that is officially denied by china.
- bombless 12y agoI live in China. We have a department named 工信部, it blocks a site by polluting the DNS and point the domain name to random foreign IP. I think this explains what happened here.
- methou 12y agoA Chinese here, I'm deeply ashamed by this malicious action at a national level. though DNS poisoning wasn't new for Chinese netizens but it was always poisoning A records to varies foreign military sites like DoD or some Korea sites (which also makes no sense at all). At scale of this big, those shameless bastards running state censorship machines has been outrageously attacking on random innocent site owner, this is far beyond unacceptable. And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse. Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.
- sorpaas 12y agoAlso Chinese here, GFW is really a shame for us.
- jasonm23 12y agoMany of us outside GFW/Cn are aware of the "decoration" (we call it Astroturfing in the US) - it's so transparent that it's often joked about on Reddit and similar sites.
- fallous 12y agoI have seriously contemplated a special request handler for Chinese traffic that is being directed to the wrong host on our servers. Something along the lines of "Chinese government censors broke your internet" or the like. I figure it either helps inform the citizens as to why this link isn't working or the message itself causes the GFW to ban our IP as well.
- haosdent 12y agoToday, GFW start block VPN. So it only would become worse in the future..............
- deleted 12y ago[deleted]
- deleted 12y ago[deleted]
- userbinator 12y agoSeeing /announce reminded me of this: https://torrentfreak.com/bittorrent-users-ddos-websites-without-knowing-120919/ https://torrentfreak.com/bittorrent-users-ddos-websites-with... Using DHT is also a particularly effective way of getting massive amounts of UDP traffic directed at you, which does not stop even after stopping the torrents. It subsides eventually, but can be quite irritating.
- kesor 12y agoI wrote a blog post about a DDoS on a website by the Chinese at http://dvps.me/ddos-attack-by-torrent http://dvps.me/ddos-attack-by-torrent. Falsely I assumed that it was torrent tracker injection, while in fact this looks like a spike in the affect of DNS poisoning in China. http://serverfault.com/questions/656093/mod-security-block-requests-by-http-host-header/ http://serverfault.com/questions/656093/mod-security-block-r... describes the problem as well. https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepiratebayorg+Resolving+to+Various+Hosts/19175/ https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... also describes the same thing. As well as http://www.webhostingtalk.com/showthread.php?p=9351951 http://www.webhostingtalk.com/showthread.php?p=9351951
- kesor 12y agoOne possibility is to block requests coming with a different 'Host:' header than your website. Or just block the URLs like described in dvps.me/ddos-attack-by-torrent
- kesor 12y agohttp://en.wikipedia.org/wiki/Golden_Shield_Project http://en.wikipedia.org/wiki/Golden_Shield_Project
- avinassh 12y agoand also: http://www.youtube.com/watch?v=rH8UitdFb2o http://www.youtube.com/watch?v=rH8UitdFb2o
- pilif 12y agoNitpick: >One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases: >>If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address. ServerName is the name Apache reports in Error messages and using for redirects. It has nothing to do with the Host-Header sent by the client. If it did, a reverse lookup definitely wouldn't be of any help either. His remedy was still ok though - just for the wrong reasons.
- xorcist 12y agoIt's not a nitpick. It's the difference between making stuff up and reading the documentation. It does matter when reason about how to configure your server.
- chockenberry 12y agoI'm the author of the post linked above: thanks for the clarification about ServerName. I spent a lot of time reading that documentation and got pretty confused about how it was actually being used.
- marcosdumay 12y agoComplementing it, when there is no Host HTTP header, apache displays the default virtual server for the port used. Unless you've set it, it's the first entry on your config file.
- bastih 12y agoNew @ GFW: Now providing infrastructure stress testing for the western world.
- nl 12y agoSurely at this level of traffic it would make sense to get in contact with your provider and see if they can block it upstream?
- thspimpolds 12y agoThis is why i cloud front personal hosts, and fastly work hosts. I never open my server to the worlds with cdn' being so cheap or free now
- 19890604 12y agoA Chinese here. I would suggest, if possible, use 301 to redirect the malicious requests(which are without recognized hostnames in HTTP header) to <www.gov.cn>, which is the government's face.
- smilekzs 12y agoGood idea!
- antihero 12y ago> thousands of Apache child processes Were you still using preforking MPM or something? Surely switching to threaded or even better, event MPM would have mitigated this to a degree...
- cesarb 12y agoOr put a nginx in front of the Apache, disable keepalives in the Apache server (letting nginx deal with keepalive from the clients), and reduce MaxClients so even if they're all active the server does not go into swap. (I've done exactly that for a friend, it made a huge difference in how much traffic his box can handle).
- m11r 12y agoThis is something I was wondering, too. Using nginx, I can easily top 50,000 req/s (45Mbit/s) on smaller, static/cached content, using 4 cores and < 1GB RAM.
- WillHuxtable 12y agoMaybe China are trying a new approach to censorship. Instead of blocking things themselves, perhaps they're trying to get sysadmins to block all of China. I imagine it would save them a lot of work.
- pdoconnell 12y agoWe started seeing this around the 9th I believe. We were not the only people as well, based on a somewhat late ISC post[1]. We thought it was a new form of intelligent blackholing. Instead of sending traffic to IPs that could easily be blacklisted by tools to get around the firewall, the Great Firewall would start sending them to random "good" IPs for the same result. Others seem to think the same thing[2]. [1] https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepiratebayorg+Resolving+to+Various+Hosts/19175/ https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... [2] https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visitors-blocked-sites-redirected-porn https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visi...
- sneakest 12y agoI had one of my clients have the same thing happen to them 3 nights in a row about 10 days ago. All of the sudden they got dumped a ton of traffic out of no where. Torrent tracker updates and what seemed like legitimate traffic routed to the wrong ip. Then after about an hour most of it stopped. During the attack and after we changed some firewall settings and clamped down our request per second and connection per ip limits on the web servers. One big change was having the web servers respond 444 to any host request that was not configured (i.e. default). So if someone came to the ip looking for say google.com they would get a 444 response as it is obviously misrouted traffic. Then the next day the same thing happened but much more traffic. The 444 change helped some but there was just too much traffic for the web servers to handle quickly so they bogged down to a crawl. Luckily we were able to figure out through severfault and some other searches that it appears to be DNS poisoning coming from China. We ended up banning the entire country. The third day the same thing happened but because we had blocked the traffic from China at our outside firewall the servers were unaffected. Since then we have seen some flashes of traffic being blocked but not nearly as much as before. In all cases the increased volume of traffic only lasted about an hour. The only thing I can surmise about the length of time is DNS records only being cached for about an hour. So after that time the poisoned cached DNS records were replaced by the real resolving ip address. In the case of the article's author their servers suffered a much longer attack than we did. I am not sure if it is the Chinese government doing it or an attack for hire scheme using holes in many Chinese based DNS servers. TLDR; Remove default server settings from your web servers and have the default server block respond 444 or 404. This may help stave off the attack until your incoming traffic takes up all available resources of the web servers. Of course you could always block all traffic based in China like we did. Here is info on the status codes incase you are curious: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_C...