5 ms·
Incident Report – DDoS Attack
- latch 12y agoI need to learn to let things go, but: https://news.ycombinator.com/item?id=4280515 https://news.ycombinator.com/item?id=4280515 I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago from a 50Gbps attack), but once my year is up, I'm switching to Route53. The addition of the Geo DNS Queries was key for me. It isn't clear to me why I shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda like having actual scaling costs forwarded to customers.
- kyledrake 12y agoI've had a similar thought RE using Route53 for Neocities. Here's the problem with Route53 though. If you get a DDoS attack using it, it's quite plausible that you would be charged for resources used in the DDoS attack. A recent Vice article discussed this: http://motherboard.vice.com/read/inside-the-unending-cyber-siege-of-hong-kong http://motherboard.vice.com/read/inside-the-unending-cyber-s... DDoS is a nasty problem. We've received a DDoS attack that shut the entire site down for days. We can't use Cloudflare because they don't support wildcard domains without their very expensive plan. I've also heard stories from people using Cloudflare that have still not been able to resolve DDoS issues (I'm not knocking Cloudflare, they're a great company that does a really good job fighting this very hard problem, but sometimes even they have trouble with it). I'll be completely honest and say that I have no idea how to solve this problem. It's really, really, really hard. Switching to different service providers won't get you very far against the monster DDoS attacks that some people can execute.
- stevekemp 12y agoIf you're going to go the Amazon route then you absolutely need to keep an eye on billing, and set up alerts so that any DDoS which caused a spike in your costs would be caught as soon as possible.
- Intermernet 12y agoI was burnt by this in the first 48 hours of using Amazon DNS. Very unlucky I guess... I'm amazed they still bill for DDOS traffic, or even traffic from black-listed IPs. It seems many of their competitors don't.
- aeden 12y ago"and even then you can still be screwed if your bandwidth is saturated" Which is exactly what happened in this case. It sucks to be on the receiving end of this. We couldn't defend against it and let our customers down, and that hurts me deeply. We choose one approach to defense, which was internal, and that was a mistake. We're going to work on rectifying that now.
- b0k 12y agoI really don't understand why some of these low-grade DNS hosting services are so popular when Route53 is available. With Route53 you get a top-grade DNS service that is equivalent, if not better, than the enterprise hosted DNS solutions but at the price of the low-end consumer style services. I swear by Route53, it is the only service I use on AWS and I have moved a lot of my clients over to it.
- fapjacks 12y agoI agree, but there are some low-end DNS providers which have good services that give you more "domains" for less than what it costs with Route53. I use Route53 for a lot of my sites, but for tiny client sites (and personal stuff which has lots of domains), it's hard to beat $60 a year for 25 domains at DNSMadeEasy. That's less than half of what it costs to use AWS for the same number of zones. Granted, the price drops after those 25 on Route53, so if you have thousands of zones in one account, best to use Route53. Just as an example of an edge case.
- abalone 12y agoSo who do you think the "well-known third-party service that provides external DDoS protection using reverse DNS proxies" is they're going to use now? CloudFlare?
- crystaln 12y agoHopefully not. CloudFlare is remarkably unreliable for a service that claims to improve uptime.
- holic 12y agoCurious to hear more about this.
- tomw1808 12y ago+1 never had any problems so far.
- ad_hominem 12y ago[citation needed] Last I checked CloudFlare routinely handles[1] 10Gbps to 65Gbps attacks, and has successfully handled attacks as large as 300Gbps and 400Gbps. According to this report DNSSimple crumbled under 25Gbps. [1]: https://support.cloudflare.com/hc/en-us/articles/200170216-How-large-of-a-DDoS-attack-can-CloudFlare-handle- https://support.cloudflare.com/hc/en-us/articles/200170216-H...
- etcet 12y agoTheir last significant outage was only 2 months ago: https://blog.cloudflare.com/route-leak-incident-on-october-2-2014/ https://blog.cloudflare.com/route-leak-incident-on-october-2...
- xxdesmus 12y agoAs the blog post outlines, the outage was related to an upstream network provider leaking routes. Note exactly something we can prevent for them.
- stephenr 12y agoThe solution here is one for customers, not providers. Manage your DNS at one location on "master" (potentially a "private" server with IP restricted access and zone transfer ACLs). Setup 2+ accounts with "DNS providers" that support incoming zone transfers - that is, they can operate as "slave" DNS servers, pulling records automatically from your "master" (once access rules are set of course) and returning results directly to clients making DNS queries. Most "Secondary DNS" packages are < $50 year, so use a few, and don't worry about individual DNS networks being burnt to the ground.
- jhealy 12y agoIt seems like inbound and outbound zone transfers aren't offered by a number of providers (like AWS). Do you know of a list of DNS providers that support either option?
- stephenr 12y agoA search for "secondary DNS service" should give you several results. My research into it is from a "manage your DNS records internally, then use a couple of providers for all public facing responders". In that situation all you need them to support is inbound transfers, which several do.
- mike-cardwell 12y agoI used to use these two services together do this: https://puck.nether.net/dns https://acc.rollernet.us/ They're both free to sign up, provide free secondary DNS, zone transfers and fully support IPv6. I only stopped using them because I wanted to run my own DNS service.
- JonahBraun 12y agoEasyDNS provides integration with AWS: http://easyroute53.com/ http://easyroute53.com/ They have an interesting blog post about setting up secondary DNS: http://blog.easydns.org/2013/09/10/what-we-are-doing-about-chronic-ddos-attacks/ http://blog.easydns.org/2013/09/10/what-we-are-doing-about-c... I have no affiliation with them, just a happy customer.
- kator 12y ago> A new customer signed up for our service and brought in multiple domains that were already facing a DDoS attack. The customer had already tried at least 2 other providers before DNSimple. Once the domains were delegated to us, we began receiving the traffic from the DDoS. I'm curious did they know this in advance or discovered it after the fact? I often wonder about business models where the core expense is "unlimited and free". The reality is there is nothing unlimited or free for the service provider. It seems with a business model like this you open yourself to people abusing your service either by accident or by choice. Imagine poor Mr. Customer here who most likely was having horrible problems thinking to themselves "These guys can do it and for free, if I go to X service they'll cost me a lot of money". I'm a big believer in business models that incentivize both parties properly. I'm sure in general this service provider is arbitraging the 99.9% of domains that barely need any services. That said it only takes a couple of "opps" customers to drive your operational costs through the roof.
- aeden 12y agoAnthony from DNSimple here. We discovered it after the fact, via a tip from other DNS providers.
- rpug 12y agoAs someone who has been down this road many times before - I can't stress this enough: DDoS mitigation solutions don't solve the problem of an app-specific layer7 attack and it is important to do some testing of how well your mitigation service responds (and that it isn't a silver bullet.) Additionally, you need to make sure your team has tested and proven procedures for engaging the service, respond to attacks, etc. Services like NimbusDDoS (www.nimbusddos.com) are good because you can do some real scenario testing and make sure your team and infrastructure is prepared. There are other services out there too that I am less familiar with, but either way really good stuff to do.
- b0k 12y ago"unlimited" plans are subsidised by low utilization users who are getting less than what they paid for. to pull it off properly as a service provider, you really need to have a solid understanding of user usage patterns. one of the big problems that tips the low/high utilization ratio unfavorably is that unlimited plans that are primarily marketed for being unlimited tend to attract users in the high utilization bracket. so the challenge for service providers is not just understanding users and understanding that ratio but figuring out how you are going to market to, and signup, those users who will be in the low utilization bracket and will essentially be paying for something they won't be user (which is hard to do) it isn't hard to find case studies of companies that launch optimistically with one pricing plan around unlimited, to then only go back and revise their pricing and break promises because they didn't understand their users and were unable to market to and signup low utilization users. one recent example is Bitcasa
- BrindsleyQuives 12y ago"...we experienced a major volumetric DDoS attack.." Do you mean big? <facepalm>
- milos_cohagen 12y agoWhat was the overall makeup of the attack traffic? For example, 50% tcp syn, etc.
- cm2187 12y agoOut of curiosity, what are the follow ups of an attack like that? The perpetrators are probably using their own servers or compromised clients or servers. Would DNS Simple follow up on this with the abuse/complaint dept of the ISP of the attackers? Are ISP typically responsive to abuse and complaints? If they are not is there any way to black list blocks of IPs assigned to ISP who do not care about being the source of DDoS attacks? Investing in anti DDoS devices is important but even more important is for the perpetrators to face the consequences of their acts (or anyone who lets his machine being used by pirates - terminating or suspending their contract would be a fair response).
- iancarroll 12y agoI was looking at http://map.ipviking.com http://map.ipviking.com earlier and it was apparent it was a botnet, most likely innocent home users with a virus.
- coldpie 12y agoRemember to keep any machine under your control up to date! I'm looking at you, XP die-hards. If you're able to, monitor your network traffic periodically as well.
- bobbykjack 12y agoIt's easy to toss out statements like "monitor your network traffic"; do you have any good suggestions for how an average developer with relatively little understanding of networking can go about doing so?