7 ms·
Zerigo DNS services down for 6+ hours due to massive DDoS
- _phred 14y agoApparently no ETA for restore as of 2 hours ago: https://twitter.com/zerigo/status/227322909230768128 https://twitter.com/zerigo/status/227322909230768128
- PonyGumbo 14y agoComodo's DNS.com appears to be down too.
- redguava 14y agoI understand things happen, but this is a huge outage for a DNS service. What makes it much worse is the lack of communication or updates from them. They have intermittently posted on their status site or twitter account, but not once responded to any support request I have made (attempted phone, email and twitter). I would recommend avoid them. Things happen, it's how you deal with them that matters and in my opinion that's where they have failed.
- St-Clock 14y agoThis took down services like Fogbugz on demand.
- kevingessner 14y agoWe've switched our DNS provider and we're waiting for it to propagate. Check http://fogcreekstatus.typepad.com/ http://fogcreekstatus.typepad.com/ for updates.
- _phred 14y agoGoing on 8 hours of Zerigo's downtime I've had to move all of our Zerigo DNS to DNSMadeEasy. It's a shame, because I really, really like Zerigo, especially their API. Shit happens, but 99.9% (8 hours a year of downtime) is completely unacceptable for a DNS provider.
- manveru 14y agoWell, that explains why my wife woke me up complaining about half the internet not working. Our ISP is 3 (drei.at) and she was using their DNS, guess there are issues all over Europe.
- jbarham 14y agoI run a DNS hosting service (SlickDNS, www.slickdns.com) and have seen a spike in signups today as a direct result of the Zerigo DDOS attack. I can't claim that SlickDNS is invulnerable to DDOS attack, but FWIW it does run tinydns name servers which have good performance and excellent security. So if you're impacted by the Zerigo outage, feel free to check out SlickDNS. There's a 30-day free trial with all plans and record updates are pushed through to all the name servers in under 5 seconds.
- _phred 14y agoSounds good... but do you have a REST API? That's the primary reason I chose Zerigo in the first place.
- jbarham 14y agoIt's a FAQ: https://www.slickdns.com/faq/#api https://www.slickdns.com/faq/#api ;) The REST API is in final testing, and should be released later this week. It will ship with libraries for Python, Ruby and PHP.
- aeden 14y agoAs you're probably aware the server that you use has little impact when the DDoS sends enough traffic to actually saturate your allocated bandwidth. Anycast provides a good way to handle DDoS, along with proactive monitoring and defense mechanisms, but at the end of the day DDoS are still extremely difficult to defend from completely. The downside is that Anycast is expensive and thus you need the capital to build it out and run it - which often raises the cost of systems built using it.
- Igal_Zeifman 14y agoGood point but good deflation service will actually provide full-proof DDoS protection, just as long as it got the necessary resources. Having said that, you are 100% correct - if you service provider cannot handle the extra bandwidth, brutal force DDoS will win.
- sstarr 14y agoAdd these to your hosts file to access your account: 64.27.57.25 manage.zerigo.com 64.27.57.8 dns.zerigo.com Source: https://twitter.com/coldclimate/status/227369346891132928 https://twitter.com/coldclimate/status/227369346891132928
- PonyGumbo 14y agoThank you so much!
- AdamGibbins 14y agoAnd this is why I use Route 53, I'm a lot more confident in Amazon's abilities to mitigate DDoS attacks. Which really sucks, DDoS are really hard to combat and Zerigo are an awesome company.
- aeden 14y agoI run DNSimple (https://dnsimple.com https://dnsimple.com) and we have a full REST API and support domain registrations, transfers and SSL certificates as well. Plus we have an ALIAS record type that's very useful for pointing your apex to services where they only provide a hostname. I'll be happy to answer any questions you have regarding our service either here or through our support channels.
- redguava 14y agoI just switched to DNSimple and their support has been great already.
- destraynor 14y agoLong time user of DNSimple. Great product, great support, great team, great price. Recommended.
- dedene 14y agoMe too, I can only recommend DNSimple (I'm coming from Zerigo - switched when they were acquired by 8x8), their service is awesome!
- chrismdp 14y agoJust to second this one: I've used DNSimple for years, and it's an awesome product. I can't imagine doing DNS without it now.
- lessallan 14y agoDNSimple is the best, seriously, no really, seriously.
- rfc2616 14y agoI admit that Zerigo was first to spoil me with a simple interface, but I came to DNSimple from there because of all the extra labor-saving features it has. I figure I get back at least 1-2 weeks of my life every year as a result of being a 100% DNSimple shop.
- 14y ago
- slig 14y agoWhat are the main advantages of paying for DNS hosting like Zerigo or SlickDNS instead of using the one provided for free with web host companies (E.g. Linode's DNS Manager)?
- aeden 14y agoA dedicated DNS provider will often focus on the experience around managing DNS, including APIs and advanced features. Additionally some folks don't like putting all of their eggs in one basket and prefer to have their registrar be one company, their DNS be another and their hosts be another.
- jbarham 14y agoFWIW the SlickDNS name servers are hosted by Linode so I'm a fan of their server hosting. For DNS management, the Linode interface is fine if you have a handful of domains with simple configurations, but beyond that it's unwieldy IMHO. I'd say the main reason to use a DNS hosting service is to consolidate your DNS management for all of your domains regardless of registrars or server hosting providers. E.g., I personally have domains registered with 5 registrars and use two server providers. And because they specialize in DNS, DNS hosting providers should have superior interfaces, APIs and support for DNS hosting compared to generalist hosting providers. The SlickDNS interface has two features in particular that I haven't seen in any other DNS hosting service: automatic management of "alias domains" and mapping IP addresses to named servers. See https://www.slickdns.com/features/ https://www.slickdns.com/features/ for details.
- deleted 14y ago[deleted]
- latch 14y agoSeems like if you are serious about mitigating this type of issue (as a consumer), you really should be specifying name servers from different providers. Your primary DNS server can be from dnsimple/zerigo/dnsmadeeasy and your secondary can be route53, or you could run your own. The only problem seems to be keeping them in sync. Seems like you'd have to poll the primary (using whatever API it exposes) to update the secondary. Mostly thinking out loud, surely someone more experienced could provide better guidance?
- aeden 14y agoIdeally your primary provider would support AXFR and NOTIFY which are part of the DNS zone transfer protocol. It's something we're working on adding to DNSimple, but we're not quite ready to launch it yet. The primary and secondary providers also both need to report the correct authoritative name server delegation details so the primary needs to ensure that that data is in the zone file. There is another challenge in that we're pushing the envelope a bit by offering features that rely on more than just a DNS record (for example ALIAS and POOL records). These are useful features for some people, but if you're using these types of features then they won't be portable to secondary providers.
- cbsmith 14y agoI can totally buy DDoS flooding network capacity, but I'm befuddled these days by statements saying the servers are "under load", which typically means "out of CPU". It's kind of hard for me to imagine even an i5 not being able to saturate a gigE line with DNS lookups (yes, it is a lot of packets, but it can be done) unless DNSSec is going on. Even 10gigE, if you can amortize interrupts, seems like it'd not be hard to saturate with today's hardware. What am I missing here?
- gabriel-samfia 14y agoThere are many types of DDoS. Some max out your CPU, some your network. Given that a DDoS (Distributed Denial of Service) involves potentially thousands of willing or unwilling systems, it's relatively easy to make a server unresponsive. I have a 100 Mb/s internet connection. Scale that up to 10000, and you have saturated even the fastest of internet connections. Mitigating a DDoS is not easy. Heck, its damn near impossible, considering the fact that DNS DDoS attacks are done via UDP, which allow you to spoof the source IP address. Even if you do block the IP address of al the attackers, your upstream provider is still impacted by the packets trying to come into your server. Most upstream ISPs will blackhole your server IP to diminish the impact on their network.
- silverlight 14y agoLooks like this took Trello down, too...
- kevingessner 14y agoWe've switched our DNS provider and we're waiting for the change to propagate. http://fogcreekstatus.typepad.com/2012/07/index.html http://fogcreekstatus.typepad.com/2012/07/index.html has all the details.
- gaia 14y agoBest thing Zerigo could do for their customers at this point is export all zone information and email it to them or make available for DL. I have a feeling this is going to be a long outage. In the meanwhile, here is a great list of free DNS providers (dont get caught without a secondary DNS provider): http://www.lowendtalk.com/wiki/free-dns-providers http://www.lowendtalk.com/wiki/free-dns-providers
- Uchikoma 14y agoRunning with DNSMadeEasy, is there a way to integrate it with Route 53 through AXFR to have two providers?
- aeden 14y agoThis might help: http://route53d.googlecode.com/svn-history/r2/trunk/README http://route53d.googlecode.com/svn-history/r2/trunk/README Looks like they are close to getting NOTIFY and IXFR (incremental AXFR) working. It's an interesting approach none-the-less.
- Uchikoma 14y agoCould I integrate DNSimple with DNSMadeEasy via NOTIFY/IXFR/AXFR?
- deleted 14y ago[deleted]
- sleighboy 14y agoUS-Based customer here. Our DNS just started working again.
- metalruler 14y agoI've been seeing a lot of reflector attacks in the past couple of weeks, where the attacker sends a relatively small query for a valid domain that will return a large reply. The trick is that they spoof the source IP, so the DNS reply goes to the victim. I ended up hacking something together to firewall any IPs which sent more than 1000 requests in a short period of time.
- dedene 14y agoDo you mind sharing the script / code to accomplish that? (some gist somewhere) I'm seeing a lot of these sort of things on our servers too..
- metalruler 14y agoIt really is a disgusting hack, and specific to FreeBSD. It does need to be a bit more sophisticated than "block an IP if it floods me" because as it is now someone can simply spoof the IP of an ISP's DNS server and effectively firewall them, blocking their users from being able to resolve the domain names I'm hosting. I can give you one tip to get you started: if you're running named, you can enable logging of every query, something like (hope this formats ok) : logging { channel query_logging { file "/var/log/named/querylog" versions 3 size 100M; print-time yes; // timestamp log entries }; category queries { query_logging; }; };
- gabriel-samfia 14y agoWe've seen the same kind of attack. We ended up limiting our DNS resolvers only to our own prefixes. It's a simple ACL in bind that allows recursion (domains your DNS server is not authoritative for), only to our subnets.
- piggity 14y agoDays later and what do we have from them? One solitary email and a few half-assed status page updates.