6 ms·

Linode-hosted DNS zones were down

43 points by mwpmaybe 10y ago

mwpmaybe 10y ago
My monitors started going berserk around 3:36 PM CT. Here's an example of the problem (I am not hostmaster for The Onion, but they are a well-known Linode customer): $ whois theonion.com | grep -i 'name server' Name Server: NS1.LINODE.COM Name Server: NS2.LINODE.COM Name Server: NS3.LINODE.COM Name Server: NS4.LINODE.COM Name Server: NS5.LINODE.COM $ dig @ns1.linode.com theonion.com ns +short $ dig @ns1.linode.com theonion.com a +short
- cft 10y ago
  I prefer DJBDNS tools: #dnsq a theonion.com NS1.LINODE.COM 1 theonion.com: temporary failure
brajkovic 10y ago
At one point, this said it was due to a Denial of Service attack. That text has since been pulled, unfortunately I didn't get a screenshot.
- slester 10y ago
  The email I received: DNS Performance Issue Incident Report for Linode Investigating We are currently experiencing denial of service attacks that are targeting our DNS infrastructure. Aug 23, 20:48 UTC This incident affects: Hosted DNS Service.
methodover 10y ago
This is one reason why I'm moving our startup over to AWS. :/ They seem to experience less of these catastrophic DDoS attacks. Edit: Err, not a DDoS attack apparently. But catastrophic nonetheless.
- josephb 10y ago
  Route53 also: A) Serves DNS from multiple domain names. A number of variants in .com .org and others etc B) Serves DNS from multiple internet networks / sources. Queries from an end user don't all hit the same Anycast subnet.
  - kyledrake 10y ago
    Route 53 also has no IPv6 support, and no stated policy on how it deals with DDoS attacks, or if you get charged for the attack if it's directed at you. Good luck!
    - jtrtoo 10y ago
      It might be more accurate to say R53 has partial v6 support. It'll return v6 records (AAAA) happily. The nameservers themselves do not respond on v6 (which is unfortunate and certainly means v6 support is lacking). Practically, however, it's pretty unlikely, that a v6 only host isn't going to also have access to a v6-to-v4 gateway. Not ideal, but generally transparent for most intents and purposes ...and generally used by just about anyone with a v6 address assignment today, everyday. For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a lot of countermeasures, some of which can be found via linked PDFs at https://aws.amazon.com/security/ https://aws.amazon.com/security/ Various R53 items are included in their DDoS document there. Officially you get charged. Unofficially, it depends on the circumstances. There's also a world of difference between a DDoS attack against a fully managed offering (like R53) versus, say, an EC2 instance. Have you seen a better approach taken by other providers? I don't mean technical approach (which is near impossible to compare other than by track record), but from a policy perspective?
mkorsak 10y ago
I can confirm this was not a DDoS, and we are actively working to fix the issue right now. We'll be updating our status page as soon as we have more updates to deliver.
- Mahn 10y ago
  Do you have any ETA? Sorry, I don't mean to be an ass, it's just our entire business is currently down because of this.
  - mkorsak 10y ago
    I understand, and I don't have an ETA quite yet, but it appears that service is already starting to return. I've tested my domains hosted on Linode and a number of others that I know are and they appear to be working now. We're still working asap to ensure everything is coming back appropriately.
    - jpmattia 10y ago
      Thank you for even that partial info. We can confirm that we're seeing some response now.
    - Mahn 10y ago
      Everything seems to be back now in our case. Thanks!
- empressplay 10y ago
  Isn't half the point of having 5 bloody DNS servers is that they shouldn't _all_ go down at the same time? Get it together Linode!
  - josephb 10y ago
    The internet facing part of Linode's DNS service is hosted by Cloudflare.
mindslight 10y ago
I recently migrated away from using Linode as secondary when they pushed everyone over to Cloudflare. That Internet-destroying company already has enough power!
- viraptor 10y ago
  Why do you think Cloudflare is internet-destroying? Serious question.
  - curried_haskell 10y ago
    They are growing into a very dangerous position of acting as a gatekeeper to the Internet. Try accessing any major website through Tor.
    - angry-hacker 10y ago
      And try having a webserver being constantly attacked by TOR. They are not doing it for fun. If you think they are bad, give them suggestions or new tech how to solve this problem. I thank cloudflare for making my life easier.
    - duskwuff 10y ago
      Not only a gatekeeper, but also a keymaster. (As it were.) Cloudflare's role as a reverse proxy, and as a SSL CA, and as a DNS provider leaves users and webmasters both highly vulnerable to surveillance and/or content tampering by CF. Not saying that they're doing anything malicious, of course. Merely that it'd be very easy for them to do so.
  - DanielDent 10y ago
    So far Cloudflare appears to be acting in a non-destroying way. They take as content-neutral a stance as they can, and have had to defend lawsuits as a result of that stance. They recognize that the internet only functions and they can only be part of the infrastructure of the internet if they take that stance. The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford. In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.
dino2k 10y ago
This smells like a major fuckup
FreeKill 10y ago
Seems like it's coming back up! My resources are reachable again.
therealmarv 10y ago
Is not it possible to outsource that critical part? What are good and proofed dns services out there?
- josephb 10y ago
  They do outsource to Cloudflare. One assumes some behind the scenes infrastructure exists to push and sync updates to Cloudflare.
  - ehPReth 10y ago
    They probably use CloudFlare's DNS proxy service: https://www.cloudflare.com/virtual-dns/ https://www.cloudflare.com/virtual-dns/
- viraptor 10y ago
  Unless you want to rely on one company with anycast (like amazon or cloudflare), you can always just keep the same zone in a number of unrelated companies.
gr2020 10y ago
This was just the kick in the pants I needed to set up secondary DNS at Route 53!

43 points by mwpmaybe 10y ago

My monitors started going berserk around 3:36 PM CT. Here's an example of the problem (I am not hostmaster for The Onion, but they are a well-known Linode customer): $ whois theonion.com | grep -i 'name server' Name Server: NS1.LINODE.COM Name Server: NS2.LINODE.COM Name Server: NS3.LINODE.COM Name Server: NS4.LINODE.COM Name Server: NS5.LINODE.COM $ dig @ns1.linode.com theonion.com ns +short $ dig @ns1.linode.com theonion.com a +short

cft 10y ago
I prefer DJBDNS tools: #dnsq a theonion.com NS1.LINODE.COM 1 theonion.com: temporary failure

brajkovic 10y ago

At one point, this said it was due to a Denial of Service attack. That text has since been pulled, unfortunately I didn't get a screenshot.

slester 10y ago
The email I received: DNS Performance Issue Incident Report for Linode Investigating We are currently experiencing denial of service attacks that are targeting our DNS infrastructure. Aug 23, 20:48 UTC This incident affects: Hosted DNS Service.

methodover 10y ago

This is one reason why I'm moving our startup over to AWS. :/ They seem to experience less of these catastrophic DDoS attacks. Edit: Err, not a DDoS attack apparently. But catastrophic nonetheless.

josephb 10y ago
Route53 also: A) Serves DNS from multiple domain names. A number of variants in .com .org and others etc B) Serves DNS from multiple internet networks / sources. Queries from an end user don't all hit the same Anycast subnet.
- kyledrake 10y ago
  Route 53 also has no IPv6 support, and no stated policy on how it deals with DDoS attacks, or if you get charged for the attack if it's directed at you. Good luck!
  - jtrtoo 10y ago
    It might be more accurate to say R53 has partial v6 support. It'll return v6 records (AAAA) happily. The nameservers themselves do not respond on v6 (which is unfortunate and certainly means v6 support is lacking). Practically, however, it's pretty unlikely, that a v6 only host isn't going to also have access to a v6-to-v4 gateway. Not ideal, but generally transparent for most intents and purposes ...and generally used by just about anyone with a v6 address assignment today, everyday. For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a lot of countermeasures, some of which can be found via linked PDFs at https://aws.amazon.com/security/ https://aws.amazon.com/security/ Various R53 items are included in their DDoS document there. Officially you get charged. Unofficially, it depends on the circumstances. There's also a world of difference between a DDoS attack against a fully managed offering (like R53) versus, say, an EC2 instance. Have you seen a better approach taken by other providers? I don't mean technical approach (which is near impossible to compare other than by track record), but from a policy perspective?

mkorsak 10y ago

I can confirm this was not a DDoS, and we are actively working to fix the issue right now. We'll be updating our status page as soon as we have more updates to deliver.

Mahn 10y ago
Do you have any ETA? Sorry, I don't mean to be an ass, it's just our entire business is currently down because of this.
- mkorsak 10y ago
  I understand, and I don't have an ETA quite yet, but it appears that service is already starting to return. I've tested my domains hosted on Linode and a number of others that I know are and they appear to be working now. We're still working asap to ensure everything is coming back appropriately.
  - jpmattia 10y ago
    Thank you for even that partial info. We can confirm that we're seeing some response now.
  - Mahn 10y ago
    Everything seems to be back now in our case. Thanks!
empressplay 10y ago
Isn't half the point of having 5 bloody DNS servers is that they shouldn't _all_ go down at the same time? Get it together Linode!
- josephb 10y ago
  The internet facing part of Linode's DNS service is hosted by Cloudflare.

mindslight 10y ago

I recently migrated away from using Linode as secondary when they pushed everyone over to Cloudflare. That Internet-destroying company already has enough power!

viraptor 10y ago
Why do you think Cloudflare is internet-destroying? Serious question.
- curried_haskell 10y ago
  They are growing into a very dangerous position of acting as a gatekeeper to the Internet. Try accessing any major website through Tor.
  - angry-hacker 10y ago
    And try having a webserver being constantly attacked by TOR. They are not doing it for fun. If you think they are bad, give them suggestions or new tech how to solve this problem. I thank cloudflare for making my life easier.
  - duskwuff 10y ago
    Not only a gatekeeper, but also a keymaster. (As it were.) Cloudflare's role as a reverse proxy, and as a SSL CA, and as a DNS provider leaves users and webmasters both highly vulnerable to surveillance and/or content tampering by CF. Not saying that they're doing anything malicious, of course. Merely that it'd be very easy for them to do so.
- DanielDent 10y ago
  So far Cloudflare appears to be acting in a non-destroying way. They take as content-neutral a stance as they can, and have had to defend lawsuits as a result of that stance. They recognize that the internet only functions and they can only be part of the infrastructure of the internet if they take that stance. The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford. In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.

dino2k 10y ago

This smells like a major fuckup

FreeKill 10y ago

Seems like it's coming back up! My resources are reachable again.

therealmarv 10y ago

Is not it possible to outsource that critical part? What are good and proofed dns services out there?

josephb 10y ago
They do outsource to Cloudflare. One assumes some behind the scenes infrastructure exists to push and sync updates to Cloudflare.
- ehPReth 10y ago
  They probably use CloudFlare's DNS proxy service: https://www.cloudflare.com/virtual-dns/ https://www.cloudflare.com/virtual-dns/
viraptor 10y ago
Unless you want to rely on one company with anycast (like amazon or cloudflare), you can always just keep the same zone in a number of unrelated companies.

gr2020 10y ago

This was just the kick in the pants I needed to set up secondary DNS at Route 53!