4 ms·

Given that a large swath of SaaS services, infrastructure providers, and major sites across the internet are impacted, this seems harsh. Are you unhappy with P

Given that a large swath of SaaS services, infrastructure providers, and major sites across the internet are impacted, this seems harsh. Are you unhappy with PagerDuty's choice of DNS provider, or something else they have control over? I don't think anyone saw this particular problem coming.

pjlegato 10y ago
A company that bills themselves as a reliable, highly available disaster handling tool ought to know better than to have a single point of failure anywhere in its infrastructure. Specifically, they shouldn't have all of their DNS hosted with one company. That is a major design flaw for a disaster-handling tool.
- kirizt 10y ago
  This is exactly my point.
- komali2 10y ago
  I'm not using the service, but I'm curious what an acceptable threshold for this company is. Like, if half the DNS servers are attacked? If hostile actors sever fiber optic lines in the Pacific? I ask because my secondary question, as a network noob, is was anybody prepared / preparing for a DDOS on a DNS like this? Were people talking about this before? I live in Mountain View so I've been thinking today about the steps I and my company could take in case something horrifying happens - I remember reading on reddit years ago about local internets, wifi nets, etc, and would love to start building some fail safes with this in mind. Two pronged comment, sorry.
  - corvus_sapiens 10y ago
    I'm not using the service either, but I noticed this comment [1]. It's not the first time that a DNS server has been DDoS-ed, so it has been discussed before (e.g. [2]). At minimum, I would expect a company that exists for scenarios like this to have more than one DNS server. Staying up when half of existing DNS servers are down is a new problem that no one has faced yet, but this is an old, solved one. [1] https://news.ycombinator.com/item?id=12759653 https://news.ycombinator.com/item?id=12759653 [2] https://www.tune.com/blog/importance-dns-redundancy/ https://www.tune.com/blog/importance-dns-redundancy/
  - ortusdux 10y ago
    Re question #2, Amazon uses UltraDNS as a backup and seemed to be relatively unaffected by today's attack. Re question #1, check out PagerDuty's reliability page here: https://www.pagerduty.com/features/always-on-reliability/ https://www.pagerduty.com/features/always-on-reliability/ Namely "Uninterrupted Service at Scale - Our service is distributed across multiple data centers and hosting providers, so that if one goes down, we stay available." It seems fair to expect them to have a backup dns too, but I am not an expert.
  - 464192002d7fe1c 10y ago
    > is was anybody prepared / preparing for a DDOS on a DNS like this Yes. I have, personally, been under attack with as-large or larger than todays attacks at my DNS infrastructure and survived.
derwiki 10y ago
From the perspective of my service being down, my customers being pissed, and me not being notified.. yes, maybe PD should be held to a higher standard of uptime. Seems core to their value prop.
cm2187 10y ago
> I don't think anyone saw this particular problem coming. Knocking half of the web off the grid because their DNS provider is under attack? It happened recently to DNSimple. https://blog.dnsimple.com/2014/12/incident-report-ddos/ https://blog.dnsimple.com/2014/12/incident-report-ddos/ The irony is that I noticed it when dotnetrocks.com went offline, at that time dotnetrocks was sponsored by dnsimple...

pjlegato 10y ago

A company that bills themselves as a reliable, highly available disaster handling tool ought to know better than to have a single point of failure anywhere in its infrastructure. Specifically, they shouldn't have all of their DNS hosted with one company. That is a major design flaw for a disaster-handling tool.

kirizt 10y ago
This is exactly my point.
komali2 10y ago
I'm not using the service, but I'm curious what an acceptable threshold for this company is. Like, if half the DNS servers are attacked? If hostile actors sever fiber optic lines in the Pacific? I ask because my secondary question, as a network noob, is was anybody prepared / preparing for a DDOS on a DNS like this? Were people talking about this before? I live in Mountain View so I've been thinking today about the steps I and my company could take in case something horrifying happens - I remember reading on reddit years ago about local internets, wifi nets, etc, and would love to start building some fail safes with this in mind. Two pronged comment, sorry.
- corvus_sapiens 10y ago
  I'm not using the service either, but I noticed this comment [1]. It's not the first time that a DNS server has been DDoS-ed, so it has been discussed before (e.g. [2]). At minimum, I would expect a company that exists for scenarios like this to have more than one DNS server. Staying up when half of existing DNS servers are down is a new problem that no one has faced yet, but this is an old, solved one. [1] https://news.ycombinator.com/item?id=12759653 https://news.ycombinator.com/item?id=12759653 [2] https://www.tune.com/blog/importance-dns-redundancy/ https://www.tune.com/blog/importance-dns-redundancy/
- ortusdux 10y ago
  Re question #2, Amazon uses UltraDNS as a backup and seemed to be relatively unaffected by today's attack. Re question #1, check out PagerDuty's reliability page here: https://www.pagerduty.com/features/always-on-reliability/ https://www.pagerduty.com/features/always-on-reliability/ Namely "Uninterrupted Service at Scale - Our service is distributed across multiple data centers and hosting providers, so that if one goes down, we stay available." It seems fair to expect them to have a backup dns too, but I am not an expert.
- 464192002d7fe1c 10y ago
  > is was anybody prepared / preparing for a DDOS on a DNS like this Yes. I have, personally, been under attack with as-large or larger than todays attacks at my DNS infrastructure and survived.

derwiki 10y ago

From the perspective of my service being down, my customers being pissed, and me not being notified.. yes, maybe PD should be held to a higher standard of uptime. Seems core to their value prop.

cm2187 10y ago

> I don't think anyone saw this particular problem coming. Knocking half of the web off the grid because their DNS provider is under attack? It happened recently to DNSimple. https://blog.dnsimple.com/2014/12/incident-report-ddos/ https://blog.dnsimple.com/2014/12/incident-report-ddos/ The irony is that I noticed it when dotnetrocks.com went offline, at that time dotnetrocks was sponsored by dnsimple...