6 ms·

I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you

I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues. We sincerely apologize for the inconvenience and thank our customers for their support and patience. Real-time updates on all incidents can be found on our status page and on Twitter at @pagerdutyops and @pagerduty. In case of outages with our regular communications channels, we will update you via email directly. In addition you can reach out to our customer support team at support@pagerduty.com or +1 (844) 700-3889. Tim Armandpour, SVP of Product Development, PagerDuty

pjlegato 10y ago
"Challenges" is exactly the sort of Dilbertesque euphemism that you should never say in a situation like this. Calling it a "challenge" implies that there is some difficult, but possible, action that the customer could take to resolve the issue. Since that is not the case, this means either you don't understand what's going on, or you're subtly mocking your customers inadvertently. Try less to make things sound nice and MBAish, and try more to just communicate honestly and directly using simple language.
kirizt 10y ago
I appreciate the update, but your service has been unavailable for hours already. This is unacceptable for a service whose core value is to ensure that we know about any incidents.
- warmwaffles 10y ago
  You apparently don't understand DNS. DynDNS is a very reputable DNS provider that twitter and many others use. For someone to take them down requires a massive amount of effort. Running a redundant DNS provider is expensive as all hell.
  - deleted 10y ago
    [deleted]
  - QUFB 10y ago
    Really? Route53 on AWS is $0.50/zone and $0.40/million queries. API integration is also very easy. Using something like Route53 as a backup is significantly cheaper than suffering from the current Dyn outage.
    - warmwaffles 10y ago
      That is not helpful if you want vanity name servers
      - chrisbolt 10y ago
        http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/white-label-name-servers.html http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/whi...
      - detaro 10y ago
        I assume your clients would prefer working nameservers over vanity ones. Especially if you are in a critical business like PagerDuty.
  - deleted 10y ago
    [deleted]
  - mancerayder 10y ago
    Running a redundant DNS provider is expensive as all hell. While 'expensive' is a relative term, I disagree that it's cost-prohibitive for most firms, as I looked into this specifically (ironically considered using Dyn as our secondary). The challenge isn't coming up with the funds, it's if you happen to use 'intelligent DNS' features; these are proprietary (by nature) and thus they don't translate 1:1 between providers. In addition to having to bridge the divide yourself, by analyzing the intelligent DNS features and using the API from each provider to simultaneously push changes to both providers, you have to write and maintain automation/tooling that ensures your records are the same (or as close as possible) between the providers. If you don't do this right, you'll get different / less predictable results between the providers, making troubleshooting something of a headache. Thus in that case the 'cost' in man effort (and risk, given that APIs change and tooling can go wrong) in addition to the monthly fee. If all you're doing is simple, standard DNS (no intelligent DNS features), it's not as hard, and it's just another monthly cost. Since you typically get charged by queries/month, if you run a popular service you're probably well able to afford the redundancy of a second provider.
- nickbauman 10y ago
  Flush your DNS like the parent said.
  - deleted 10y ago
    [deleted]
  - mayreck 10y ago
    Flushing DNS wont do shit
    - cdegroot 10y ago
      pagerduty.com moved to Route53, but the TTL on NS records can be very long. Flushing (restarting, ...) whatever can cache DNS records in your infra will help to quickly pick up the new nameservers.
    - Florin_Andrei 10y ago
      Not on your laptop. On your local DNS resolver.
- saltyhiker 10y ago
  Given that a large swath of SaaS services, infrastructure providers, and major sites across the internet are impacted, this seems harsh. Are you unhappy with PagerDuty's choice of DNS provider, or something else they have control over? I don't think anyone saw this particular problem coming.
  - pjlegato 10y ago
    A company that bills themselves as a reliable, highly available disaster handling tool ought to know better than to have a single point of failure anywhere in its infrastructure. Specifically, they shouldn't have all of their DNS hosted with one company. That is a major design flaw for a disaster-handling tool.
    - kirizt 10y ago
      This is exactly my point.
    - komali2 10y ago
      I'm not using the service, but I'm curious what an acceptable threshold for this company is. Like, if half the DNS servers are attacked? If hostile actors sever fiber optic lines in the Pacific? I ask because my secondary question, as a network noob, is was anybody prepared / preparing for a DDOS on a DNS like this? Were people talking about this before? I live in Mountain View so I've been thinking today about the steps I and my company could take in case something horrifying happens - I remember reading on reddit years ago about local internets, wifi nets, etc, and would love to start building some fail safes with this in mind. Two pronged comment, sorry.
      - corvus_sapiens 10y ago
        I'm not using the service either, but I noticed this comment [1]. It's not the first time that a DNS server has been DDoS-ed, so it has been discussed before (e.g. [2]). At minimum, I would expect a company that exists for scenarios like this to have more than one DNS server. Staying up when half of existing DNS servers are down is a new problem that no one has faced yet, but this is an old, solved one. [1] https://news.ycombinator.com/item?id=12759653 https://news.ycombinator.com/item?id=12759653 [2] https://www.tune.com/blog/importance-dns-redundancy/ https://www.tune.com/blog/importance-dns-redundancy/
AlanBoyce69 10y ago
How am i meant to see twitter status updates when twitter is down?
- jorts 10y ago
  Please check our status page as an alternative method for updates. Unfortunately, it's also been encountering the same issue so we're sending out an email with the latest updates.
  - BoycieRules 10y ago
    I didnae get an email
    - jorts 10y ago
      It's still a work in progress. If you have any immediate issues please contact us at support@pagerduty.com or (844) 700-3889.
cjbprime 10y ago
The outage started more than eight hours before you posted this message..
- pfarnsworth 10y ago
  To be fair, Hacker News isn't exactly the first place where companies post status messages. I actually applaud him for posting his message here.
patrickg_zill 10y ago
Sorry if this sounds dickish, but renting 3 servers @ $75 apiece from 3 different dedicated server companies in the USA, putting TinyDNS on them, and using them as backup servers, would have solved your problems hours ago. Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K/second performance on 700Mhz PIII CPUs: https://lists.isc.org/pipermail/bind-users/2001-June/029457.html https://lists.isc.org/pipermail/bind-users/2001-June/029457.... EDIT to add: and lengthening TTLs temporarily would mean that those 10K queries would quickly lessen the outage, since each query might last for 12 hours; and large ISPs like Comcast would cache the queries for all their customers, so a single successful query delivered to Comcast would have (some amount) of multiplier effect.
- BoycieRules 10y ago
  That does sound dickish.
  - patrickg_zill 10y ago
    Thanks, account created 9 minutes ago.
    - BoycieRules 10y ago
      Just like yer dickish edits. 3 edits. Way to go. Can you not get it correct first time retard?
      - sctb 10y ago
        You can't comment like this on Hacker News. Please read the guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- nodesocket 10y ago
  That's not how that sound be done. Just use a mix of two providers. Using your own servers and TinyDNS is silly for million/billion dollar companies. See MaxCDN for example who uses a mix of dns providers (AWS Route53 and NS1): ns-5.awsdns-00.com. ['205.251.192.5'] [TTL=172800] ns-926.awsdns-51.net. ['205.251.195.158'] [TTL=172800] ns-1762.awsdns-28.co.uk. ['205.251.198.226'] (NO GLUE) [TTL=172800] ns-1295.awsdns-33.org. ['205.251.197.15'] (NO GLUE) [TTL=172800] dns1.p03.nsone.net. ['198.51.44.3'] [TTL=172800] dns2.p03.nsone.net. ['198.51.45.3'] [TTL=172800] dns3.p03.nsone.net. ['198.51.44.67'] [TTL=172800] dns4.p03.nsone.net. ['198.51.45.67'] [TTL=172800] Curious, are you the kind of person that runs their own smtp email server and complains about GitHub pricing being too expensive?
nodesocket 10y ago
Running multiple DNS providers is not actually that difficult and certainly not cost prohibitive. I am sure after this, we will see lots of companies adding multiple DNS providers and switching to AWS Route53 (which has always been solid for me).
pfarnsworth 10y ago
I had the privilege of being on-call during this entire fiasco today and I have to say I was really really disappointed. It's surprising how broken your entire service was when DNS went down. I couldn't acknowledge anything, and my secondary on-call was getting paged because it looked like I wasn't trying to respond. I was getting phone calls for alerts that wasn't even showing up on the web client, etc. Overall, it caused chaos and I was really disappointed.
- zippergz 10y ago
  How does the service you're responsible for work when DNS stops functioning?
- JPHPJ 10y ago
  "It's surprising how broken your entire service was when DNS went down." lol
- tedmiston 10y ago
  Hopefully you have a nice SLA with them.
recycle 10y ago
PagerDuty outage is the real low point of this whole situation. Email alerts from PagerDuty that should have alerted of the outage in the first place, only got delivered hours later after the whole mess cleared out.

pjlegato 10y ago

"Challenges" is exactly the sort of Dilbertesque euphemism that you should never say in a situation like this. Calling it a "challenge" implies that there is some difficult, but possible, action that the customer could take to resolve the issue. Since that is not the case, this means either you don't understand what's going on, or you're subtly mocking your customers inadvertently. Try less to make things sound nice and MBAish, and try more to just communicate honestly and directly using simple language.

kirizt 10y ago

I appreciate the update, but your service has been unavailable for hours already. This is unacceptable for a service whose core value is to ensure that we know about any incidents.

warmwaffles 10y ago
You apparently don't understand DNS. DynDNS is a very reputable DNS provider that twitter and many others use. For someone to take them down requires a massive amount of effort. Running a redundant DNS provider is expensive as all hell.
- deleted 10y ago
  [deleted]
- QUFB 10y ago
  Really? Route53 on AWS is $0.50/zone and $0.40/million queries. API integration is also very easy. Using something like Route53 as a backup is significantly cheaper than suffering from the current Dyn outage.
  - warmwaffles 10y ago
    That is not helpful if you want vanity name servers
    - chrisbolt 10y ago
      http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/white-label-name-servers.html http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/whi...
    - detaro 10y ago
      I assume your clients would prefer working nameservers over vanity ones. Especially if you are in a critical business like PagerDuty.
- deleted 10y ago
  [deleted]
- mancerayder 10y ago
  Running a redundant DNS provider is expensive as all hell. While 'expensive' is a relative term, I disagree that it's cost-prohibitive for most firms, as I looked into this specifically (ironically considered using Dyn as our secondary). The challenge isn't coming up with the funds, it's if you happen to use 'intelligent DNS' features; these are proprietary (by nature) and thus they don't translate 1:1 between providers. In addition to having to bridge the divide yourself, by analyzing the intelligent DNS features and using the API from each provider to simultaneously push changes to both providers, you have to write and maintain automation/tooling that ensures your records are the same (or as close as possible) between the providers. If you don't do this right, you'll get different / less predictable results between the providers, making troubleshooting something of a headache. Thus in that case the 'cost' in man effort (and risk, given that APIs change and tooling can go wrong) in addition to the monthly fee. If all you're doing is simple, standard DNS (no intelligent DNS features), it's not as hard, and it's just another monthly cost. Since you typically get charged by queries/month, if you run a popular service you're probably well able to afford the redundancy of a second provider.
nickbauman 10y ago
Flush your DNS like the parent said.
- deleted 10y ago
  [deleted]
- mayreck 10y ago
  Flushing DNS wont do shit
  - cdegroot 10y ago
    pagerduty.com moved to Route53, but the TTL on NS records can be very long. Flushing (restarting, ...) whatever can cache DNS records in your infra will help to quickly pick up the new nameservers.
  - Florin_Andrei 10y ago
    Not on your laptop. On your local DNS resolver.
saltyhiker 10y ago
Given that a large swath of SaaS services, infrastructure providers, and major sites across the internet are impacted, this seems harsh. Are you unhappy with PagerDuty's choice of DNS provider, or something else they have control over? I don't think anyone saw this particular problem coming.
- pjlegato 10y ago
  A company that bills themselves as a reliable, highly available disaster handling tool ought to know better than to have a single point of failure anywhere in its infrastructure. Specifically, they shouldn't have all of their DNS hosted with one company. That is a major design flaw for a disaster-handling tool.
  - kirizt 10y ago
    This is exactly my point.
  - komali2 10y ago
    I'm not using the service, but I'm curious what an acceptable threshold for this company is. Like, if half the DNS servers are attacked? If hostile actors sever fiber optic lines in the Pacific? I ask because my secondary question, as a network noob, is was anybody prepared / preparing for a DDOS on a DNS like this? Were people talking about this before? I live in Mountain View so I've been thinking today about the steps I and my company could take in case something horrifying happens - I remember reading on reddit years ago about local internets, wifi nets, etc, and would love to start building some fail safes with this in mind. Two pronged comment, sorry.
    - corvus_sapiens 10y ago
      I'm not using the service either, but I noticed this comment [1]. It's not the first time that a DNS server has been DDoS-ed, so it has been discussed before (e.g. [2]). At minimum, I would expect a company that exists for scenarios like this to have more than one DNS server. Staying up when half of existing DNS servers are down is a new problem that no one has faced yet, but this is an old, solved one. [1] https://news.ycombinator.com/item?id=12759653 https://news.ycombinator.com/item?id=12759653 [2] https://www.tune.com/blog/importance-dns-redundancy/ https://www.tune.com/blog/importance-dns-redundancy/

AlanBoyce69 10y ago

How am i meant to see twitter status updates when twitter is down?

jorts 10y ago
Please check our status page as an alternative method for updates. Unfortunately, it's also been encountering the same issue so we're sending out an email with the latest updates.
- BoycieRules 10y ago
  I didnae get an email
  - jorts 10y ago
    It's still a work in progress. If you have any immediate issues please contact us at support@pagerduty.com or (844) 700-3889.

cjbprime 10y ago

The outage started more than eight hours before you posted this message..

pfarnsworth 10y ago
To be fair, Hacker News isn't exactly the first place where companies post status messages. I actually applaud him for posting his message here.

patrickg_zill 10y ago

Sorry if this sounds dickish, but renting 3 servers @ $75 apiece from 3 different dedicated server companies in the USA, putting TinyDNS on them, and using them as backup servers, would have solved your problems hours ago. Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K/second performance on 700Mhz PIII CPUs: https://lists.isc.org/pipermail/bind-users/2001-June/029457.html https://lists.isc.org/pipermail/bind-users/2001-June/029457.... EDIT to add: and lengthening TTLs temporarily would mean that those 10K queries would quickly lessen the outage, since each query might last for 12 hours; and large ISPs like Comcast would cache the queries for all their customers, so a single successful query delivered to Comcast would have (some amount) of multiplier effect.

BoycieRules 10y ago
That does sound dickish.
- patrickg_zill 10y ago
  Thanks, account created 9 minutes ago.
  - BoycieRules 10y ago
    Just like yer dickish edits. 3 edits. Way to go. Can you not get it correct first time retard?
    - sctb 10y ago
      You can't comment like this on Hacker News. Please read the guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
nodesocket 10y ago
That's not how that sound be done. Just use a mix of two providers. Using your own servers and TinyDNS is silly for million/billion dollar companies. See MaxCDN for example who uses a mix of dns providers (AWS Route53 and NS1): ns-5.awsdns-00.com. ['205.251.192.5'] [TTL=172800] ns-926.awsdns-51.net. ['205.251.195.158'] [TTL=172800] ns-1762.awsdns-28.co.uk. ['205.251.198.226'] (NO GLUE) [TTL=172800] ns-1295.awsdns-33.org. ['205.251.197.15'] (NO GLUE) [TTL=172800] dns1.p03.nsone.net. ['198.51.44.3'] [TTL=172800] dns2.p03.nsone.net. ['198.51.45.3'] [TTL=172800] dns3.p03.nsone.net. ['198.51.44.67'] [TTL=172800] dns4.p03.nsone.net. ['198.51.45.67'] [TTL=172800] Curious, are you the kind of person that runs their own smtp email server and complains about GitHub pricing being too expensive?

nodesocket 10y ago

Running multiple DNS providers is not actually that difficult and certainly not cost prohibitive. I am sure after this, we will see lots of companies adding multiple DNS providers and switching to AWS Route53 (which has always been solid for me).

pfarnsworth 10y ago

I had the privilege of being on-call during this entire fiasco today and I have to say I was really really disappointed. It's surprising how broken your entire service was when DNS went down. I couldn't acknowledge anything, and my secondary on-call was getting paged because it looked like I wasn't trying to respond. I was getting phone calls for alerts that wasn't even showing up on the web client, etc. Overall, it caused chaos and I was really disappointed.

zippergz 10y ago
How does the service you're responsible for work when DNS stops functioning?
JPHPJ 10y ago
"It's surprising how broken your entire service was when DNS went down." lol
tedmiston 10y ago
Hopefully you have a nice SLA with them.

recycle 10y ago

PagerDuty outage is the real low point of this whole situation. Email alerts from PagerDuty that should have alerted of the outage in the first place, only got delivered hours later after the whole mess cleared out.