4 ms·

Ask HN: DNS Outage?

67 points by dgritsko 5y ago

There seems to be a pretty widespread DNS-related issue at the moment. Does anyone have insight as to the root cause?

jkingsman 5y ago
I can't speculate on cause, but it certainly is incredibly widespread. Just about every service shows a spike in problem reports: https://downdetector.com/ https://downdetector.com/
- dgritsko 5y ago
  Yeah, I was surprised that there wasn't an HN thread on it yet.
cett 5y ago
Possibly Akamai?
chrishynes 5y ago
Seeing a "Service Unavailable - DNS failure The server is temporarily unable to service your request. Please try again later." on many sites.
bpeebles 5y ago
So far it seems to be an Akamai internal DNS look up problem. That is, it seems if they need to resolve the hostname of the origin to serve a page they front for, it fails but I haven't seen any actual DNS failures on my side or other places, just inside Akamai-served pages saying they failed to resolve with an Akamai reference code.
- Aaargh20318 5y ago
  It's not just their internal DNS, I was about to order some gear online but the hostname for my bank no longer resolves.
  - bpeebles 5y ago
    Ah, I hadn't come across them. My feeling is still a minority of sites use Akamai's DNS for their public-facing names, so probably those sites would be affected that way. Although either way the site is down once the cache expires.
pgl 5y ago
https://edgedns.status.akamai.com/ https://edgedns.status.akamai.com/ We are aware of an emerging issue with the Edge DNS service. We are actively investigating the issue. If you have questions or are experiencing impact due to this issue, please contact Akamai Technical Support. In the interest of time, we are providing you the most current information available, which is subject to changes, corrections, and updates. Jul 22, 16:09 UTC
- rkhacker 5y ago
  Is fidelity outage related to this as well?
neogodless 5y ago
https://downdetector.com/ https://downdetector.com/ Confirmed failing for me: ups.com steam.com steamcommunity.com homedepot.com newegg.com Still working amazon.com americanexpress.com vanguard.com
deleted 5y ago
[deleted]
elric 5y ago
DNS is decentralized, but it seems like Akamai breaking breaks a huge swathe of websites.
- codegeek 5y ago
  Yes but services like Akamai or Cloudflare etc. centralize quite a few things. They go down, DNS being decentralized doesn't matter much.
christophilus 5y ago
Related thread (that I just opened)[0]. It's affecting the major brokerages. Will be interesting to see the postmortem on this. [0] https://news.ycombinator.com/item?id=27920592 https://news.ycombinator.com/item?id=27920592
dialtone 5y ago
anyone using akamai as a CDN is having issues, if they use it also as a DNS they won't even resolve their primary domains.
sethetter 5y ago
Being covered by independent.co.uk: https://www.independent.co.uk/life-style/gadgets-and-tech/internet-down-update-outage-amazon-b1888831.html https://www.independent.co.uk/life-style/gadgets-and-tech/in... Edit: formatting
monkaiju 5y ago
For anyone trying to quantify the impact of this on their setup, check out dug! Its an opensource global dns propagation check on your cli! https://dug.unfrl.com/ https://dug.unfrl.com/
- jkingsman 5y ago
  This appears to be an internal DNS lookup failure; the domain itself resolves but Akamai's internal network falls over. If this is the case, a DNS propagation check will not indicate any issues.
  - monkaiju 5y ago
    Ah, good point!
unemphysbro 5y ago
yup, our workplace covid check-in system is down. :)
schemathings 5y ago
Probably related - carriers are reporting AS701 BGP issues with Verizon at Equinix NY2 datacenter.
mkranjec 5y ago
According to pingdom about 30k sites affected at the peak.

Ask HN: DNS Outage?

67 points by dgritsko 5y ago

There seems to be a pretty widespread DNS-related issue at the moment. Does anyone have insight as to the root cause?

I can't speculate on cause, but it certainly is incredibly widespread. Just about every service shows a spike in problem reports: https://downdetector.com/ https://downdetector.com/

dgritsko 5y ago
Yeah, I was surprised that there wasn't an HN thread on it yet.

cett 5y ago

Possibly Akamai?

chrishynes 5y ago

Seeing a "Service Unavailable - DNS failure The server is temporarily unable to service your request. Please try again later." on many sites.

bpeebles 5y ago

So far it seems to be an Akamai internal DNS look up problem. That is, it seems if they need to resolve the hostname of the origin to serve a page they front for, it fails but I haven't seen any actual DNS failures on my side or other places, just inside Akamai-served pages saying they failed to resolve with an Akamai reference code.

Aaargh20318 5y ago
It's not just their internal DNS, I was about to order some gear online but the hostname for my bank no longer resolves.
- bpeebles 5y ago
  Ah, I hadn't come across them. My feeling is still a minority of sites use Akamai's DNS for their public-facing names, so probably those sites would be affected that way. Although either way the site is down once the cache expires.

pgl 5y ago

https://edgedns.status.akamai.com/ https://edgedns.status.akamai.com/ We are aware of an emerging issue with the Edge DNS service. We are actively investigating the issue. If you have questions or are experiencing impact due to this issue, please contact Akamai Technical Support. In the interest of time, we are providing you the most current information available, which is subject to changes, corrections, and updates. Jul 22, 16:09 UTC

rkhacker 5y ago
Is fidelity outage related to this as well?

neogodless 5y ago

https://downdetector.com/ https://downdetector.com/ Confirmed failing for me: ups.com steam.com steamcommunity.com homedepot.com newegg.com Still working amazon.com americanexpress.com vanguard.com

deleted 5y ago

[deleted]

elric 5y ago

DNS is decentralized, but it seems like Akamai breaking breaks a huge swathe of websites.

codegeek 5y ago
Yes but services like Akamai or Cloudflare etc. centralize quite a few things. They go down, DNS being decentralized doesn't matter much.

christophilus 5y ago

Related thread (that I just opened)[0]. It's affecting the major brokerages. Will be interesting to see the postmortem on this. [0] https://news.ycombinator.com/item?id=27920592 https://news.ycombinator.com/item?id=27920592

dialtone 5y ago

anyone using akamai as a CDN is having issues, if they use it also as a DNS they won't even resolve their primary domains.

sethetter 5y ago

Being covered by independent.co.uk: https://www.independent.co.uk/life-style/gadgets-and-tech/internet-down-update-outage-amazon-b1888831.html https://www.independent.co.uk/life-style/gadgets-and-tech/in... Edit: formatting

monkaiju 5y ago

For anyone trying to quantify the impact of this on their setup, check out dug! Its an opensource global dns propagation check on your cli! https://dug.unfrl.com/ https://dug.unfrl.com/

jkingsman 5y ago
This appears to be an internal DNS lookup failure; the domain itself resolves but Akamai's internal network falls over. If this is the case, a DNS propagation check will not indicate any issues.
- monkaiju 5y ago
  Ah, good point!

unemphysbro 5y ago

yup, our workplace covid check-in system is down. :)

schemathings 5y ago

Probably related - carriers are reporting AS701 BGP issues with Verizon at Equinix NY2 datacenter.

mkranjec 5y ago

According to pingdom about 30k sites affected at the peak.