5 ms·

DNSSEC Outage on www.cloudflare.com – 2019-03-21

73 points by wglb 8y ago

bluejekyll 8y ago
"why should a DS record for www.cloudflare.com matter?!" - that was my exact question when I looked at the output from the DNSSEC Debugger. Now I'm curious as to why www has NS records... (DS records are used to tie DNSKEYs to a child zone from the parent zone).
- prdonahue 8y ago
  It has NS records because that's how the owner of cloudflare.com (within the Cloudflare Dashboard) delegates control of www.cloudflare.com to another team.
  - bluejekyll 8y ago
    Ah. That makes a lot of sense, thanks.
deleted 8y ago
[deleted]
localhostdotdev 8y ago
time doesn't match exactly but that might be it: https://www.cloudflarestatus.com/incidents/3k7gtg4fml1p https://www.cloudflarestatus.com/incidents/3k7gtg4fml1p (https://www.cloudflarestatus.com/history https://www.cloudflarestatus.com/history for the list)
ohyeshedid 8y ago
Somewhere tptacek is in tears, and laughing.
- masklinn 8y ago
  https://twitter.com/tqbf/status/1116112320567087104 https://twitter.com/tqbf/status/1116112320567087104
- dcbadacd 8y ago
  Just like you can f* up HTTPS when you have Strict Transport Security, you can f* up DNSSEC. While I don't really like current DNSSEC implementations, this is not DNSSEC's fault really. The idea I do like, that recursive resolvers can't tamper my resolving - that the recursive resolver is left out of the trust chain.
- kadendogthing 8y ago
  I can't wait to see what disingenuous statements he makes about a config failure.
- belorn 8y ago
  DNS delegation errors are kind of rare. There are so many configuration errors in web deployment, and the most common errors I see which causes websites to be inaccessible are misconfigured certificates where: • You get the default certificate for the web server rather than the correct one for the vhost. • You have a redirect from www->bare or bare->www and the redirect vhost don't have a certificate. • The certificate don't include the www subdomain name. • A wild card certificate doesn't include the bare name. • There are multiple web servers listed in DNS but only a subset has a correct HTTPS configuration.
gscott 8y ago
The Cloudflare Android 1.1.1.1 VPN app was down most of the day.
dsl 8y ago
We really need to push back against centralization on a small number of providers. As we have seen with previous Cloudflare issues and the big Dyn incident, a single issue can basically make the internet unusable.
- therealmarv 8y ago
  I'm also really worried about that. (It's just a feeling but..) for a service as big as Cloudflare I think it may be not as failure proof and clustered designed as it should be for a massive DNS provider. I for myself pay for a smaller DNS provider like CloudDNS with a bigger redundancy in NS servers (4 instead of 2 at Cloudflare) than going for free at Cloudflare.
  - cbg0 8y ago
    > bigger redundancy in NS servers (4 instead of 2 at Cloudflare) I don't think this is really all that important, as typically all these servers are running anycast. If you really wanted redundancy, you would have NS records from at least two different providers.
    - therealmarv 8y ago
      That's a good point. A backup DNS on another service is on my bucket list once sites get a little bigger and more relevant (right now it's not worth it).
  - peterwwillis 8y ago
    You're paying for a service that's less robust because they have more IP addresses serving their database?
    - therealmarv 8y ago
      For the time beeing I'm not sure which is more or less robust because I have not seen any DNS outages in both.
  - 8y ago
prdonahue 8y ago
This blip was caused by configuration changes we made to our marketing site (www), which is distinct from any control plane (api/dash) or edge services provided to our customers. Specifically, we were breaking out our "www" subdomain as a separately managed zone, and some of the configuration wasn't properly migrated. This resolution failure was regrettable, but there was no impact to the service we provide for our customers.
iofiiiiiiiii 8y ago
The title gives the impression that this was an outage of Cloudflare's DNSSEC service, whereas in reality the Cloudflare website was down due to a DNSSEC error. Ideally the title should be clarified to make it obvious that Cloudflare customers were not affected.

73 points by wglb 8y ago

"why should a DS record for www.cloudflare.com matter?!" - that was my exact question when I looked at the output from the DNSSEC Debugger. Now I'm curious as to why www has NS records... (DS records are used to tie DNSKEYs to a child zone from the parent zone).

prdonahue 8y ago
It has NS records because that's how the owner of cloudflare.com (within the Cloudflare Dashboard) delegates control of www.cloudflare.com to another team.
- bluejekyll 8y ago
  Ah. That makes a lot of sense, thanks.

deleted 8y ago

[deleted]

localhostdotdev 8y ago

time doesn't match exactly but that might be it: https://www.cloudflarestatus.com/incidents/3k7gtg4fml1p https://www.cloudflarestatus.com/incidents/3k7gtg4fml1p (https://www.cloudflarestatus.com/history https://www.cloudflarestatus.com/history for the list)

ohyeshedid 8y ago

Somewhere tptacek is in tears, and laughing.

masklinn 8y ago
https://twitter.com/tqbf/status/1116112320567087104 https://twitter.com/tqbf/status/1116112320567087104
dcbadacd 8y ago
Just like you can f* up HTTPS when you have Strict Transport Security, you can f* up DNSSEC. While I don't really like current DNSSEC implementations, this is not DNSSEC's fault really. The idea I do like, that recursive resolvers can't tamper my resolving - that the recursive resolver is left out of the trust chain.
kadendogthing 8y ago
I can't wait to see what disingenuous statements he makes about a config failure.
belorn 8y ago
DNS delegation errors are kind of rare. There are so many configuration errors in web deployment, and the most common errors I see which causes websites to be inaccessible are misconfigured certificates where: • You get the default certificate for the web server rather than the correct one for the vhost. • You have a redirect from www->bare or bare->www and the redirect vhost don't have a certificate. • The certificate don't include the www subdomain name. • A wild card certificate doesn't include the bare name. • There are multiple web servers listed in DNS but only a subset has a correct HTTPS configuration.

gscott 8y ago

The Cloudflare Android 1.1.1.1 VPN app was down most of the day.

dsl 8y ago

We really need to push back against centralization on a small number of providers. As we have seen with previous Cloudflare issues and the big Dyn incident, a single issue can basically make the internet unusable.

therealmarv 8y ago
I'm also really worried about that. (It's just a feeling but..) for a service as big as Cloudflare I think it may be not as failure proof and clustered designed as it should be for a massive DNS provider. I for myself pay for a smaller DNS provider like CloudDNS with a bigger redundancy in NS servers (4 instead of 2 at Cloudflare) than going for free at Cloudflare.
- cbg0 8y ago
  > bigger redundancy in NS servers (4 instead of 2 at Cloudflare) I don't think this is really all that important, as typically all these servers are running anycast. If you really wanted redundancy, you would have NS records from at least two different providers.
  - therealmarv 8y ago
    That's a good point. A backup DNS on another service is on my bucket list once sites get a little bigger and more relevant (right now it's not worth it).
- peterwwillis 8y ago
  You're paying for a service that's less robust because they have more IP addresses serving their database?
  - therealmarv 8y ago
    For the time beeing I'm not sure which is more or less robust because I have not seen any DNS outages in both.
- 8y ago

prdonahue 8y ago

This blip was caused by configuration changes we made to our marketing site (www), which is distinct from any control plane (api/dash) or edge services provided to our customers. Specifically, we were breaking out our "www" subdomain as a separately managed zone, and some of the configuration wasn't properly migrated. This resolution failure was regrettable, but there was no impact to the service we provide for our customers.

iofiiiiiiiii 8y ago

The title gives the impression that this was an outage of Cloudflare's DNSSEC service, whereas in reality the Cloudflare website was down due to a DNSSEC error. Ideally the title should be clarified to make it obvious that Cloudflare customers were not affected.