3 ms·

https://dnsviz.net/d/slack.com/YVXX_g/dnssec/ https://dnsviz.net/d/slack.com/YVXX_g/dnssec/ the dnsviz analysis showing the slack.com zone DNSKEY existing at 12

by terom 5y ago

https://dnsviz.net/d/slack.com/YVXX_g/dnssec/ https://dnsviz.net/d/slack.com/YVXX_g/dnssec/ the dnsviz analysis showing the slack.com zone DNSKEY existing at 12:55, followed by the the .com zone DS record at 15:30. However, the next analysis at 17:24 shows both the .com zone DS and slack.com DNSKEY records have disappeared! Given that the slack.com DNSKEY shows up with a 1h TTL and the .com zone DS has a 24h TTL, they are screwed in the presence of cached slack.com DS records from the .com zone. Do not throw away your DNSKEY until your delegation's TTL has absolutely positively surely expired from any resolver caches! The slack.com domain is an AWS Route 53 zone, I'd be really interested to see a post-mortem explaining what happened here. Are they unable to recover the KSK/ZSK and restore the DNSKEY/etc records?

dnsmichi 5y ago
Great analysis, thanks! Slack support says that users should tell their ISPs to invalidate the DNS cache for slack.com https://status.slack.com/2021-09/06c1e17de93e7dc2 https://status.slack.com/2021-09/06c1e17de93e7dc2 (access with 8.8.8.8 as resolver - fallback https://slack-status.azureedge.net/ https://slack-status.azureedge.net/) Since the faulty DS record was in .com, everyone has a max wait-for-ttl-to-expire time of 24h. Google/Cloudflare etc. seem to also invalidate .com caching very quickly, 8.8.8.8 quickly was the first workaround. Meanwhile, 14 hours later, DTAG in Germany still does not resolve. The default resolvers have dnssec enabled. dig slack.com +cd tells the resolver to skip dnssec validation tests, and then it works again. Screenshots with the command output in https://twitter.com/dnsmichi/status/1443840645513293853?s=2 https://twitter.com/dnsmichi/status/1443840645513293853?s=2 Very interested in the post-mortem analysis. I think there were similar mistakes as with nasa.gov incident and the comcast analysis in 2012: https://www.internetsociety.org/blog/2012/01/comcast-releases-detailed-analysis-of-nasa-gov-dnssec-validation-failure/ https://www.internetsociety.org/blog/2012/01/comcast-release... Learnings for me: - dnstracer (https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L57 https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L5...) helps with detecting missing glue records, but not dnssec - dnstrace (https://github.com/rs/dnstrace https://github.com/rs/dnstrace) is a better alternative with dnssec

by terom 5y ago

dnsmichi 5y ago

Great analysis, thanks! Slack support says that users should tell their ISPs to invalidate the DNS cache for slack.com https://status.slack.com/2021-09/06c1e17de93e7dc2 https://status.slack.com/2021-09/06c1e17de93e7dc2 (access with 8.8.8.8 as resolver - fallback https://slack-status.azureedge.net/ https://slack-status.azureedge.net/) Since the faulty DS record was in .com, everyone has a max wait-for-ttl-to-expire time of 24h. Google/Cloudflare etc. seem to also invalidate .com caching very quickly, 8.8.8.8 quickly was the first workaround. Meanwhile, 14 hours later, DTAG in Germany still does not resolve. The default resolvers have dnssec enabled. dig slack.com +cd tells the resolver to skip dnssec validation tests, and then it works again. Screenshots with the command output in https://twitter.com/dnsmichi/status/1443840645513293853?s=2 https://twitter.com/dnsmichi/status/1443840645513293853?s=2 Very interested in the post-mortem analysis. I think there were similar mistakes as with nasa.gov incident and the comcast analysis in 2012: https://www.internetsociety.org/blog/2012/01/comcast-releases-detailed-analysis-of-nasa-gov-dnssec-validation-failure/ https://www.internetsociety.org/blog/2012/01/comcast-release... Learnings for me: - dnstracer (https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L57 https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L5...) helps with detecting missing glue records, but not dnssec - dnstrace (https://github.com/rs/dnstrace https://github.com/rs/dnstrace) is a better alternative with dnssec