3 ms·
> * The incredibly bad track record DNSSEC has for reliability; for instance, Slack had a multi-hour total sitewide outage earlier this year that came from atte
by ThePowerOfFuet 4y ago
> * The incredibly bad track record DNSSEC has for reliability; for instance, Slack had a multi-hour total sitewide outage earlier this year that came from attempting to turn it on.
It was actually from turning it _off_, relying on expert advice that turned out to be incorrect:
>Our Traffic team was confident that reverting the DS record published in MarkMonitor was sufficient to eventually resolve DNS resolution problems. As things were not getting any better, and given the severity of the incident, we decided to rollback DNSSEC signing in our slack.com authoritative and delegated zones, wanting to recover our DNS configuration to the last previous healthy state, allowing us to completely rule out DNSSEC as a problem.
>As soon as the rollback was pushed out things got much worse; our Traffic team was paged due to DNS resolution issues for slack.com from multiple resolvers across the world.
>Based on expert advice, our understanding at the time was that DS records at the .com zone were never cached, so pulling it from the registrar would cause resolvers to immediately stop performing DNSSEC validation. This assumption turned out to be untrue. The ‘.com.’ zone will ask resolvers to cache the slack.com DS record for 24h by default (non modifiable by zone owners). Luckily, some resolvers cap that TTL with a lower value (i.e. Google’s 8.8.8.8 has a 6h TTL on DS records).
https://slack.engineering/what-happened-during-slacks-dnssec-rollout/ https://slack.engineering/what-happened-during-slacks-dnssec...