Last week. Updating DNSSEC infrastructure for a bunch of zones we manage. As part of that update I added extra DS* records at the parent zones. We were using

Last week. Updating DNSSEC infrastructure for a bunch of zones we manage. As part of that update I added extra DS* records at the parent zones. We were using SHA-256 and since our new signer generated SHA-384 signatures as well I included them. All nice and standard. Our tests were all good. dnsviz.net said everything was good, but... It turns out the version of Unbound that was included in older versions of Ubuntu and Redhat can't handle SHA-384 DS records and instead of ignoring them like they are supposed to simply fail validation. Naturally those obsolete nameservers are in use by fun people like Comcast who proceeded to drop lookups for the entire zone. This particular zone was the major commercial zone for a small country ie. co.tld. Around a million subdomains. Fortunately it this was a subdomain inside a country so we could get the SHA-384 signatures dropped at the parent quickly, but yeah tracking down the problem was ... Lets just say it was a bad day. I wonder if I can put "I broke the internet" on my CV? --- * English for everyone who doesn't speak DNSSEC: DNSSEC is kind of like SSL but for DNS. A DS (Delegation Signer) record is a hash that goes in the parent zone so that resolvers can verify you are using the correct DNSKEYs. Its kind of like X509 certificates in reverse. DNSSEC is VERY complicated and brittle with many ways to hurt yourself.