3 ms·
Technical tl;dr: The DS record was cached for 24 hours but the DNSKEY record was cached for 1 hour. The DS and DNSKEY records were rotated due to an annual ro
by mmsc 3y ago
Technical tl;dr:
The DS record was cached for 24 hours but the DNSKEY record was cached for 1 hour.
The DS and DNSKEY records were rotated due to an annual rotation of the signing key.
The (cached) old DS record referenced the old DNSKEY. This inconsistency invalidated records for resolvers which validated dnssec records (85% of New Zealanders users, mostly through their ISPs’ resolvers.)
Procedural tl;dr:
Management was not informed of the incident until the next day.
DNS operators were not made aware of the rotation of the KSK (key signing key) so the problem by outside operators was difficult to theorise.
- comex 3y agoAlso, the software they were using (OpenDNSSEC) had an automated delay to avoid exactly this issue, but the delay period was based on a setting in its config file that was supposed to reflect the TTL of the DS record, and they forgot to change it when the TTL changed.
- jeroenhd 3y agoI think that is the core issue. I do wonder why this TTL is determined by a config file and not a direct TTL query to the root servers. A mismatch between the two should at least provide a warning, I would think?
- mcesch 3y agoOpenDNSSEC's first release was before the root was signed so perhaps there's some assumptions in it that need revisiting.