6 ms·

Oh. I thought it burned down because of their engineers not having fully acquainted themselves with the tool before applying it. It's misguided to hold DNSSEC c

Oh. I thought it burned down because of their engineers not having fully acquainted themselves with the tool before applying it. It's misguided to hold DNSSEC culpable for Slack's engineers' goof-up. Like advising people against ever going near scissors because they might run with one in their hands.

tptacek 1y ago
LOL. https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
- daneel_w 1y ago
  Miserable list. Now share a link that shows all the successful deployments that never had any hiccups.
  - Avamander 1y ago
    It's not like anyone is actually monitoring it or would notice. I know for a fact there are a lot of broken deployments out there, have been for a decade if not more, nobody really gives a rats ass. From a technical perspective, lack of DNSSEC transparency is also a major downside of DNSSEC compared to WebPKI.
- growse 1y ago
  Apparently the way to do sane risk management is just "don't be an idiot"?
marcusb 1y ago
That is an extremely uncharitable take for an outage that involved two of the most poorly defined and difficult to (correctly) implement DNS features (wildcards and NSEC records) that the three major DNS operators mentioned in the Slack post-mortem (AWS, Cloudflare, and Google) all implemented differently.
- teddyh 1y ago
  IIRC, Slack backed out of their DNSSEC deployment because of a bug with wildcards and NSEC records (in Route53, not Slack), but the problem Slack subsequently experienced was not caused by that bug, but was instead caused by the boneheaded way in which Slack tried to back out of DNSSEC. I.e. Slack’s problem was entirely their own doing, and completely avoidable if they had had any idea of what they were doing.
  - tptacek 1y ago
    Slack's actual internal take on DNSSEC is now recorded in this thread, for what it's worth.
    - growse 1y ago
      This feels like a classic example of "actions at the sharp end resolve all ambiguity" from Richard Cook's 1998 paper on complex systems.
  - marcusb 1y ago
    Having read the post mortem, I disagree. Slack engineers did something dumb, under pressure during an outage. Even if they hadn't, they still would have been in a degraded state until they could properly remove their DNSSEC records and/or get the Route53 bug they hit fixed. In other words, they still would have had a 24+ hour outage, albeit with a smaller blast radius. The design of DNSSEC is simply not fit for purpose for zone operators. It is far too easy to screw your zone up for far too marginal a benefit, to say nothing of the huge increase in CPU resource required to authenticate DNSSEC record chains. The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options. To give just one example, consider the NSEC3 iteration limit field. NSEC3 itself was designed mostly[0] to prevent zone enumeration when validating negative responses (which is trivial to perform with NSEC.) The iteration count was designed to give zone operators the ability to increase the cost to an attacker of generating a dictionary of nsec3 names[1]. Of course, a high iteration count also raises the cost to a non-malicious resolver of validating a negative response for a nsec3-enabled zone. In good old DNSSEC fashion, the iterations field is a single number that is subject to a... wide variety of potential limits: * 16 bits by the wire protocol * 150 for 1,024 bit keys (RFC 5155 10.3[2]) * 500 for 2,048 bit keys (RFC 5155 10.3[2]) * 2,500 for 4,096 bit keys (RFC 5155 10.3[2]) * 0 (RFC 9276 3.2) Why 0? It was noted -- after publishing the NSEC3 spec -- that high iterations just don't provide that much benefit, and come with a high cost to throughput. Appendix B of RFC 9276 shows a roughly 50% performance degradation with an iteration count of 100. So, RFC 9276 3.2 says: Validating resolvers MAY also return a SERVFAIL response when processing NSEC3 records with iterations larger than 0. Of course, their guidance to implementers is to set the limits a bit higher, returning insecure responses at 100 iterations and SERVFAIL at 500. That said, if you want to be maximally interoperable, as a zone operator, you should pretend like the iteration count field doesn't exist: it is standards compliant for a validating resolver to refuse an nsec3 response with more than a single hash round. As I said, this is one example, but I'm not cherry picking here. The whole of the DNSSEC spec corpus is filled with incomprehensible verbiage and opportunities for conflicting interpretations, far beyond what you see in most protocol specs. 0 - also to reduce the size of signed top-level zones 1 - all NSEC and NSEC3 records, while responsive to queries about names that don't exist, consist of obfuscated names that do exist. 2 - According to the letter of the standard, the limits applied to the iterations field should be 149, 499, and 2,499. Implementations are inconsistent about this.

tptacek 1y ago

LOL. https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html

daneel_w 1y ago
Miserable list. Now share a link that shows all the successful deployments that never had any hiccups.
- Avamander 1y ago
  It's not like anyone is actually monitoring it or would notice. I know for a fact there are a lot of broken deployments out there, have been for a decade if not more, nobody really gives a rats ass. From a technical perspective, lack of DNSSEC transparency is also a major downside of DNSSEC compared to WebPKI.
growse 1y ago
Apparently the way to do sane risk management is just "don't be an idiot"?

marcusb 1y ago

That is an extremely uncharitable take for an outage that involved two of the most poorly defined and difficult to (correctly) implement DNS features (wildcards and NSEC records) that the three major DNS operators mentioned in the Slack post-mortem (AWS, Cloudflare, and Google) all implemented differently.

teddyh 1y ago
IIRC, Slack backed out of their DNSSEC deployment because of a bug with wildcards and NSEC records (in Route53, not Slack), but the problem Slack subsequently experienced was not caused by that bug, but was instead caused by the boneheaded way in which Slack tried to back out of DNSSEC. I.e. Slack’s problem was entirely their own doing, and completely avoidable if they had had any idea of what they were doing.
- tptacek 1y ago
  Slack's actual internal take on DNSSEC is now recorded in this thread, for what it's worth.
  - growse 1y ago
    This feels like a classic example of "actions at the sharp end resolve all ambiguity" from Richard Cook's 1998 paper on complex systems.
- marcusb 1y ago
  Having read the post mortem, I disagree. Slack engineers did something dumb, under pressure during an outage. Even if they hadn't, they still would have been in a degraded state until they could properly remove their DNSSEC records and/or get the Route53 bug they hit fixed. In other words, they still would have had a 24+ hour outage, albeit with a smaller blast radius. The design of DNSSEC is simply not fit for purpose for zone operators. It is far too easy to screw your zone up for far too marginal a benefit, to say nothing of the huge increase in CPU resource required to authenticate DNSSEC record chains. The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options. To give just one example, consider the NSEC3 iteration limit field. NSEC3 itself was designed mostly[0] to prevent zone enumeration when validating negative responses (which is trivial to perform with NSEC.) The iteration count was designed to give zone operators the ability to increase the cost to an attacker of generating a dictionary of nsec3 names[1]. Of course, a high iteration count also raises the cost to a non-malicious resolver of validating a negative response for a nsec3-enabled zone. In good old DNSSEC fashion, the iterations field is a single number that is subject to a... wide variety of potential limits: * 16 bits by the wire protocol * 150 for 1,024 bit keys (RFC 5155 10.3[2]) * 500 for 2,048 bit keys (RFC 5155 10.3[2]) * 2,500 for 4,096 bit keys (RFC 5155 10.3[2]) * 0 (RFC 9276 3.2) Why 0? It was noted -- after publishing the NSEC3 spec -- that high iterations just don't provide that much benefit, and come with a high cost to throughput. Appendix B of RFC 9276 shows a roughly 50% performance degradation with an iteration count of 100. So, RFC 9276 3.2 says: Validating resolvers MAY also return a SERVFAIL response when processing NSEC3 records with iterations larger than 0. Of course, their guidance to implementers is to set the limits a bit higher, returning insecure responses at 100 iterations and SERVFAIL at 500. That said, if you want to be maximally interoperable, as a zone operator, you should pretend like the iteration count field doesn't exist: it is standards compliant for a validating resolver to refuse an nsec3 response with more than a single hash round. As I said, this is one example, but I'm not cherry picking here. The whole of the DNSSEC spec corpus is filled with incomprehensible verbiage and opportunities for conflicting interpretations, far beyond what you see in most protocol specs. 0 - also to reduce the size of signed top-level zones 1 - all NSEC and NSEC3 records, while responsive to queries about names that don't exist, consist of obfuscated names that do exist. 2 - According to the letter of the standard, the limits applied to the iterations field should be 149, 499, and 2,499. Implementations are inconsistent about this.