6 ms·

I know we’ve all collectively accepted that DNSSEC is a terrible, complicated blight on the world but I still find it incredible that that an organisation with

I know we’ve all collectively accepted that DNSSEC is a terrible, complicated blight on the world but I still find it incredible that that an organisation with Slacks resources and access to expertise can’t make it work.

toomuchtodo 5y ago
No tech company is infallible. All of them have outages, some lasting hours, even days. Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless. One day it might just be your turn to break production.
- tptacek 5y ago
  The subtext here isn't that Slack is bad at this (they are not), but that DNSSEC is somehow intrinsically unsafe (it probably is).
  - toomuchtodo 5y ago
    I agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS. Perhaps I did not read the room appropriately. Mea culpa.
    - Operyl 5y ago
      Did Roblox finally come out with their postmortem blaming Consul? As far as I know we just assumed it, but have had no update since October.
      - toomuchtodo 5y ago
        https://news.ycombinator.com/item?id=29063026 https://news.ycombinator.com/item?id=29063026
tptacek 5y ago
You say Slack, and I agree, that's telling, but you have to add to that AWS itself, which had a DNSSEC bug in its wildcard record support as well. Slack and AWS together couldn't make this feature work. Further: the open source tooling Slack (like most places) relies on for deployment is also DNSSEC-hostile: one of their problems is that Terraform's Route53 provider doesn't safely disable DNSSEC once enabled. It's a mess everywhere you look. I think another interesting question here is why Slack bothered in the first place. As was pointed out on the other DNSSEC thread today: practically nobody in the technology industry uses DNSSEC in the first place. Presumably, Slack did DNSSEC (they don't anymore!) in service of FedRAMP compliance. Why? Slack has one of the most popular products in all of computing. What bad thing was going to happen if they said "nah, we're going to go with Cloud.gov's recommendation and not this FedRAMP document"?
- vimda 5y ago
  Gotta be Fedramp compliant to do business with the US government. Even worse, you have to be Fedramp compliant to work with anyone who works with the US government. From a business (if not an engineering) standpoint, there's plenty to gain in going through the motions
  - tptacek 5y ago
    As was pointed out downthread, there are tech companies that are "more" FedRAMP compliant (FedRAMP "High") without DNSSEC support. (Kenn White points out on Twitter that some of this may be due to grandfathering --- though, the FedRAMP DNSSEC requirement is pretty old.)
    - aidenn0 5y ago
      I don't know about FedRAMP, but with other government requirements, the easiest way to get an exception was to fail badly at implementing the retirement. When the DOD tried to mandate Ada, lots of projects were bid as Ada, then switched to C++ at the very first sign of any trouble whatsoever. I would 100% believe it if someone told me that this horrible rollout could be leveraged into an exemption from needing DNSSEC
belorn 5y ago
"It turned out that some resolvers become more strict when DNSSEC signing is enabled at the authoritative name servers, even while signing was not enabled at the root name servers (i.e. before DS records were published to COM nameservers). This strict DNS spec enforcement will reject a CNAME record at the apex of a zone (as per RFC-2181), including the APEX of a sub-delegated subdomain" Slack's second attempt wasn't a DNSSEC problem. Slack depended on a permissive fallback of revolvers when encountering a plain DNS protocol error. It is similar to how some websites in the past relied on permissive browsers implementation when facing broken HTML/JS/CSS. Slack fixed their broken DNS as a result of this. Slack's third attempt was not the fault of Slack but rather a software bug at Amazon. I would make the argument that Amazon's primary product isn't DNS services, but they did fixed their bug after this. The general conclusion I get from the article is not that DNSSEC is broken, nor that is too complicated. It is that when doing changes with your core infrastructure to make it more secure, bugs that may have been laying dormant might pop up and bite. I am sure some people has had that experience in domains outside of DNS.
- ignoramous 5y ago
  You are not wrong, but by steering clear of DNSSEC, Slack would not have had the outage they did. What one can't ignore is the underlying chicken-and-egg problem that DNSSEC must overcome: Not many DNSSEC deployments and hence not much of it has been tested in the real-world, which results in colossal outages despite the attention of some of the most qualified engs, including the ones running one of the largest nameserver deployments in the world. TLS and WebPKI has had a similar, perhaps even more painful route to ubiquity. So, this problem isn't unique to DNSSEC. What isn't working in DNSSEC's favour is, the world has not just moved on, but it has built solutions atop DNS' weaknesses, like it once did with IPv4 and NAT. Internet's strong network-effects coupled with its heterogeneity, make battling "the System" an even harder proposition. See also: System design explains the world: Vol 1, https://apenwarr.ca/log/20201227 https://apenwarr.ca/log/20201227
technion 5y ago
I know HN has collectively accepted but every time I'm associated with an organisation that pays for a penetration test it comes in as a high risk finding, so much so that I've given in to deploying it to avoid sitting with non-technical managers doing the "here's why I disagree" all over again. Outside of this group I definitely feel like I'm on my own in that view.

toomuchtodo 5y ago

No tech company is infallible. All of them have outages, some lasting hours, even days. Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless. One day it might just be your turn to break production.

tptacek 5y ago
The subtext here isn't that Slack is bad at this (they are not), but that DNSSEC is somehow intrinsically unsafe (it probably is).
- toomuchtodo 5y ago
  I agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS. Perhaps I did not read the room appropriately. Mea culpa.
  - Operyl 5y ago
    Did Roblox finally come out with their postmortem blaming Consul? As far as I know we just assumed it, but have had no update since October.
    - toomuchtodo 5y ago
      https://news.ycombinator.com/item?id=29063026 https://news.ycombinator.com/item?id=29063026

tptacek 5y ago

You say Slack, and I agree, that's telling, but you have to add to that AWS itself, which had a DNSSEC bug in its wildcard record support as well. Slack and AWS together couldn't make this feature work. Further: the open source tooling Slack (like most places) relies on for deployment is also DNSSEC-hostile: one of their problems is that Terraform's Route53 provider doesn't safely disable DNSSEC once enabled. It's a mess everywhere you look. I think another interesting question here is why Slack bothered in the first place. As was pointed out on the other DNSSEC thread today: practically nobody in the technology industry uses DNSSEC in the first place. Presumably, Slack did DNSSEC (they don't anymore!) in service of FedRAMP compliance. Why? Slack has one of the most popular products in all of computing. What bad thing was going to happen if they said "nah, we're going to go with Cloud.gov's recommendation and not this FedRAMP document"?

vimda 5y ago
Gotta be Fedramp compliant to do business with the US government. Even worse, you have to be Fedramp compliant to work with anyone who works with the US government. From a business (if not an engineering) standpoint, there's plenty to gain in going through the motions
- tptacek 5y ago
  As was pointed out downthread, there are tech companies that are "more" FedRAMP compliant (FedRAMP "High") without DNSSEC support. (Kenn White points out on Twitter that some of this may be due to grandfathering --- though, the FedRAMP DNSSEC requirement is pretty old.)
  - aidenn0 5y ago
    I don't know about FedRAMP, but with other government requirements, the easiest way to get an exception was to fail badly at implementing the retirement. When the DOD tried to mandate Ada, lots of projects were bid as Ada, then switched to C++ at the very first sign of any trouble whatsoever. I would 100% believe it if someone told me that this horrible rollout could be leveraged into an exemption from needing DNSSEC

belorn 5y ago

"It turned out that some resolvers become more strict when DNSSEC signing is enabled at the authoritative name servers, even while signing was not enabled at the root name servers (i.e. before DS records were published to COM nameservers). This strict DNS spec enforcement will reject a CNAME record at the apex of a zone (as per RFC-2181), including the APEX of a sub-delegated subdomain" Slack's second attempt wasn't a DNSSEC problem. Slack depended on a permissive fallback of revolvers when encountering a plain DNS protocol error. It is similar to how some websites in the past relied on permissive browsers implementation when facing broken HTML/JS/CSS. Slack fixed their broken DNS as a result of this. Slack's third attempt was not the fault of Slack but rather a software bug at Amazon. I would make the argument that Amazon's primary product isn't DNS services, but they did fixed their bug after this. The general conclusion I get from the article is not that DNSSEC is broken, nor that is too complicated. It is that when doing changes with your core infrastructure to make it more secure, bugs that may have been laying dormant might pop up and bite. I am sure some people has had that experience in domains outside of DNS.

ignoramous 5y ago
You are not wrong, but by steering clear of DNSSEC, Slack would not have had the outage they did. What one can't ignore is the underlying chicken-and-egg problem that DNSSEC must overcome: Not many DNSSEC deployments and hence not much of it has been tested in the real-world, which results in colossal outages despite the attention of some of the most qualified engs, including the ones running one of the largest nameserver deployments in the world. TLS and WebPKI has had a similar, perhaps even more painful route to ubiquity. So, this problem isn't unique to DNSSEC. What isn't working in DNSSEC's favour is, the world has not just moved on, but it has built solutions atop DNS' weaknesses, like it once did with IPv4 and NAT. Internet's strong network-effects coupled with its heterogeneity, make battling "the System" an even harder proposition. See also: System design explains the world: Vol 1, https://apenwarr.ca/log/20201227 https://apenwarr.ca/log/20201227

technion 5y ago

I know HN has collectively accepted but every time I'm associated with an organisation that pays for a penetration test it comes in as a high risk finding, so much so that I've given in to deploying it to avoid sitting with non-technical managers doing the "here's why I disagree" all over again. Outside of this group I definitely feel like I'm on my own in that view.