8 ms·
The case of the recursive resolvers: What happened during Slack’s DNSSEC rollout
- jeffbee 5y agoSeems like an organizational failure, as they got conned by their 3PAO into believing that DNSSEC was a requirement for FedRAMP moderate when it's not. The disproof of this belief is that Google has FedRAMP High (for Google Cloud and Workspace) but does not use DNSSEC for google.com.
- goalieca 5y agoIf you use https everywhere, you will have a server certificate with the hostname embedded in it. This is how TLS knows you’re talking to the right server.
- mpyne 5y agoThe ultimate arbiter of whether a cloud service gets used isn't FedRAMP, it's the Agency Authorizing Official. FedRAMP just makes much of the work reusable. With GCP, you can build something that obeys and uses DNSSEC without needing google.com to participate in DNSSEC. Google Workspace is a good point though. I know there are many users of it in government... maybe some AOs are fine signing off on it even without the needed security controls, which is an option they have in their discretion with and without FedRAMP.
- dsXLII 5y agoIt's always DNS.
- dogecoinbase 5y agoIn addition to the other note that DNSSEC is _not_ required for FedRAMP certification (it's even discouraged by cloud.gov! https://cloud.gov/docs/compliance/domain-standards/ https://cloud.gov/docs/compliance/domain-standards/ ), this is some weirdly intellectually dishonest phrasing (linking to tptacek's article Against DNSSEC: https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/ ): > While we are aware of the debate around the utility of DNSSEC among the DNS community, we are still committed to securing Slack for our customers. The argument is specifically that it doesn't provide that security. At least it's neat to see actual begging the question in the wild, I guess.
- tylermenezes 5y ago> intellectually dishonest phrasing Not everyone agrees with the linked argument. For example, I disagree that browsers can't take advantage of DNSSEC, since many are using DoH, and the rest of the article reads like someone complaining that we need to wait for the perfect protocol or nothing at all. That's the thing about a debate... it's got arguments on both sides.
- tptacek 5y agoI mean, I agree with you and don't find the language disingenuous (I felt like it was more of a tell that the people working on this cursed project weren't super read into DNSSEC and DNS security in general, which isn't a knock; it's a boring thing to keep up with, especially when the best-practice answer is so simple --- just don't bother with DNSSEC). But I'd also say that DoH (1) largely obviates any need for DNSSEC (the last-mile DNS problem is the only on-the-wire DNS security problem that needs solving) and (2) doesn't enable DANE in browsers, which is what people are talking about when they talk about DNSSEC intersecting with browsers in any way other than randomly making sites fall off the Internet.
- dogecoinbase 5y agoIt's fine to disagree with the linked argument, but you actually have to do so. This is them presupposing that "securing Slack for [their] customers" requires DNSSEC -- it's not engaging with the argument at all.
- Joe8Bit 5y agoI know we’ve all collectively accepted that DNSSEC is a terrible, complicated blight on the world but I still find it incredible that that an organisation with Slacks resources and access to expertise can’t make it work.
- toomuchtodo 5y agoNo tech company is infallible. All of them have outages, some lasting hours, even days. Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless. One day it might just be your turn to break production.
- tptacek 5y agoThe subtext here isn't that Slack is bad at this (they are not), but that DNSSEC is somehow intrinsically unsafe (it probably is).
- toomuchtodo 5y agoI agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS. Perhaps I did not read the room appropriately. Mea culpa.
- Operyl 5y agoDid Roblox finally come out with their postmortem blaming Consul? As far as I know we just assumed it, but have had no update since October.
- toomuchtodo 5y ago
- tptacek 5y agoAdditional discussion, indirectly and spurred from this, is here: https://news.ycombinator.com/item?id=29381778 https://news.ycombinator.com/item?id=29381778 That thread, which is big, is probably the right place to take general discussion of DNSSEC itself, though I'll snipe DNSSEC here too. :)
- deleted 5y ago[deleted]
- teddyh 5y agoFrom what I can tell, the problem was not caused by DNSSEC directly. It was caused by: 1. A bug in Route 53 which caused wildcard record not to work with DNSSEC signing. Anyone not using Route 53 would not have had any problems with DNSSEC. 2. Slack decided to revert the DNSSEC rollout, but botched the process badly, effectively locking themselves in the trunk and throwing away the key. If they hadn’t tried to revert the DNSSEC rollout, or if they had been a bit more deliberate and careful while doing it, this would not have happened.
- native_samples 5y agoThe issue wasn't really lacking deliberateness or care but rather that they had adopted a culture of over automation. Why are they using Terraform to make changes that are rare and critical instead of the GUI? The whole point of a GUI is to make computers easier and safer to use, but if you insist on accepting everything then you lose that.
- daper 5y agoFrom the described mistakes two come from lack of understanding how exactly DNS works. But I agree it's in fact hard, see [1]). 1. "This strict DNS spec enforcement will reject a CNAME record at the apex of a zone (as per RFC-2181), including the APEX of a sub-delegated subdomain. This was the reason that customers using VPN providers were disproportionately" - This is non intuitive and maay people are surprised by that. You cannot create any subdomain (even www.domain.tld) if you created "domain.tld CNAME something...". Looks like not every server/resolver enforces that restriction. 2. "based on expert advice, our understanding at the time was that DS records at the .com zone were never cached, so pulling it from the registrar would cause resolvers to immediately stop performing DNSSEC validation." - like any other record, they can be cached. DNS has also negative caching (caching of "not found responses". Moreover there are resolvers that allow configuring minimum TTL that can be higher that what your NS servers returns (like unbound - "cache-min-ttl" option) or can be configured to serve stale responses in case of resolution failures after the cached data expires [2]. That means returning TTL of "1s" will not work as you expect. [1] https://blog.powerdns.com/2020/11/27/goodbye-dns-goodbye-powerdns/ https://blog.powerdns.com/2020/11/27/goodbye-dns-goodbye-pow... [2] https://www.isc.org/blogs/2020-serve-stale/ https://www.isc.org/blogs/2020-serve-stale/
- btown 5y agoMy (basic and conservative) mental model that "in DNS, everything including the lack of presence of a thing can be cached" is why I'm very cautious before rolling out anything from DKIM to DNSSEC. A deep understanding of specifications is vital. I'm somewhat surprised an organization of Slack's scale didn't have a consultant on the level of "I designed DNSSEC" on hand for this.