26 ms·
DNSSEC KSK rollover breaks DNS resolution for .nz domains
- throwaway892238 3y agoIf a safety upgrade to driving a car made people crash their cars more, you'd call that a bug. For DNS it's a feature called DNSSEC.
- gg-plz 3y agoif your browser ignored all certificate errors, I guess you'd call that a feature?
- deleted 3y ago[deleted]
- tptacek 3y agoIf your browser ignored all certificate errors, you'd have a real security problem. That's not at all the case for DNSSEC: it's possible that all of the DNSSEC root keys could hit Pastebin and nobody would really need to be paged.
- belorn 3y agoBrowsers did ignore most certificate errors back in the early 2000s. HTTPS sites were fairly rare and most people did not care about it or even considered https to be a negative. Many administrators considered it as bad technology that only increased instability with no obvious benefit. "Who cares about what people post to a forum?" was something I personally heard when I added https to one site. It was only really banks with plain passwords that needed https, and then external hardware devices really made https obsolete for that problem. For more fun diving into this topic, I can recommend a famous old presentation called the "Everything you Never Wanted to Know about PKI but were Forced to find out", and godzilla crypto tutorial written by the same author (Peter gutmann). The certificates in browsers has had a long history of problems and ill designs. People did not like them, and they definitively did not like them when they caused major issues.
- teddyh 3y agoLinks: Everything you Never Wanted to Know about PKI but were Forced to Find Out: <http://www.cs.auckland.ac.nz/~pgut001/pubs/pkitutorial.pdf http://www.cs.auckland.ac.nz/~pgut001/pubs/pkitutorial.pdf> Godzilla Crypto Tutorial: <https://www.cs.auckland.ac.nz/~pgut001/tutorial/ https://www.cs.auckland.ac.nz/~pgut001/tutorial/>
- acdha 3y ago> Browsers did ignore most certificate errors back in the early 2000s. HTTPS sites were fairly rare and most people did not care about it or even considered https to be a negative. Many administrators considered it as bad technology that only increased instability with no obvious benefit. I’m not sure what you’re basing that on but every claim is the opposite of my experience back then. Even in the 90s it was expected that you used HTTPS for any site selling things, for example, as the credit card companies would block a business who let numbers go over the network in plaintext. Early on there were concerns about performance but that was mostly over by the turn of the century for all but large file transfers. The primary drawback was the cost of a certificate back then.
- fomine3 3y agoOld certification invalid dialog was terrible. I believe most people just ignored it. https://cdn.appuals.com/wp-content/uploads/2018/11/identity-cannot-be-verified-error-message.jpg https://cdn.appuals.com/wp-content/uploads/2018/11/identity-...
- acdha 3y agoYes, that’s why I found the assertion that it didn’t exist so odd since almost anyone who supported web sites or browsers back then was familiar with that dialog.
- belorn 3y agoI recall well those discussions. Web stores did indeed often use https to protect credit cards. The argument was however that physical stores did not need to have similar protection, and that the issue really was with the weak security of credit cards. HTTPS was a unstable solution for a problem which people argued should had been solved with the credit card system. Physical security devices was again lifted as the future solutions to this problem. It should also be mentioned here that credit card numbers as a security token has actually slowly been phased out in favor of other forms of payment systems online, and many banks today implement additional security requirement if you pay with a credit card. Black market with stolen CC numbers, despite https use by web stores, used to be one of the biggest issues with the internet, so even with all the stores using https it wasn't a solution to that problem. I remember people talking about performance issues with https until the early 2010. "Every single micro second slower means reduced sales" was something people was very concerned about. I even heard it from people during an IETF meeting. It was talked in similar tone to how people today talk about SEO.
- josephcsible 3y agoYour analogy would only be valid for a bug that made impersonating a site easier somehow, which this one didn't.
- Spooky23 3y agoFortunately, New Zealanders benefit from all of the problems solved by DNSSEC.
- Gigachad 3y agoIsn't DNSSEC basically obsoleted by DoH?
- tptacek 3y agoIn a sense, yes, but not so much so that DoH is a dispositive argument for deprecating it. The difference is that DoH protects transactions and DNSSEC protects the authenticity of records. It's perfectly possible for a DoH server to feed you bogus cached records; you have to trust the DoH server you're talking to, where you wouldn't have to do that if all the records on the chain of lookups you're doing are signed with DNSSEC, and you're running DNSSEC on your local system rather than a stub resolver than talks to full resolver server you have to trust (this is an uncommon set of circumstances and in practice you have exactly the same server trust problem with DNSSEC that you do with DoH). Muddying the waters further, the attacks DNSSEC protects against overlap with the ones DoH protects again, so that if the whole Internet managed to switch to DoH, you'd have bottom-up built 95% of the security feature DNSSEC is attempting to provide (DoH has massively better deployment stats than DNSSEC, so this is plausible). It's better to think of DoH as one of a catalog of different arguments that together make a clear case for sticking a fork in DNSSEC and calling it done.
- cmeacham98 3y ago> DoH has massively better deployment stats than DNSSEC, so this is plausible Is this actually true where it matters? (i.e. the root servers and authoritative servers for TLDs)?
- tptacek 3y agoWhere what matters? On-path DNS attacks occur everywhere across the Internet, and are probably more common on the lookup side and at the edges. Certainly, the use of DoH to protect authority transactions isn't common, yet!
- belorn 3y agoSome car automatically lock their doors during driving in order to prevent hijackings, especially at red lights. Same cars has issues with drivers being locked inside if the car goes into the water. That is a bug. Is the solution to abandon locks on cars, or is the solution to fix the problem of the car doors staying locked when submerged into water? The security system by now is fairly advanced and addressing issues with accidents is a real problem. No one however would sell cars without locks.
- tptacek 3y agoSeems simple: we should abandon these particular car door locks, but not necessarily the concept of car door locks altogether.
- belorn 3y agoIf we follow this analogy further, why should we keep the concept of car doors if particular car locks can be made with bugs in them? Doesn't the possibility of bugs in locks means that there will always be a risk, even if we abandon specific locks that has demonstrated to have a bug in them?
- tptacek 3y agoWe should keep the locks that work and don't cause other problems, and ditch the other ones. Again, seems simple.
- smaudet 3y agoDepends whether locks can be made to work. Solutions exist within problem spaces, or "paradigms", a car lock is only useful for a specific set of things i.e. deterring thieves or unwanted entry e.g. at stop lights. What I really want is a force-field to keep people and highway debris out while driving around, and a secure storage solution while not in operation. If you could sell me a force-field car with a retracting tent, and it were safer/had less issues, I might not care to have locks on my car. See also - door-less/open air vehicles. Car door locks have some pretty serious issues. There are some problems they just can't solve.
- bawolff 3y agoI'm not really a fan of dnssec, but in fairness to the analogy - car keys really do make life harder if you need to rush someone to the hospital and dont have the keys.
- neurostimulant 3y agoMaybe the cars crash more, but at least they won't take any wrong turn.
- deleted 3y ago[deleted]
- lucgagan 3y agoI hope the situation gets resolved swiftly, and lessons learned from this incident can contribute to stronger and more reliable DNSSEC practices in the future.
- tptacek 3y agoThe root KSK rollover had to be postponed twice, for several years, because of operational problems pulling this off in the US. It's difficult to do because the nature of the DNS makes it difficult. The utility of DNSSEC is so marginal that it's kind of sad that you're right, and that engineers are gradually getting better doing this hard, stupid thing.
- tedunangst 3y agoYou're not learning if you're not failing, so failing is good actually.
- CognitiveLens 3y agoI think the pithy saying is that we learn best from failure, not that there is no other way to learn.
- theamk 3y agoYep: "don't deploy DNSSEC, rely on TLS"
- matthew9219 3y agoTLS security is rooted in DNS. It's ACME DNS-01. If your threat model includes nation states, this is a non-solution
- Avamander 3y agoIf your threat model includes nation-states then DNSSEC won't help you either. WebPKI at least has a method for keeping track of and detecting misissuance, DNSSEC doesn't.
- talideon 3y agoBeen there, done that, albeit on a smaller scale. And I don't envy the people running the .nz registry at all, and they have my best wishes.
- cyberax 3y agoDNSSEC designers screwed up by making rollovers to be atomic. Instead, they should have allowed the responses to be signed by two keys. And a way to specify as a hint which key should be used, so that the zone owner could gather feedback on the rollover safety.
- teddyh 3y agoNo, DNSSEC does allow for multiple signatures. You can use tools like dnsviz.net to see which key is valid from upstream (if you don’t know how to do it manually).
- silisili 3y agoYou can and should sign by multiple keys before/during a rollover. That's exactly how it's supposed to work. The client is to check all, and any valid chain.
- quink 3y agohttps://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
- amenghra 3y agoSo many outages. About one every other month affecting country TLDs.
- zeff_henry 3y agoFortunately, I was spared from witnessing any swamping in this DNSSEC, and all credit goes to onenz for instigating positive transformation.
- hsbauauvhabzb 3y agoIn 2020 I scraped fortune top 500 companies for dnssec and found iirc one domain using dnssec. It certainly feels like the wrong way of solving problems (ramming more into the domain registry always seems like a bad option). Is the technology dead or destined to fail? Edit: rationale: dnssec solves domain validity, but https tls solves almost the same problem but has better backing (azure said they don’t support dnssec and recommended tls as a better alternative). Dnssec also does not solve bgp hijacking, which combined with ip based tls signing servers moots any value dnssec has - sure you could registrar lock your domain via dns (preventing letsencrypt signing things), but if a threat actor has the capability to bgp hijack to perform such an attack and is targeting you, you probably have bigger issues elsewhere.
- teddyh 3y agoWhen criticizing DNSSEC, you can’t assume that the system for TLS certificates – i.e. CAs – is perfect. They both have their weak points and drawbacks. Both BGP and certificate issuance have bootstrapping problems, which are handled today by imperfect TOFU-like solutions. DNSSEC is, IMHO, perfectly positioned to solve both of those problems. I.e. use certificates all you like, but verify them by looking up the TLSA record in the DNS using DNSSEC. No need to trust CAs. BGP could possibly use the same solution, using the reverse lookup .arpa DNS space. DNSSEC is the building block from which secure certificates and BGP routes can be built, without the ad-hoc CA system we have today.
- tptacek 3y agoPeople say this all the time, but of course the WebPKI has Certificate Transparency, requiring every issuer to register every certificate issued in a globally monitored tamper-proof log, and DNSSEC doesn't. Moreover, the WebPKI got CT because the browser root programs were able to force the CAs to join it. They have no such influence over DNS registrars, many of which are de jure controlled by world governments and will never consent to transparency logging. This very much includes the US, which actively manipulates the DNS for policy ends. If Comodo knowingly misissues a Google Mail certificate, Google will nuke them from orbit, as it has done in the past with other major CAs. Google can't do anything about .COM mis-signatures. Thankfully, practically none of .COM is signed.
- teddyh 3y agoWhat the real problem probably is, is that all this is still being done by hand. It should be automated, and there is some work being done in that area: <https://github.com/DNSSEC-Provisioning/music https://github.com/DNSSEC-Provisioning/music>
- davidu 3y agoDNSSEC is easily the worst upgrade, multiplying complexity and brittleness, with the least amount of net benefit (without even adding encryption), that could have been solved in much simpler ways, that the Internet has ever attempted -- and that's including IPv6 (which is now quite workable). Speaking as someone who most people consider a DNS expert and actually did help develop and deploy something substantially additive that is in widespread use today (DNSCrypt). ¯\_(ツ)_/¯
- oittaa 3y agoAt this point it feels like DNS should be given to Cloudflare or Google and let them design it from scratch. I'm only half joking.
- XorNot 3y agoI'm not real enthused about Google doing standards. OAuth/OAuth2 are both so half-baked that we now have OIDC built atop them to try and make it look like a consistent workable standard. Google is very enthusiastic it seems about things which force users to use Google Chrome, and very unenthusiastic about users doing anything easily from the command line because it has the notable quality of removing a place you can show ads. And what I note about the whole OAuth ecosystem is that you wind up having to puppet a web browser in order to get through sign-ins and the like. "Oh but you do it infrequently" says every single company implementing their own bespoke way of entering a username, password and TOTP while salivating at all that unused <div> space for ads.
- 1vuio0pswjnm7 3y ago"Google is very enthusiastic it seems about things which force users to use Google Chrome, and very unenthusiastic about users doing anything easily from the command line because it has the notable quality of removing a place you can show ads." What else would we expect from an advertising company.
- silisili 3y agoFor sure. DNS is definitely something that should either be canned after 2 years, or protected by 50 captchas.
- contingencies 3y agoEconomic impact of no-dns-resolutio.nz?
- matthew9219 3y agoIt seems like some folks are missing the motivation for DNSSec and suggesting TLS instead. If your threat model includes global adversaries, you have can't rely on TLS because governments can trivially compromise TLS providers and TLS exposes users to the lowest common denominator TLS. The lowest common denominator TLS (ACME DNS-1) and the mitigation to the TLS provider problem (CAA records) are both based on DNS. So you either accept that TLS is the global maxima for security and world governments can basically permanently compromise the internet, or you build private PKI systems, or you want something like DNSSec. And DNSSec is something like DNSSec.
- insanitybit 3y agoLet's say the US wanted to perform an attack that DNSSEC would have prevented, what does that attack look like?
- matthew9219 3y agoThe US seizes the cryptographic material for a US based root, issues keys and certificates for the domains it wants to compromise and intercepts and modifies the traffic for targeted users. There's some additional asterisks around not getting caught and certificate transparency logs and browser reporting structure, but for many classes of devices, it will suffice to simply also hijack the domains used for requesting the transparency log or the domains used for reporting certificates that don't appear in the log. Users who are concerned about a government like the United States can use DNSSec to prevent a threat like this by using a non-US based TLD that employs DNSSec, and by running their client in a mode that requires valid DNSSec records for their domains. Of course, such services would practically need to be located outside of the country of concern as well.
- CommanderData 3y agoFor a state like the US, with it's laws and history on surveillance. I assume PKI has been compromised. I don't check or audit my CA's and don't think most people do either. Wouldn't be surprised if more than one of these has been compromised in some fashion already. It only takes one and there's plenty to target. The next thing you'd need is a mitm attack and again that's entirely possible for a nation state to pull off at scale.
- denton-scratch 3y agoWhy can't I read this in reader-mode? Supplementary question: Why do so many sites these days opt for tiny font-sizes in some shade of pale-grey on white?