9 ms·

DNSSEC disruption affecting .de domains – Resolved

747 points by warpspin 5mo ago

adamas 5mo ago
I wasn't even aware that was possible..?
warpspin 5mo ago
Whole .de TLD seems to go offline right now due to dnssec or missing nic.de nameservers?
- fweimer 5mo ago
  This works: $ unbound-host -t A www.denic.de www.denic.de has address 81.91.170.12 This does not: $ unbound-host -D -t A www.denic.de www.denic.de has address 81.91.170.12 validation failure <www.denic.de. A IN>: signature crypto failed from 194.246.96.1 for DS denic.de. while building chain of trust So it does seem DNSSEC-related. EDIT My explanation was wrong, this is not how keytags work. The published keytag data is consistent: de. 3600 IN DNSKEY 256 3 8 AwEAAfRLmzuIXVf7x5A0+U7hke0dS+GEJG0EdPhnOthCCLhy0t0WqLyoXJOhnfsTJ8vQX5fd9qOJc9gyr3SWJZkXAhPm3yPSC7FWWHF70WZTKKM9CekmKdqwMwq6ZCjMSUcecCuSF4Sbt1MRszV7rFmfGVklA1l5UzNbqwD+Dr5vfcLn ;{id = 33834 (zsk), size = 1024b} de. 3600 IN DNSKEY 257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b} de. 3600 IN DNSKEY 256 3 8 AwEAAbTe1PJi8EgIudNGb+KRTxBL2aCu5rXkZ+aIe/TC88pwRdrXYeXODp1ihZWFop5CrbWRBLrk/YUPBE8aBc6oJP+58dSkdMLYkjSkmvdvYx+zXnRLWlF2bapxvZxshATJDfGjGbCiWxKEOoyRx3UhICtHC+cUSddsEvzfacUcBb6n ;{id = 32911 (zsk), size = 1024b} de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519030655 20260505013655 26755 de. ke56T5GZt/X6zMBAF+ouyCTnAd7RY7MsnDcfa9jyyOwSouRXhvzim/V13JDTMBAnpAHxWQXoruXrAZ6A6re5N+8Pp2utVkAEKTWs0r4UOLNKoZ2+zMwNplKjNNnY5PJIbHfa5myyziLiIsi//qDIgQEACFk+pZcHXrRdqRoXPCL3UtfaXjk3+duDQdlPnYsJys5UshjVpkALSMChW7J0anzr0sG+f9ytstBneymMwFYOUC3NqbejbLPZsXGPZBQKPAoVJuV5q3znopbcqrDFfjI7bmX3QPYNvOaiT1ElBfi2piJVpDzMaMAmm2jCmvrf5VeTOBccMroh8sBtDPsaEg== ;{id = 26755} The signature on the SOA record still does not verify: de. 86400 IN SOA f.nic.de. dns-operations.denic.de. 1778014672 7200 7200 3600000 7200 de. 86400 IN RRSIG SOA 8 1 86400 20260519205754 20260505192754 33834 de. aZoiAJ+PaHUDVSHNXfV/R26ZK3GpFB7ek2Z46VnZdmPEDaTww+a7PkiQ98W83xohUunXYSvQCMeGYfUre5UT76eBKThdxW2a6ImX9/x/oEzQ9x/69Y/NSeTckOv9m3HCLBOug01op1koiHOIAVEvonOmXEHHqo1P4sR/fNbcVg4= ;{id = 33834}
  - deleted 5mo ago
    [deleted]
    - fweimer 5mo ago
      [dead]
- kaltsturm 5mo ago
  not all: https://www.heise.de/ https://www.heise.de/ works
  - warpspin 5mo ago
    Probably just a high TTL.
    - 0123456789ABCDE 5mo ago
      can confirm, at least another 54k seconds from where i sit
  - edb_123 5mo ago
    Doesn't work here, at least not anymore. Every single .de domain I have tried doesn't resolve.
kangalioo 5mo ago
So glad I found someone mention this. Amazon.de, SPIEGEL.de is down. Highly prominent sites unreachable. I wonder how long this will last and how big of a thing this ends up being once people talk about it :o Feels big to me
- moltar 5mo ago
  Both examples open for me
  - irundebian 5mo ago
    Some domains work, some not. I assume that working domains are cached.
- balou23 5mo ago
  amazon.de, spiegel.de are down for me, too. heise.de works, but that might've been cached somewhere on my side.
  - yk 5mo ago
    dig manages to dig out ips for heise.de and tagesschau.de but not spiegel.de amazon.de and google.de However, dig @8.8.8.8 has still amazon.de cached, unlike 1.1.1.1 so perhaps Google to the rescue? [Edit] After playing around with it, google seems to have at least some pages cached. After setting dns to 8.8.8.8 amazon.de and spiegel.de work again, my blog does not.
- theanonymousone 5mo ago
  idealo.de, ebay.de, and spiegel.de are down, but amazon.de opens for me.
deleted 5mo ago
[deleted]
- axiosgunnar 5mo ago
  [dead]
hmilch99 5mo ago
https://pastebin.com/2mQUB8xX https://pastebin.com/2mQUB8xX seems like someone's going to have a lot of fun tonight
deleted 5mo ago
[deleted]
krystofbe 5mo ago
Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer. Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover.
- qazwsxedchac 5mo ago
  So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.
  - walrus01 5mo ago
    It looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed.
    - account42 5mo ago
      Would be interesting to know how something could get missed. You'd think the system was set up so that new keys could not be published without being verified working in a staging system.
    - wildylion 5mo ago
      .ru [had this](https://habr.com/ru/news/790214/ https://habr.com/ru/news/790214/) in 2024. On the other hand, the russian government in itself had been doing a stellar job at breaking the internet. (I hope I'll live to see them all sentenced to life without parole)
nuil 5mo ago
Looks Like a DNSSEC error: https://dnssec-analyzer.verisignlabs.com/nic.de https://dnssec-analyzer.verisignlabs.com/nic.de
binghatch 5mo ago
Wow… it’s definitely not all .de TLDs, but a lot of prominent ones definitely.
- phit_ 5mo ago
  its gonna be all .de domains once caches dry out, anything that still works right now is bound to eventually fail until the underlying issue is resolved
  - fossdd 5mo ago
    Any .de domain with DNSSEC
    - meineerde 5mo ago
      Any .de domain is affected, regardless of the domain's dnssec deployment status, as long as you use a resolver which validates dnssec.
    - mrngm 5mo ago
      Unfortunately, even domains that did not have DNSSEC enabled earlier today are affected. We observed issues on a non-DNSSEC .de domain at 19:45Z and confirmed around 20:12Z it wasn't just us, but also more high profile domain names.
- eliaskg 5mo ago
  Amazon is completely down in Germany. Not only on amazon.de, even in the app.
sundiver 5mo ago
Yes, all .de domains down because of DNSSEC failure at Denic https://dnsviz.net/d/de/dnssec/ https://dnsviz.net/d/de/dnssec/
- taegee 5mo ago
  https://i.imgur.com/eAwdKEC.png https://i.imgur.com/eAwdKEC.png Edit: Alternative link: https://www.cyberciti.biz/media/new/cms/2017/04/dns.jpg https://www.cyberciti.biz/media/new/cms/2017/04/dns.jpg
  - notpushkin 5mo ago
    {"data":{"error":"Imgur is temporarily over capacity. Please try again later."},"success":false,"status":403} There is some strange irony to this, I suppose.
    - yjftsjthsd-h 5mo ago
      In my experience, that error is a lie and is what you get if they've IP blocked you. (Easy to hit on a VPN, in particular)
    - deleted 5mo ago
      [deleted]
    - itvision 5mo ago
      A protection against bad networks, including VPN. It's been like that for over two years now.
    - ricardo81 5mo ago
      I get "content not viewable in your region", from the UK. Not an ideal image sharing website nowadays.
      - 9dev 5mo ago
        Rather, not an ideal legislation nowadays…
pogii123 5mo ago
For me bmw.de works but www.bmw.de not
- benny_s 5mo ago
  bmw.de is down for me too
  - MikeNotThePope 5mo ago
    Both domains page load for me from Amsterdam. I wonder if there's communication disruption. Undersea cable severed?
    - pogii123 5mo ago
      $ nslookup bmw.de ~ Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: bmw.de Address: 160.46.226.165 $ nslookup www.bmw.de ~ ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 8.8.4.4 Address: 8.8.4.4#53 * server can't find www.bmw.de: SERVFAIL
    - dark-star 5mo ago
      You mean the big undersea cable between the Netherlands and Germany? ;-)
      - MikeNotThePope 5mo ago
        Lol, I meant between users across the sea who couldn't see and users in Europe who could.
  - dark-star 5mo ago
    both work for me from inside Germany
jamietanna 5mo ago
Was wondering why a few of my sites aren't CSSing, as they use https://classless.de https://classless.de
- kaltsturm 5mo ago
  cache
lpcvoid 5mo ago
[dead]
iknowstuff 5mo ago
Kurzgesagt predicted this, Germany is OVER
- irundebian 5mo ago
  Danke Merkel
  - mschuster91 5mo ago
    Not sure if serious or /s
    - Medicineguy 5mo ago
      Almost certainly /s. "Danke Merkel" ("Thanks Merkel") was once a sincere criticism from conservatives regarding her policies (esp. during 2015 refugee crisis), but it quickly evolved into a sarcastic, deadpan joke used to blame her for literally anything that goes wrong in daily Germany - even years after she left. Interesting phenomenon...
      - tommit 5mo ago
        It's our version of Thanks Obama.
      - nkydr0i0 5mo ago
        What about "Thanks Obama"[0] [0] https://en.wikipedia.org/wiki/Thanks,_Obama https://en.wikipedia.org/wiki/Thanks,_Obama
merb 5mo ago
Well at least it’s night time which means it’s hopefully resolved in the morning. Looks like it failed after a maintenance: https://www.namecheap.com/status-updates/planned-denic-de-registry-scheduled-maintenance-may-5-2026/ https://www.namecheap.com/status-updates/planned-denic-de-re... https://status.denic.de/ https://status.denic.de/
- gpvos 5mo ago
  If so, it still worked for several hours after the maintenance was completed.
1vuio0pswjnm7 5mo ago
.de TLD is online. DNS working fine DNSSEC not working If using an open resolver, i.e., a shared DNS cache, e.g., third party DNS service such as Google, Cloudflare, etc., then it might fail, or it might not. It depends on the third party DNS provider https://datatracker.ietf.org/meeting/118/materials/slides-118-maprg-unresolved-issues-characterizing-open-dns-resolver-misbehavior-for-dnssec-queries-00 https://datatracker.ietf.org/meeting/118/materials/slides-11...
- jeroenhd 5mo ago
  DNS worked fine. The responses that the root DNS servers were sending were wrong. It's the cryptographic version of that one time the same TLD told the world domains starting with certain letters didn't exist: https://www.theregister.com/2010/05/12/germany_top_level_domain_glitch/ https://www.theregister.com/2010/05/12/germany_top_level_dom...
- deleted 5mo ago
  [deleted]
- 1vuio0pswjnm7 5mo ago
  To me, a response from a "root DNS server", i.e., [a-m].root-servers.net, is not "wrong" if it contains the correct data that I'm requesting, e.g., domainnames and associated IP numbers I'm never requesting RRSIGs as I do not use that data. For me, it's just cruft that now comes in the response
blmaniac 5mo ago
[dead]
lxgr 5mo ago
Wow, I thought I was somehow unaffected but my resolver must just have cached the sites I'd tried.
kuerbel 5mo ago
I just spent the better half of an hour to debug unbound and the pihole because I thought it's a me problem... Good news though, if you add domain-insecure: "de" to your unbound config everything works fine
- victorbjorklund 5mo ago
  Same haha
- chromehearts 5mo ago
  SAMEEEEE !!!
- Bender 5mo ago
  I don't even enable DNSSEC in Unbound. There just isn't enough adoption yet for me to feel like I am missing out on something, yet. "Cloudflare Radar data shows 8.11% of domains are signed with DNSSEC, but only 0.47% of queries are validated end-to-end." [1] Zones I may care about: - Amazon.com: unsigned - My banks: unsigned - Hacker News: unsigned - Email that I do not host: unsigned - My power companies billing: unsigned - I found some! id.me and irs.gov are signed. [1] - https://technologychecker.io/blog/dnssec-adoption https://technologychecker.io/blog/dnssec-adoption
  - tptacek 5mo ago
    The Tranco list is an academic research project to generate a "top N zones" list. Here's the portion of the top 1000 that is signed: https://dnssecmenot.fly.dev/ https://dnssecmenot.fly.dev/
    - Bender 5mo ago
      That's cool, ty for that. The only one I put credentials into is Amazon it is unsigned. [1] There probably needs to be a DNSSECv2 .vbis that reduces risk somehow to get more adoption. [1] - https://dnssec-analyzer.verisignlabs.com/amazon.com https://dnssec-analyzer.verisignlabs.com/amazon.com
__michaelg 5mo ago
Finally establishing the concept of Feiertag on the internet. Come back tomorrow.
- 9753268996433 5mo ago
  [flagged]
  - sgbeal 5mo ago
    > will result in the police kicking in your door the next morning But not before 8am.
- throw1234567891 5mo ago
  Internetfreie Dienstage, 21st century variant of Autofreie Sonntage.
sunaookami 5mo ago
https://status.denic.de/ https://status.denic.de/ says "Partial Service Disruption" for DNS Nameservice now. EDIT: it says "Service Disruption" now
- MASNeo 5mo ago
  At least they have some humor left. Edit: Now even the humor is gone.
  - sunaookami 5mo ago
    Can only be topped when the status page is not reachable anymore :D EDIT: called it...
    - lschueller 5mo ago
      Or only accessible through a german dns server
- niklasrde 5mo ago
  It says "Server Not Found" now
- port3000 5mo ago
  Even when every site in the world’s 3rd biggest economy goes down it’s still just a ‘Partial’ service disruption :D
  - yorwba 5mo ago
    Not every site, just the ones using DNSSEC. Clearly, denic.de was online, for instance.
    - tom1337 5mo ago
      Sites without DNSSEC have also been affected. If you could reach any .de page, you had the DNS entries cached.
chromehearts 5mo ago
I was STRESSING tf out because I wasn't able to connect to my services & apps through my domains like at all .. they only work when using my phone data ? .. thank god it's not my fault this time
- Locke80 5mo ago
  But we're Germans, and we need someone to blame.
  - AndroTux 5mo ago
    I'm blaming chromehearts anyways
    - chromehearts 5mo ago
      I can live with that
  - lschueller 5mo ago
    Thank god for the german chain of blame: 1. The system 2. The neighbor 3. China
    - warpspin 5mo ago
      You definitely forgot Merkel and Habeck.
      - Cockbrand 5mo ago
        Danke Merkel!!1!11!!
victorbjorklund 5mo ago
I was just wondering what was up with our .de site.
jiggawatts 5mo ago
I work with a few people specialised in IT security, and some of them take their jobs too seriously and will "lock down" everything to the point that it becomes a very real risk that they lock out everyone including themselves. Fundamentally, security is a solution to an availability problem: The desire of the users is for a system to remain available despite external attack. Systems that become unavailable to everyone fail this requirement. A door with its keyhole welded shut is not "secure", it's broken.
- QuantumNomad_ 5mo ago
  Security is not just a solution to availability. It is also to keep sensitive data (PII, or business secrets, or passwords, or cryptographic private keys, and so on) away from the hands of bad actors. If I’m unable to use Amazon for 24 hours it doesn’t really matter. If a photo copy of my passport is leaked that’s worries and potential troubles for years.
- senkora 5mo ago
  Security = Confidentiality + Integrity + Availability or alternatively, Security = (exclude unauth'd reads) + (exclude unauth'd writes) + (include auth'd reads and auth'd writes) Gotta satisfy all parts in order to have security.
  - jiggawatts 5mo ago
    If you squint at it, you can convert all three to just availability. Confidentiality = available to us, but nobody else. Integrity = available to us in a pristine condition. It's a bit reductive, I'll admit, but it can be a useful exercise in the same way that everything in an economy can be reduce to units of either: "human time", "money" or "energy". Roughly speaking they're interchangeable. E.g.: What's the benefit to you if your data is so confidential that you can't read it either? This is a real problem with some health information systems, where I can't access my own health records! Ditto with many government bureaucracies that keep my records safe and secure from me.
    - 5mo ago
yosamino 5mo ago
The last time .de I remember .de had a major outage like this was 2010. I would cite some sources but... you know. That was a fun afternoon, though. I am very happy that it doesn't happen more often.
kaltsturm 5mo ago
https://dnsviz.net/d/spiegel.de/dnssec/ https://dnsviz.net/d/spiegel.de/dnssec/ yes indeed
dark-star 5mo ago
How come I have zero problems with any .de domain I tried accessing in the last half hour?
- pw6hv 5mo ago
  cache
  - dark-star 5mo ago
    unlikely, as I have also successfully tried domains that I never visited before (at least not in the last 12 months) and according to my PiHole log they were successfully retrieved from 1.1.1.1. and/or 8.8.8.8, which should use DNSSEC
- AndroTux 5mo ago
  maybe your upstream doesn't validate DNSSEC?
  - dark-star 5mo ago
    maybe? I'm using PiHole and 8.8.8.8/1.1.1.1 as upstream, and both options show "DNSSEC" next to their options in settings, so I assumed DNSSEC was enabled (unless I have to enable this somewhere else as well?)
    - warpspin 5mo ago
      That's weird cause 8.8.8.8/1.1.1.1 will already answer with SERVFAIL right now, unless the domain is still in the cache.
      - dark-star 5mo ago
        [dead]
kaltsturm 5mo ago
even their own status page is not reachable: https://status.denic.de/ https://status.denic.de/ As fallback they should use their X account: https://x.com/denic_de https://x.com/denic_de
- dgellow 5mo ago
  Seems to be up now? May 5, 2026 23:28 CEST May 5, 2026 21:28 UTC INVESTIGATING Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible. Based on current information, users and operators of .de domains may experience impairments in domain resolution. Further updates will be provided as soon as reliable findings on the cause and recovery are available. DENIC asks all affected parties for their understanding. For further enquiries, DENIC can be contacted via the usual channels.
  - elch 5mo ago
    All .de domains are down for me.
  - kaltsturm 5mo ago
    with firefox: KO with chrome: OK
- sunaookami 5mo ago
  They did now! https://x.com/denic_de/status/2051779175908774148 https://x.com/denic_de/status/2051779175908774148
  - niklasrde 5mo ago
    And in English: https://x.com/denic_de/status/2051779740080521646 https://x.com/denic_de/status/2051779740080521646
tarruda 5mo ago
Mailbox.org (also from Germany) seems to be experiencing issues too.
pocksuppet 5mo ago
I must be early. There's not a single tptacek DNSSEC rant in this thread yet.
- aberoham 5mo ago
  He’s busy with MathAcademy earning XP-SEC
- mike-cardwell 5mo ago
  Perhaps he's moribund
  - deleted 5mo ago
    [deleted]
- 0123456789ABCDE 5mo ago
  doesn't this event speak for itself though?
  - Avamander 5mo ago
    Kind-of. But there are worse things than outages when it's PKIs we're talking about. DNSSEC is also extremely opaque and unmonitored. Any compromise will not be noticed. Nor will anyone have any recourse against misbehaving roots. Fun fact, CloudFlare has used the same KSK for zones it serves more than a decade now.
    - daneel_w 5mo ago
      Which is fine. Not because KSK rollover is supposedly complicated, but if you can't manage to keep your private keys and PKI safe in the first place then key rotation is just a security circus trick. But if you do know how to keep them safe, then...
      - Avamander 5mo ago
        It is not fine. Keeping key material safe is not a boolean between "permanently safe" and "leaks immediately". Keeping key material secure for more than a decade while it's in active use is vastly more complex than keeping it secure for a month, until it rotates. For all we know, some ex-employee might be walking around with that KSK, theoretically being able to use it for god knows what for an another decade.
elevation 5mo ago
I've considered hard-coding some addresses into firmware as a fallback for a DNS outtage (which is more likely than not just misconfigured local DNS.) Events like this help justify this approach to the unconcerned.
- whalesalad 5mo ago
  The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.
  - cedilla 5mo ago
    denic is the single source of truth for zones under .de. The only problem with DNSSEC here is that it's complex.
    - akerl_ 5mo ago
      A complex thing where making a mistake makes your domains drop off the internet seems like a pretty big "only problem".
      - account42 5mo ago
        There is no more complexity other than what is inherent to the task.
        akerl_ 5mo ago
        It's not clear to me what you're trying to say.
  - jeroenhd 5mo ago
    The global and distributed system relies on the system actually returning valid responses. If the root servers are broken, whether it's a problem with RRSIG records or A records, the TLD is broken. If my domains' DNS servers start pointing at localhost, that doesn't mean DNS is a broken protocol.
siginator 5mo ago
how is that possible?
- deleted 5mo ago
  [deleted]
- aweiher 5mo ago
  Solar Flares
  - dnnddidiej 5mo ago
    Took more than cloud flares?
siginator 5mo ago
[dead]
dwedge 5mo ago
On a slightly unrelated note, I was setting nameservers for two .de domains a few weeks ago and thought my provider was being crazily strict because they kept getting rejected. Turns out you can't point to a nameserver until that nameserver has a zone for the domain, and you can't use nameservers from two providers unless those two providers are both in the NS records at both ends
- whalesalad 5mo ago
  Common paint point with DNSSEC. It’s brutal in the domain industry because when you buy a name with DNSSEC enabled it oftentimes can’t be setup to resolve due to these sorts of issues. Typically seller needs to deactivate first.
siva7 5mo ago
Crazy. I can't remember an incident like this ever happened before and it's still not fixed? .de is probably the most important unrestricted domain after .com from an economical perspective. Millions of businesses are "down".
- lschueller 5mo ago
  It's Germany, pessimistic time estimation + 1/3 and you are in a realistic time frame for the issue being resolved.
  - warpspin 5mo ago
    It's night. Somebody has to fill a form to approve night work first.
    - deleted 5mo ago
      [deleted]
    - snapetom 5mo ago
      Luckily it's not Sunday. Everyone would be out in the country hiking.
      - lschueller 5mo ago
        Or reading the latest prints about tax filings and how to conduct a compliance audit with pen and paper.
        thih9 5mo ago
        Or sweeping. https://en.wikipedia.org/wiki/Kehrwoche https://en.wikipedia.org/wiki/Kehrwoche
        layer8 5mo ago
        That's a sweeping generalization.
whalesalad 5mo ago
You can visually see this anomaly in many of CF Radar's charts: https://radar.cloudflare.com/dns/de?dateRange=1d https://radar.cloudflare.com/dns/de?dateRange=1d
g4cg54g54 5mo ago
funfact: enabling DNS sec NOW will fix your domain instantly if dnssec was disabled before -> no idea if that also "heals" anyone who had dnssec on before. -> no idea if maybe they need to roll back something and then rebreak the new dnssec i made a minute later lol...
- efreak 5mo ago
  I had dnssec enabled from 2018 until 1984 hosting messed it up in 2023 and I had to remove/disable keys from the registrar (oddly no information about this on their website or elsewhere, the issue apparently only exists in emails. Apparently they had an issue with BIND upgrade and it made up new keys...). I thought about reenabling it, but very few people other than me access my server, it has a static IP, and Firefox still doesn't validate dnssec anyways.
deleted 5mo ago
[deleted]
tom1337 5mo ago
I have never used DNSSEC and never really bothered implementing it, but do I understand it correctly that we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it which now breaks because the central organisation managing this certificate has an outage taking basically all domains with them?
- Medowar 5mo ago
  What you see here is decentralisation working. The issue is with the operator of the de TLD, and as such only that TLD is affected. DNS is not decentralised in such a way, that multiple organisations run the infrastructure of a TLD, those are always run by a single entity.(.com and .net are operated by Verisign) So what the issue is, that the operator has, does not change the impact.
  - AndroTux 5mo ago
    What if the root (.) certificate breaks?
    - pocksuppet 5mo ago
      Resolvers are free to cache each TLD's keys. There's a finite, well-known list of TLDs and their keys - you can download all the root zone data from IANA: https://www.iana.org/domains/root/files https://www.iana.org/domains/root/files (it's a few megabytes in uncompressed text form) The world might be a little bit better with more decentralization of the root zone.
- gucci-on-fleek 5mo ago
  > which now breaks because the central organisation managing this certificate has an outage The ".de" TLD is inherently managed by a single organization, and things wouldn't be much better if its nameservers went down. Some of the records would be cached by downstream resolvers, but not all of them, and not for very long. > we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it DNSSEC actually makes DNS more decentralized: without DNSSEC, the only way to guarantee a trustworthy response is to directly ask the authoritative nameservers. But with DNSSEC, you can query third-party caching resolvers and still be able to trust the response because only a legitimate answer will have a valid signature. Similarly, without DNSSEC, a domain owner needs to absolutely trust its authoritative nameservers, since they can trivially forge trusted results. But with DNSSEC, you don't need to trust your authoritative nameservers nearly as much [0], meaning that you can safely host some of them with third-parties. [0]: https://news.ycombinator.com/item?id=47409728 https://news.ycombinator.com/item?id=47409728
0x80h 5mo ago
Am I reading this correctly? All .de domains are down? Looking forward to reading the postmortem.
nfreising 5mo ago
They can join the (rather long) list of TLD DNSSEC outages https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
edo888 5mo ago
https://community.letsencrypt.org/t/issues-getting-certificates-for-de-zone/247023/3 https://community.letsencrypt.org/t/issues-getting-certifica...
kaltsturm 5mo ago
Denic should work out a desaster recovery test - like: https://blog.apnic.net/2022/02/14/disaster-recovery-with-dnssec/ https://blog.apnic.net/2022/02/14/disaster-recovery-with-dns...
sanbaideng 5mo ago
aiimageupscaler
Aldipower 5mo ago
Apparently the DENIC team was on a party this evening! Party hard, but not too hard. https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h
- FinnKuhn 5mo ago
  A real party killer if I have ever seen one.
  - SOLAR_FIELDS 5mo ago
    At least all of the appropriate people were in a room together when the outage happened
    - deleted 5mo ago
      [deleted]
    - SpaceNoodled 5mo ago
      Sounds like poor risk pooling. If that room crashed, we'd have nobody to fix this.
      - bflesch 5mo ago
        nation state actor picking right time to sabotage a tiny part of the key rotation process. on monday someone cut major fiber lines, on tuesday DENIC is failing. maybe someone is showing off?
        SOLAR_FIELDS 5mo ago
        Unironically yeah, we are at the level of weaponizable sophistication that this metaphorical dick waving you are suggesting is probably something that happens
- walrus01 5mo ago
  Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.
kaltsturm 5mo ago
Denic will be added to the "Major DNSSEC Outages and Validation Failures" list: https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
yowmamasita 5mo ago
The same day Kurzgesagt posted their video “Germany is over”. Huh. https://youtu.be/n-gYFcVx-8Y https://youtu.be/n-gYFcVx-8Y
- cwassert 5mo ago
  Kurzgesagt is once more highlighting the neoliberal solution: more growth. Surely a wealth tax is not worth mentioning.
  - deleted 5mo ago
    [deleted]
  - bwb 5mo ago
    I watched that video yesterday, their solution was to fix the pension funding problem :) They made the point that more immigration / growth wouldn't help fix the core problem if they don't fix that asap.
yassiniz 5mo ago
Shops open normally from 8am to 8pm in Germany. Today we decided to pilot opening hours for .de domains as well
- deleted 5mo ago
  [deleted]
bflesch 5mo ago
On Monday there was a huge outage affecting several cities quite close to Frankfurt because someone cut major fiber line; today DENIC is having a party and right when everyone is drunk this happens because some post-rotation task cannot be completed. There are too many coincidences happening.
neverrroot 5mo ago
[flagged]
kaltsturm 5mo ago
from my analysis DENIC resigned the .de zone today (May 5, 2026, ~17:49 UTC). The DNSSEC signature (RRSIG) for the NSEC3 record covering the hash range of nearly all .de TLD is cryptographically broken (malformed).
Animux 5mo ago
Seems to be fixed now.
Oarch 5mo ago
Germany has fallen.
kaltsturm 5mo ago
With chrome it works again
edb_123 5mo ago
Things seem to be on their way up now, and https://status.denic.de/ https://status.denic.de/ is working again, at least from here. DENIC's status page currently says "Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible.
jiveturkey 5mo ago
It’s not DNS There’s no way it’s DNS It was DNSSEC
taf2 5mo ago
ok i picked a bad day to move from one register to another... i just spent the last hour frantically trying to figure out why the new register screwed us or the old register was screwing us...
amelius 5mo ago
Maybe related to this? Crazy idea, but nothing surprises me anymore. https://edition.cnn.com/2026/05/01/politics/us-troop-withdrawal-germany-trump-merz https://edition.cnn.com/2026/05/01/politics/us-troop-withdra...
tom1337 5mo ago
Cloudflare has now disabled DNSSEC validation on their 1.1.1.1 resolver: https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz
- cluckindan 5mo ago
  If it turns out the DNSSEC issue was caused by threat actors, this downstream effect could very well have been the reason to do it.
  - amluto 5mo ago
    It is indeed a bit sad that Cloudflare had to turn off DNSSEC completely. But I completely understand that they don't have a production-ready, tested path to override DNSSEC validation for only some domains.
    - vendemiat 5mo ago
      Sorry! status message was not clear. DNSSEC validation is temporarily disabled only for .de domains.
      - tptacek 5mo ago
        That's not much better!
    - fastest963 5mo ago
      [flagged]
      - tptacek 5mo ago
        It didn't originally say that. They added the clarification just a few minutes ago. The guidelines ask you not to ask people these kinds of questions, for what it's worth.
      - jonah-archive 5mo ago
Zopieux 5mo ago
That postmortem should be a fun read, can't wait.
- retired 5mo ago
  We shall transmit the postmortem to you via fax within 25 business days, ja.
- Culonavirus 5mo ago
  Ok children, sit down and listen, uncle Culonavirus will tell you a story: "It all began with the decommissioning of the last nuclear power plant, ..."
- alper 5mo ago
  Given how amateurish German IT operations is, there is no guarantee whatsoever there will be a post-mortem nor whether it then will make it out under 3-6 months with all the necessary approvals.
  - Tepix 5mo ago
    Bla bla, always easy to rant... https://blog.denic.de/denic-informiert-uber-die-behebung-der-dnssec-storung-fur-de-domains/ https://blog.denic.de/denic-informiert-uber-die-behebung-der... "Die Störung ist inzwischen behoben und alle Systeme laufen wieder stabil. Die genaue Ursache wird derzeit noch analysiert. Sobald belastbare Erkenntnisse vorliegen, wird DENIC diese transparent zur Verfügung stellen." translation: ‘The disruption has now been resolved and all systems are running smoothly again. The exact cause is currently being investigated. As soon as reliable findings are available, DENIC will make them publicly available.’
    - account42 5mo ago
      Also always easy to announce "Sobald belastbare Erkenntnisse vorliegen, wird DENIC diese transparent zur Verfügung stellen." and then remain silent until the media forgets about the incident and never actually publish anything.
SEJeff 5mo ago
Just gonna leave this absolute gem from Thomas Ptacek on DNSSEC here: https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
- betaby 5mo ago
  Aged like a milk.
  - tptacek 5mo ago
    Oh, yeah, I'm sure feeling chastened right now. You got me.
    - SAI_Peregrinus 5mo ago
      Parmigianino-Reggiano is aged milk, so I'm not sure what people have against aged milk. Aged milk can be great
    - sgc 5mo ago
      My poor fellow. You wrote about how something is a bad tool for a long list of serious reasons. Then it failed spectacularly because everybody decided to depend on it anyway - exactly what you were cautioning against. But somehow you have to respond to people who think you are the one who got it wrong! As a third party the whole affair gave me a good chuckle at least ;)
      - tptacek 5mo ago
        Germany appears to depend on it. Virtually none of North America does. I'm pretty satisfied with how this whole thing shook out!
        cyberax 5mo ago
        You're wrong. Both .com and .net are signed (`dig RRSIG com.`), and if they screw up, then all the com/net zones will become inaccessible.
deleted 5mo ago
[deleted]
basilikum 5mo ago
This is the kind of system failure that we need really good and well tested disaster recovery plans for. While not necessary this time, DENIC and any critical infrastructure provider should be able to rebuild their entire infrastructure from scratch in a tolerable amount of time (Rather days than hours in the case of a full rebuild). Importantly the disaster recovery plan has to work without reliance on either the system that is failing, but also on adjacent systems that might have hidden dependencies on the failing system. I'm really not too close to Denic and know nothing about their internals, but just close enough to have experienced the stress of someone working for DENIC second hand during the outage. From the very limited information I happened to gather DENIC had some trouble in addressing the issue because, surprise, infrastructure that they need to do so runs on de domains. [1] I'm convinced there are all kinds of extended cyclic decencies between different centralization points in the net. If some important backbone of the internet is down for an extended time, this will absolutely cause cascading failures. And thesw central points of failure are only getting worse. I love Let's Encrypt, but if something causes them to hard fail things will go really bad once certificates start to expire. We need concrete plans to cold start extended parts of the internet. If things go really bad once and communication lines start to fail, we're in for a bad time. Maybe governments have redundant, ultra resistant, low tech communication lines, war rooms and a list of important people in the industry who they can find and put in these war rooms so they can coordinate the rebuild of infrastructure. But I doubt it. [^1] I don't know if there is some kind of disaster plan in the drawer at DENIC that would address this. I don't mean to allege anything against DENIC specifically, but broadly speaking about companies and infrastructure providers, I would not be surprised if there was absolutely no plan on what to do if things really go down and how to cold start cyclic dependencies or where they even are.
- toast0 5mo ago
  > This is the kind of system failure that we need really good and well tested disaster recovery plans for. All the cool kids offer their services over multiple TLDs and have their name servers of record in multiple TLDs, too. It's not quite best practices for recursive DNS to regularly fetch the complete root zone to cache it, but it's not unreasonable to do so.
NooneAtAll3 5mo ago
quad9 seems to be having problems with DNSSEC as well
aboardRat4 5mo ago
https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html
evan0721 5mo ago
[dead]
0xbadcafebee 5mo ago
I can't wait for the .com TLD outage. Ya'll thought Cloudflare down was bad? Lol
baby 5mo ago
Should I do my usual rent about how the web PKI refuses to move to a consensus protocol
jdthedisciple 5mo ago
Seems up again. How briefly did the outage last?
alper 5mo ago
I'd expect political escalation for something like this but given that this is Germany, who knows.
farfatched 5mo ago
DNSSEC operations feels like one of those problems that should be tackled with formal methods, like how some subway controllers are. But I expect it's treated like "very serious and scary ops", which isn't wrong, but isn't enough.
egberts1 5mo ago
Resolved ... after recovering from a mass German DNSSEC drinking party? Ok.
pierroons 5mo ago
[dead]
blmaniac 5mo ago
It seems like DENIC has a problem again. My domain elektro-menne.de is down. I've been debugging this issue for more than an hour, and it seems like the domain exists in WHOIS and Cloudflare responds correctly when queried directly, but DENIC authoritative nameservers return NXDOMAIN. For example: dig @a.nic.de elektro-menne.de A returns NXDOMAIN directly from DENIC. Zonemaster also reports: "elektro-menne.de does not exist as a DNS zone." I also found multiple other .de domains currently affected by the exact same issue. Looks like the domains are present in WHOIS, but missing from the live .de DNS zone delegation.

747 points by warpspin 5mo ago

adamas 5mo ago

I wasn't even aware that was possible..?

warpspin 5mo ago

Whole .de TLD seems to go offline right now due to dnssec or missing nic.de nameservers?

fweimer 5mo ago
This works: $ unbound-host -t A www.denic.de www.denic.de has address 81.91.170.12 This does not: $ unbound-host -D -t A www.denic.de www.denic.de has address 81.91.170.12 validation failure <www.denic.de. A IN>: signature crypto failed from 194.246.96.1 for DS denic.de. while building chain of trust So it does seem DNSSEC-related. EDIT My explanation was wrong, this is not how keytags work. The published keytag data is consistent: de. 3600 IN DNSKEY 256 3 8 AwEAAfRLmzuIXVf7x5A0+U7hke0dS+GEJG0EdPhnOthCCLhy0t0WqLyoXJOhnfsTJ8vQX5fd9qOJc9gyr3SWJZkXAhPm3yPSC7FWWHF70WZTKKM9CekmKdqwMwq6ZCjMSUcecCuSF4Sbt1MRszV7rFmfGVklA1l5UzNbqwD+Dr5vfcLn ;{id = 33834 (zsk), size = 1024b} de. 3600 IN DNSKEY 257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b} de. 3600 IN DNSKEY 256 3 8 AwEAAbTe1PJi8EgIudNGb+KRTxBL2aCu5rXkZ+aIe/TC88pwRdrXYeXODp1ihZWFop5CrbWRBLrk/YUPBE8aBc6oJP+58dSkdMLYkjSkmvdvYx+zXnRLWlF2bapxvZxshATJDfGjGbCiWxKEOoyRx3UhICtHC+cUSddsEvzfacUcBb6n ;{id = 32911 (zsk), size = 1024b} de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519030655 20260505013655 26755 de. ke56T5GZt/X6zMBAF+ouyCTnAd7RY7MsnDcfa9jyyOwSouRXhvzim/V13JDTMBAnpAHxWQXoruXrAZ6A6re5N+8Pp2utVkAEKTWs0r4UOLNKoZ2+zMwNplKjNNnY5PJIbHfa5myyziLiIsi//qDIgQEACFk+pZcHXrRdqRoXPCL3UtfaXjk3+duDQdlPnYsJys5UshjVpkALSMChW7J0anzr0sG+f9ytstBneymMwFYOUC3NqbejbLPZsXGPZBQKPAoVJuV5q3znopbcqrDFfjI7bmX3QPYNvOaiT1ElBfi2piJVpDzMaMAmm2jCmvrf5VeTOBccMroh8sBtDPsaEg== ;{id = 26755} The signature on the SOA record still does not verify: de. 86400 IN SOA f.nic.de. dns-operations.denic.de. 1778014672 7200 7200 3600000 7200 de. 86400 IN RRSIG SOA 8 1 86400 20260519205754 20260505192754 33834 de. aZoiAJ+PaHUDVSHNXfV/R26ZK3GpFB7ek2Z46VnZdmPEDaTww+a7PkiQ98W83xohUunXYSvQCMeGYfUre5UT76eBKThdxW2a6ImX9/x/oEzQ9x/69Y/NSeTckOv9m3HCLBOug01op1koiHOIAVEvonOmXEHHqo1P4sR/fNbcVg4= ;{id = 33834}
- deleted 5mo ago
  [deleted]
  - fweimer 5mo ago
    [dead]
kaltsturm 5mo ago
not all: https://www.heise.de/ https://www.heise.de/ works
- warpspin 5mo ago
  Probably just a high TTL.
  - 0123456789ABCDE 5mo ago
    can confirm, at least another 54k seconds from where i sit
- edb_123 5mo ago
  Doesn't work here, at least not anymore. Every single .de domain I have tried doesn't resolve.

kangalioo 5mo ago

So glad I found someone mention this. Amazon.de, SPIEGEL.de is down. Highly prominent sites unreachable. I wonder how long this will last and how big of a thing this ends up being once people talk about it :o Feels big to me

moltar 5mo ago
Both examples open for me
- irundebian 5mo ago
  Some domains work, some not. I assume that working domains are cached.
balou23 5mo ago
amazon.de, spiegel.de are down for me, too. heise.de works, but that might've been cached somewhere on my side.
- yk 5mo ago
  dig manages to dig out ips for heise.de and tagesschau.de but not spiegel.de amazon.de and google.de However, dig @8.8.8.8 has still amazon.de cached, unlike 1.1.1.1 so perhaps Google to the rescue? [Edit] After playing around with it, google seems to have at least some pages cached. After setting dns to 8.8.8.8 amazon.de and spiegel.de work again, my blog does not.
theanonymousone 5mo ago
idealo.de, ebay.de, and spiegel.de are down, but amazon.de opens for me.

deleted 5mo ago

[deleted]

axiosgunnar 5mo ago
[dead]

hmilch99 5mo ago

https://pastebin.com/2mQUB8xX https://pastebin.com/2mQUB8xX seems like someone's going to have a lot of fun tonight

deleted 5mo ago

[deleted]

krystofbe 5mo ago

Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer. Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover.

qazwsxedchac 5mo ago
So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.
- walrus01 5mo ago
  It looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed.
  - account42 5mo ago
    Would be interesting to know how something could get missed. You'd think the system was set up so that new keys could not be published without being verified working in a staging system.
  - wildylion 5mo ago
    .ru [had this](https://habr.com/ru/news/790214/ https://habr.com/ru/news/790214/) in 2024. On the other hand, the russian government in itself had been doing a stellar job at breaking the internet. (I hope I'll live to see them all sentenced to life without parole)

nuil 5mo ago

Looks Like a DNSSEC error: https://dnssec-analyzer.verisignlabs.com/nic.de https://dnssec-analyzer.verisignlabs.com/nic.de

binghatch 5mo ago

Wow… it’s definitely not all .de TLDs, but a lot of prominent ones definitely.

phit_ 5mo ago
its gonna be all .de domains once caches dry out, anything that still works right now is bound to eventually fail until the underlying issue is resolved
- fossdd 5mo ago
  Any .de domain with DNSSEC
  - meineerde 5mo ago
    Any .de domain is affected, regardless of the domain's dnssec deployment status, as long as you use a resolver which validates dnssec.
  - mrngm 5mo ago
    Unfortunately, even domains that did not have DNSSEC enabled earlier today are affected. We observed issues on a non-DNSSEC .de domain at 19:45Z and confirmed around 20:12Z it wasn't just us, but also more high profile domain names.
eliaskg 5mo ago
Amazon is completely down in Germany. Not only on amazon.de, even in the app.

sundiver 5mo ago

Yes, all .de domains down because of DNSSEC failure at Denic https://dnsviz.net/d/de/dnssec/ https://dnsviz.net/d/de/dnssec/

taegee 5mo ago
https://i.imgur.com/eAwdKEC.png https://i.imgur.com/eAwdKEC.png Edit: Alternative link: https://www.cyberciti.biz/media/new/cms/2017/04/dns.jpg https://www.cyberciti.biz/media/new/cms/2017/04/dns.jpg
- notpushkin 5mo ago
  {"data":{"error":"Imgur is temporarily over capacity. Please try again later."},"success":false,"status":403} There is some strange irony to this, I suppose.
  - yjftsjthsd-h 5mo ago
    In my experience, that error is a lie and is what you get if they've IP blocked you. (Easy to hit on a VPN, in particular)
  - deleted 5mo ago
    [deleted]
  - itvision 5mo ago
    A protection against bad networks, including VPN. It's been like that for over two years now.
  - ricardo81 5mo ago
    I get "content not viewable in your region", from the UK. Not an ideal image sharing website nowadays.
    - 9dev 5mo ago
      Rather, not an ideal legislation nowadays…

pogii123 5mo ago

For me bmw.de works but www.bmw.de not

benny_s 5mo ago
bmw.de is down for me too
- MikeNotThePope 5mo ago
  Both domains page load for me from Amsterdam. I wonder if there's communication disruption. Undersea cable severed?
  - pogii123 5mo ago
    $ nslookup bmw.de ~ Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: bmw.de Address: 160.46.226.165 $ nslookup www.bmw.de ~ ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 8.8.4.4 Address: 8.8.4.4#53 * server can't find www.bmw.de: SERVFAIL
  - dark-star 5mo ago
    You mean the big undersea cable between the Netherlands and Germany? ;-)
    - MikeNotThePope 5mo ago
      Lol, I meant between users across the sea who couldn't see and users in Europe who could.
- dark-star 5mo ago
  both work for me from inside Germany

jamietanna 5mo ago

Was wondering why a few of my sites aren't CSSing, as they use https://classless.de https://classless.de

kaltsturm 5mo ago
cache

lpcvoid 5mo ago

[dead]

iknowstuff 5mo ago

Kurzgesagt predicted this, Germany is OVER

irundebian 5mo ago
Danke Merkel
- mschuster91 5mo ago
  Not sure if serious or /s
  - Medicineguy 5mo ago
    Almost certainly /s. "Danke Merkel" ("Thanks Merkel") was once a sincere criticism from conservatives regarding her policies (esp. during 2015 refugee crisis), but it quickly evolved into a sarcastic, deadpan joke used to blame her for literally anything that goes wrong in daily Germany - even years after she left. Interesting phenomenon...
    - tommit 5mo ago
      It's our version of Thanks Obama.
    - nkydr0i0 5mo ago
      What about "Thanks Obama"[0] [0] https://en.wikipedia.org/wiki/Thanks,_Obama https://en.wikipedia.org/wiki/Thanks,_Obama

merb 5mo ago

Well at least it’s night time which means it’s hopefully resolved in the morning. Looks like it failed after a maintenance: https://www.namecheap.com/status-updates/planned-denic-de-registry-scheduled-maintenance-may-5-2026/ https://www.namecheap.com/status-updates/planned-denic-de-re... https://status.denic.de/ https://status.denic.de/

gpvos 5mo ago
If so, it still worked for several hours after the maintenance was completed.

1vuio0pswjnm7 5mo ago

.de TLD is online. DNS working fine DNSSEC not working If using an open resolver, i.e., a shared DNS cache, e.g., third party DNS service such as Google, Cloudflare, etc., then it might fail, or it might not. It depends on the third party DNS provider https://datatracker.ietf.org/meeting/118/materials/slides-118-maprg-unresolved-issues-characterizing-open-dns-resolver-misbehavior-for-dnssec-queries-00 https://datatracker.ietf.org/meeting/118/materials/slides-11...

jeroenhd 5mo ago
DNS worked fine. The responses that the root DNS servers were sending were wrong. It's the cryptographic version of that one time the same TLD told the world domains starting with certain letters didn't exist: https://www.theregister.com/2010/05/12/germany_top_level_domain_glitch/ https://www.theregister.com/2010/05/12/germany_top_level_dom...
deleted 5mo ago
[deleted]
1vuio0pswjnm7 5mo ago
To me, a response from a "root DNS server", i.e., [a-m].root-servers.net, is not "wrong" if it contains the correct data that I'm requesting, e.g., domainnames and associated IP numbers I'm never requesting RRSIGs as I do not use that data. For me, it's just cruft that now comes in the response

blmaniac 5mo ago

[dead]

lxgr 5mo ago

Wow, I thought I was somehow unaffected but my resolver must just have cached the sites I'd tried.

kuerbel 5mo ago

I just spent the better half of an hour to debug unbound and the pihole because I thought it's a me problem... Good news though, if you add domain-insecure: "de" to your unbound config everything works fine

victorbjorklund 5mo ago
Same haha
chromehearts 5mo ago
SAMEEEEE !!!
Bender 5mo ago
I don't even enable DNSSEC in Unbound. There just isn't enough adoption yet for me to feel like I am missing out on something, yet. "Cloudflare Radar data shows 8.11% of domains are signed with DNSSEC, but only 0.47% of queries are validated end-to-end." [1] Zones I may care about: - Amazon.com: unsigned - My banks: unsigned - Hacker News: unsigned - Email that I do not host: unsigned - My power companies billing: unsigned - I found some! id.me and irs.gov are signed. [1] - https://technologychecker.io/blog/dnssec-adoption https://technologychecker.io/blog/dnssec-adoption
- tptacek 5mo ago
  The Tranco list is an academic research project to generate a "top N zones" list. Here's the portion of the top 1000 that is signed: https://dnssecmenot.fly.dev/ https://dnssecmenot.fly.dev/
  - Bender 5mo ago
    That's cool, ty for that. The only one I put credentials into is Amazon it is unsigned. [1] There probably needs to be a DNSSECv2 .vbis that reduces risk somehow to get more adoption. [1] - https://dnssec-analyzer.verisignlabs.com/amazon.com https://dnssec-analyzer.verisignlabs.com/amazon.com

__michaelg 5mo ago

Finally establishing the concept of Feiertag on the internet. Come back tomorrow.

9753268996433 5mo ago
[flagged]
- sgbeal 5mo ago
  > will result in the police kicking in your door the next morning But not before 8am.
throw1234567891 5mo ago
Internetfreie Dienstage, 21st century variant of Autofreie Sonntage.

sunaookami 5mo ago

https://status.denic.de/ https://status.denic.de/ says "Partial Service Disruption" for DNS Nameservice now. EDIT: it says "Service Disruption" now

MASNeo 5mo ago
At least they have some humor left. Edit: Now even the humor is gone.
- sunaookami 5mo ago
  Can only be topped when the status page is not reachable anymore :D EDIT: called it...
  - lschueller 5mo ago
    Or only accessible through a german dns server
niklasrde 5mo ago
It says "Server Not Found" now
port3000 5mo ago
Even when every site in the world’s 3rd biggest economy goes down it’s still just a ‘Partial’ service disruption :D
- yorwba 5mo ago
  Not every site, just the ones using DNSSEC. Clearly, denic.de was online, for instance.
  - tom1337 5mo ago
    Sites without DNSSEC have also been affected. If you could reach any .de page, you had the DNS entries cached.

chromehearts 5mo ago

I was STRESSING tf out because I wasn't able to connect to my services & apps through my domains like at all .. they only work when using my phone data ? .. thank god it's not my fault this time

Locke80 5mo ago
But we're Germans, and we need someone to blame.
- AndroTux 5mo ago
  I'm blaming chromehearts anyways
  - chromehearts 5mo ago
    I can live with that
- lschueller 5mo ago
  Thank god for the german chain of blame: 1. The system 2. The neighbor 3. China
  - warpspin 5mo ago
    You definitely forgot Merkel and Habeck.
    - Cockbrand 5mo ago
      Danke Merkel!!1!11!!

victorbjorklund 5mo ago

I was just wondering what was up with our .de site.

jiggawatts 5mo ago

I work with a few people specialised in IT security, and some of them take their jobs too seriously and will "lock down" everything to the point that it becomes a very real risk that they lock out everyone including themselves. Fundamentally, security is a solution to an availability problem: The desire of the users is for a system to remain available despite external attack. Systems that become unavailable to everyone fail this requirement. A door with its keyhole welded shut is not "secure", it's broken.

QuantumNomad_ 5mo ago
Security is not just a solution to availability. It is also to keep sensitive data (PII, or business secrets, or passwords, or cryptographic private keys, and so on) away from the hands of bad actors. If I’m unable to use Amazon for 24 hours it doesn’t really matter. If a photo copy of my passport is leaked that’s worries and potential troubles for years.
senkora 5mo ago
Security = Confidentiality + Integrity + Availability or alternatively, Security = (exclude unauth'd reads) + (exclude unauth'd writes) + (include auth'd reads and auth'd writes) Gotta satisfy all parts in order to have security.
- jiggawatts 5mo ago
  If you squint at it, you can convert all three to just availability. Confidentiality = available to us, but nobody else. Integrity = available to us in a pristine condition. It's a bit reductive, I'll admit, but it can be a useful exercise in the same way that everything in an economy can be reduce to units of either: "human time", "money" or "energy". Roughly speaking they're interchangeable. E.g.: What's the benefit to you if your data is so confidential that you can't read it either? This is a real problem with some health information systems, where I can't access my own health records! Ditto with many government bureaucracies that keep my records safe and secure from me.
  - 5mo ago

yosamino 5mo ago

The last time .de I remember .de had a major outage like this was 2010. I would cite some sources but... you know. That was a fun afternoon, though. I am very happy that it doesn't happen more often.

kaltsturm 5mo ago

https://dnsviz.net/d/spiegel.de/dnssec/ https://dnsviz.net/d/spiegel.de/dnssec/ yes indeed

dark-star 5mo ago

How come I have zero problems with any .de domain I tried accessing in the last half hour?

pw6hv 5mo ago
cache
- dark-star 5mo ago
  unlikely, as I have also successfully tried domains that I never visited before (at least not in the last 12 months) and according to my PiHole log they were successfully retrieved from 1.1.1.1. and/or 8.8.8.8, which should use DNSSEC
AndroTux 5mo ago
maybe your upstream doesn't validate DNSSEC?
- dark-star 5mo ago
  maybe? I'm using PiHole and 8.8.8.8/1.1.1.1 as upstream, and both options show "DNSSEC" next to their options in settings, so I assumed DNSSEC was enabled (unless I have to enable this somewhere else as well?)
  - warpspin 5mo ago
    That's weird cause 8.8.8.8/1.1.1.1 will already answer with SERVFAIL right now, unless the domain is still in the cache.
    - dark-star 5mo ago
      [dead]

kaltsturm 5mo ago

even their own status page is not reachable: https://status.denic.de/ https://status.denic.de/ As fallback they should use their X account: https://x.com/denic_de https://x.com/denic_de

dgellow 5mo ago
Seems to be up now? May 5, 2026 23:28 CEST May 5, 2026 21:28 UTC INVESTIGATING Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible. Based on current information, users and operators of .de domains may experience impairments in domain resolution. Further updates will be provided as soon as reliable findings on the cause and recovery are available. DENIC asks all affected parties for their understanding. For further enquiries, DENIC can be contacted via the usual channels.
- elch 5mo ago
  All .de domains are down for me.
- kaltsturm 5mo ago
  with firefox: KO with chrome: OK
sunaookami 5mo ago
They did now! https://x.com/denic_de/status/2051779175908774148 https://x.com/denic_de/status/2051779175908774148
- niklasrde 5mo ago
  And in English: https://x.com/denic_de/status/2051779740080521646 https://x.com/denic_de/status/2051779740080521646

tarruda 5mo ago

Mailbox.org (also from Germany) seems to be experiencing issues too.

pocksuppet 5mo ago

I must be early. There's not a single tptacek DNSSEC rant in this thread yet.

aberoham 5mo ago
He’s busy with MathAcademy earning XP-SEC
mike-cardwell 5mo ago
Perhaps he's moribund
- deleted 5mo ago
  [deleted]
0123456789ABCDE 5mo ago
doesn't this event speak for itself though?
- Avamander 5mo ago
  Kind-of. But there are worse things than outages when it's PKIs we're talking about. DNSSEC is also extremely opaque and unmonitored. Any compromise will not be noticed. Nor will anyone have any recourse against misbehaving roots. Fun fact, CloudFlare has used the same KSK for zones it serves more than a decade now.
  - daneel_w 5mo ago
    Which is fine. Not because KSK rollover is supposedly complicated, but if you can't manage to keep your private keys and PKI safe in the first place then key rotation is just a security circus trick. But if you do know how to keep them safe, then...
    - Avamander 5mo ago
      It is not fine. Keeping key material safe is not a boolean between "permanently safe" and "leaks immediately". Keeping key material secure for more than a decade while it's in active use is vastly more complex than keeping it secure for a month, until it rotates. For all we know, some ex-employee might be walking around with that KSK, theoretically being able to use it for god knows what for an another decade.

elevation 5mo ago

I've considered hard-coding some addresses into firmware as a fallback for a DNS outtage (which is more likely than not just misconfigured local DNS.) Events like this help justify this approach to the unconcerned.

whalesalad 5mo ago
The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.
- cedilla 5mo ago
  denic is the single source of truth for zones under .de. The only problem with DNSSEC here is that it's complex.
  - akerl_ 5mo ago
    A complex thing where making a mistake makes your domains drop off the internet seems like a pretty big "only problem".
    - account42 5mo ago
      There is no more complexity other than what is inherent to the task.
      - akerl_ 5mo ago
        It's not clear to me what you're trying to say.
- jeroenhd 5mo ago
  The global and distributed system relies on the system actually returning valid responses. If the root servers are broken, whether it's a problem with RRSIG records or A records, the TLD is broken. If my domains' DNS servers start pointing at localhost, that doesn't mean DNS is a broken protocol.

siginator 5mo ago

how is that possible?

deleted 5mo ago
[deleted]
aweiher 5mo ago
Solar Flares
- dnnddidiej 5mo ago
  Took more than cloud flares?

siginator 5mo ago

[dead]

dwedge 5mo ago

On a slightly unrelated note, I was setting nameservers for two .de domains a few weeks ago and thought my provider was being crazily strict because they kept getting rejected. Turns out you can't point to a nameserver until that nameserver has a zone for the domain, and you can't use nameservers from two providers unless those two providers are both in the NS records at both ends

whalesalad 5mo ago
Common paint point with DNSSEC. It’s brutal in the domain industry because when you buy a name with DNSSEC enabled it oftentimes can’t be setup to resolve due to these sorts of issues. Typically seller needs to deactivate first.

siva7 5mo ago

Crazy. I can't remember an incident like this ever happened before and it's still not fixed? .de is probably the most important unrestricted domain after .com from an economical perspective. Millions of businesses are "down".

lschueller 5mo ago
It's Germany, pessimistic time estimation + 1/3 and you are in a realistic time frame for the issue being resolved.
- warpspin 5mo ago
  It's night. Somebody has to fill a form to approve night work first.
  - deleted 5mo ago
    [deleted]
  - snapetom 5mo ago
    Luckily it's not Sunday. Everyone would be out in the country hiking.
    - lschueller 5mo ago
      Or reading the latest prints about tax filings and how to conduct a compliance audit with pen and paper.
      - thih9 5mo ago
        Or sweeping. https://en.wikipedia.org/wiki/Kehrwoche https://en.wikipedia.org/wiki/Kehrwoche
        layer8 5mo ago
        That's a sweeping generalization.

whalesalad 5mo ago

You can visually see this anomaly in many of CF Radar's charts: https://radar.cloudflare.com/dns/de?dateRange=1d https://radar.cloudflare.com/dns/de?dateRange=1d

g4cg54g54 5mo ago

funfact: enabling DNS sec NOW will fix your domain instantly if dnssec was disabled before -> no idea if that also "heals" anyone who had dnssec on before. -> no idea if maybe they need to roll back something and then rebreak the new dnssec i made a minute later lol...

efreak 5mo ago
I had dnssec enabled from 2018 until 1984 hosting messed it up in 2023 and I had to remove/disable keys from the registrar (oddly no information about this on their website or elsewhere, the issue apparently only exists in emails. Apparently they had an issue with BIND upgrade and it made up new keys...). I thought about reenabling it, but very few people other than me access my server, it has a static IP, and Firefox still doesn't validate dnssec anyways.

deleted 5mo ago

[deleted]

tom1337 5mo ago

I have never used DNSSEC and never really bothered implementing it, but do I understand it correctly that we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it which now breaks because the central organisation managing this certificate has an outage taking basically all domains with them?

Medowar 5mo ago
What you see here is decentralisation working. The issue is with the operator of the de TLD, and as such only that TLD is affected. DNS is not decentralised in such a way, that multiple organisations run the infrastructure of a TLD, those are always run by a single entity.(.com and .net are operated by Verisign) So what the issue is, that the operator has, does not change the impact.
- AndroTux 5mo ago
  What if the root (.) certificate breaks?
  - pocksuppet 5mo ago
    Resolvers are free to cache each TLD's keys. There's a finite, well-known list of TLDs and their keys - you can download all the root zone data from IANA: https://www.iana.org/domains/root/files https://www.iana.org/domains/root/files (it's a few megabytes in uncompressed text form) The world might be a little bit better with more decentralization of the root zone.
gucci-on-fleek 5mo ago
> which now breaks because the central organisation managing this certificate has an outage The ".de" TLD is inherently managed by a single organization, and things wouldn't be much better if its nameservers went down. Some of the records would be cached by downstream resolvers, but not all of them, and not for very long. > we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it DNSSEC actually makes DNS more decentralized: without DNSSEC, the only way to guarantee a trustworthy response is to directly ask the authoritative nameservers. But with DNSSEC, you can query third-party caching resolvers and still be able to trust the response because only a legitimate answer will have a valid signature. Similarly, without DNSSEC, a domain owner needs to absolutely trust its authoritative nameservers, since they can trivially forge trusted results. But with DNSSEC, you don't need to trust your authoritative nameservers nearly as much [0], meaning that you can safely host some of them with third-parties. [0]: https://news.ycombinator.com/item?id=47409728 https://news.ycombinator.com/item?id=47409728

0x80h 5mo ago

Am I reading this correctly? All .de domains are down? Looking forward to reading the postmortem.

nfreising 5mo ago

They can join the (rather long) list of TLD DNSSEC outages https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html

edo888 5mo ago

https://community.letsencrypt.org/t/issues-getting-certificates-for-de-zone/247023/3 https://community.letsencrypt.org/t/issues-getting-certifica...

kaltsturm 5mo ago

Denic should work out a desaster recovery test - like: https://blog.apnic.net/2022/02/14/disaster-recovery-with-dnssec/ https://blog.apnic.net/2022/02/14/disaster-recovery-with-dns...

sanbaideng 5mo ago

aiimageupscaler

Aldipower 5mo ago

Apparently the DENIC team was on a party this evening! Party hard, but not too hard. https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h

FinnKuhn 5mo ago
A real party killer if I have ever seen one.
- SOLAR_FIELDS 5mo ago
  At least all of the appropriate people were in a room together when the outage happened
  - deleted 5mo ago
    [deleted]
  - SpaceNoodled 5mo ago
    Sounds like poor risk pooling. If that room crashed, we'd have nobody to fix this.
    - bflesch 5mo ago
      nation state actor picking right time to sabotage a tiny part of the key rotation process. on monday someone cut major fiber lines, on tuesday DENIC is failing. maybe someone is showing off?
      - SOLAR_FIELDS 5mo ago
        Unironically yeah, we are at the level of weaponizable sophistication that this metaphorical dick waving you are suggesting is probably something that happens
walrus01 5mo ago
Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.

kaltsturm 5mo ago

Denic will be added to the "Major DNSSEC Outages and Validation Failures" list: https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html

yowmamasita 5mo ago

The same day Kurzgesagt posted their video “Germany is over”. Huh. https://youtu.be/n-gYFcVx-8Y https://youtu.be/n-gYFcVx-8Y

cwassert 5mo ago
Kurzgesagt is once more highlighting the neoliberal solution: more growth. Surely a wealth tax is not worth mentioning.
- deleted 5mo ago
  [deleted]
- bwb 5mo ago
  I watched that video yesterday, their solution was to fix the pension funding problem :) They made the point that more immigration / growth wouldn't help fix the core problem if they don't fix that asap.

yassiniz 5mo ago

Shops open normally from 8am to 8pm in Germany. Today we decided to pilot opening hours for .de domains as well

deleted 5mo ago
[deleted]

bflesch 5mo ago

On Monday there was a huge outage affecting several cities quite close to Frankfurt because someone cut major fiber line; today DENIC is having a party and right when everyone is drunk this happens because some post-rotation task cannot be completed. There are too many coincidences happening.

neverrroot 5mo ago

[flagged]

kaltsturm 5mo ago

from my analysis DENIC resigned the .de zone today (May 5, 2026, ~17:49 UTC). The DNSSEC signature (RRSIG) for the NSEC3 record covering the hash range of nearly all .de TLD is cryptographically broken (malformed).

Animux 5mo ago

Seems to be fixed now.

Oarch 5mo ago

Germany has fallen.

kaltsturm 5mo ago

With chrome it works again

edb_123 5mo ago

Things seem to be on their way up now, and https://status.denic.de/ https://status.denic.de/ is working again, at least from here. DENIC's status page currently says "Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible.

jiveturkey 5mo ago

It’s not DNS There’s no way it’s DNS It was DNSSEC

taf2 5mo ago

ok i picked a bad day to move from one register to another... i just spent the last hour frantically trying to figure out why the new register screwed us or the old register was screwing us...

amelius 5mo ago

Maybe related to this? Crazy idea, but nothing surprises me anymore. https://edition.cnn.com/2026/05/01/politics/us-troop-withdrawal-germany-trump-merz https://edition.cnn.com/2026/05/01/politics/us-troop-withdra...

tom1337 5mo ago

Cloudflare has now disabled DNSSEC validation on their 1.1.1.1 resolver: https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz

cluckindan 5mo ago
If it turns out the DNSSEC issue was caused by threat actors, this downstream effect could very well have been the reason to do it.
- amluto 5mo ago
  It is indeed a bit sad that Cloudflare had to turn off DNSSEC completely. But I completely understand that they don't have a production-ready, tested path to override DNSSEC validation for only some domains.
  - vendemiat 5mo ago
    Sorry! status message was not clear. DNSSEC validation is temporarily disabled only for .de domains.
    - tptacek 5mo ago
      That's not much better!
  - fastest963 5mo ago
    [flagged]
    - tptacek 5mo ago
      It didn't originally say that. They added the clarification just a few minutes ago. The guidelines ask you not to ask people these kinds of questions, for what it's worth.
    - jonah-archive 5mo ago

Zopieux 5mo ago

That postmortem should be a fun read, can't wait.

retired 5mo ago
We shall transmit the postmortem to you via fax within 25 business days, ja.
Culonavirus 5mo ago
Ok children, sit down and listen, uncle Culonavirus will tell you a story: "It all began with the decommissioning of the last nuclear power plant, ..."
alper 5mo ago
Given how amateurish German IT operations is, there is no guarantee whatsoever there will be a post-mortem nor whether it then will make it out under 3-6 months with all the necessary approvals.
- Tepix 5mo ago
  Bla bla, always easy to rant... https://blog.denic.de/denic-informiert-uber-die-behebung-der-dnssec-storung-fur-de-domains/ https://blog.denic.de/denic-informiert-uber-die-behebung-der... "Die Störung ist inzwischen behoben und alle Systeme laufen wieder stabil. Die genaue Ursache wird derzeit noch analysiert. Sobald belastbare Erkenntnisse vorliegen, wird DENIC diese transparent zur Verfügung stellen." translation: ‘The disruption has now been resolved and all systems are running smoothly again. The exact cause is currently being investigated. As soon as reliable findings are available, DENIC will make them publicly available.’
  - account42 5mo ago
    Also always easy to announce "Sobald belastbare Erkenntnisse vorliegen, wird DENIC diese transparent zur Verfügung stellen." and then remain silent until the media forgets about the incident and never actually publish anything.

SEJeff 5mo ago

Just gonna leave this absolute gem from Thomas Ptacek on DNSSEC here: https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/

betaby 5mo ago
Aged like a milk.
- tptacek 5mo ago
  Oh, yeah, I'm sure feeling chastened right now. You got me.
  - SAI_Peregrinus 5mo ago
    Parmigianino-Reggiano is aged milk, so I'm not sure what people have against aged milk. Aged milk can be great
  - sgc 5mo ago
    My poor fellow. You wrote about how something is a bad tool for a long list of serious reasons. Then it failed spectacularly because everybody decided to depend on it anyway - exactly what you were cautioning against. But somehow you have to respond to people who think you are the one who got it wrong! As a third party the whole affair gave me a good chuckle at least ;)
    - tptacek 5mo ago
      Germany appears to depend on it. Virtually none of North America does. I'm pretty satisfied with how this whole thing shook out!
      - cyberax 5mo ago
        You're wrong. Both .com and .net are signed (`dig RRSIG com.`), and if they screw up, then all the com/net zones will become inaccessible.

deleted 5mo ago

[deleted]

basilikum 5mo ago

This is the kind of system failure that we need really good and well tested disaster recovery plans for. While not necessary this time, DENIC and any critical infrastructure provider should be able to rebuild their entire infrastructure from scratch in a tolerable amount of time (Rather days than hours in the case of a full rebuild). Importantly the disaster recovery plan has to work without reliance on either the system that is failing, but also on adjacent systems that might have hidden dependencies on the failing system. I'm really not too close to Denic and know nothing about their internals, but just close enough to have experienced the stress of someone working for DENIC second hand during the outage. From the very limited information I happened to gather DENIC had some trouble in addressing the issue because, surprise, infrastructure that they need to do so runs on de domains. [1] I'm convinced there are all kinds of extended cyclic decencies between different centralization points in the net. If some important backbone of the internet is down for an extended time, this will absolutely cause cascading failures. And thesw central points of failure are only getting worse. I love Let's Encrypt, but if something causes them to hard fail things will go really bad once certificates start to expire. We need concrete plans to cold start extended parts of the internet. If things go really bad once and communication lines start to fail, we're in for a bad time. Maybe governments have redundant, ultra resistant, low tech communication lines, war rooms and a list of important people in the industry who they can find and put in these war rooms so they can coordinate the rebuild of infrastructure. But I doubt it. [^1] I don't know if there is some kind of disaster plan in the drawer at DENIC that would address this. I don't mean to allege anything against DENIC specifically, but broadly speaking about companies and infrastructure providers, I would not be surprised if there was absolutely no plan on what to do if things really go down and how to cold start cyclic dependencies or where they even are.

toast0 5mo ago
> This is the kind of system failure that we need really good and well tested disaster recovery plans for. All the cool kids offer their services over multiple TLDs and have their name servers of record in multiple TLDs, too. It's not quite best practices for recursive DNS to regularly fetch the complete root zone to cache it, but it's not unreasonable to do so.

NooneAtAll3 5mo ago

quad9 seems to be having problems with DNSSEC as well

aboardRat4 5mo ago

https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html

evan0721 5mo ago

[dead]

0xbadcafebee 5mo ago

I can't wait for the .com TLD outage. Ya'll thought Cloudflare down was bad? Lol

baby 5mo ago

Should I do my usual rent about how the web PKI refuses to move to a consensus protocol

jdthedisciple 5mo ago

Seems up again. How briefly did the outage last?

alper 5mo ago

I'd expect political escalation for something like this but given that this is Germany, who knows.

farfatched 5mo ago

DNSSEC operations feels like one of those problems that should be tackled with formal methods, like how some subway controllers are. But I expect it's treated like "very serious and scary ops", which isn't wrong, but isn't enough.

egberts1 5mo ago

Resolved ... after recovering from a mass German DNSSEC drinking party? Ok.

pierroons 5mo ago

[dead]

blmaniac 5mo ago

It seems like DENIC has a problem again. My domain elektro-menne.de is down. I've been debugging this issue for more than an hour, and it seems like the domain exists in WHOIS and Cloudflare responds correctly when queried directly, but DENIC authoritative nameservers return NXDOMAIN. For example: dig @a.nic.de elektro-menne.de A returns NXDOMAIN directly from DENIC. Zonemaster also reports: "elektro-menne.de does not exist as a DNS zone." I also found multiple other .de domains currently affected by the exact same issue. Looks like the domains are present in WHOIS, but missing from the live .de DNS zone delegation.