48 ms·
1.1.1.1 lookup failures on October 4th, 2023
- suprjami 3y agoStrangely I noticed this because some parts of eBay stopped loading. I spent a while troubleshooting my privacy/adblock nonsense because surely CloudFlare couldn't be down but that's the only conclusion I could come to.
- ryanjshaw 3y agoI thought it was a WSL (Linux on Windows) issue. I did some googling and saw some people discussing DNS issues on WSL, which were to be resolved by setting up a manual DNS config. They used 8.8.8.8 in their example, which I of course changed to 1.1.1.1! (Un)surprisingly the issue did not resolve after this, and I went on to do other things, hoping the gremlins would find something better to do in the meantime.
- ChrisArchitect 3y agoEarlier discussion while outage was active: https://news.ycombinator.com/item?id=37763143 https://news.ycombinator.com/item?id=37763143
- morugam 3y agoWe noticed this through our own, homegrown scripts that check for this, having been screwed by an outage a few years ago. I'm happy they so quickly acknowledge and explain these issues. Good work!
- LeoPanthera 3y agoDid 1.0.0.1 also go down? The article doesn't say.
- homero 3y agoOf course it did it's the same service
- toast0 3y agoA highly reliable service might run one partition on a completely separate serving stack. It's worth asking.
- tedunangst 3y agoWith a completely separate architecture that also loads the root zones in a completely different way?
- toast0 3y agoIt's certainly possible to do. Public recursive DNS isn't that big of a spec that you can't have two independent teams doing it, and mandate they do things differently when possible. Run the two /24s on different ASNs, too. I've had sales teams pitch me on their authoritative DNS service running heterogenously, although I guessing both partitions gathered config from a single place, but we didn't get that far in the potential customer pipeline. Of course, just because it's possible doesn't mean they're likely to do it.
- tedunangst 3y agoMaybe also run by a separate company with a different CEO for additional redundancy. :) so now we've got 1. and 8.
- arp242 3y agoBest to service from a different planet, in case something happens to this one.
- BlueTemplar 3y agoI hear Musk has been working on that. (j/k, 40 min ping time...)
- jacurtis 3y ago
- throwaway67743 3y ago[flagged]
- jarym 3y agoUp until a few months ago the HN crowd loved Cloudflare. How sentiment has changed in such a short period. My guess would be their weird ‘site protection’ stuff is burning too many people and negatively impacting their reputation.
- throwaway67743 3y agoI've never loved cloudflare - as someone doing this long before they existed I see through their wordy blog posts about rookie mistakes. It's embarrassing really.
- kkielhofner 3y ago> My guess would be their weird ‘site protection’ stuff is burning too many people and negatively impacting their reputation. What's always been interesting to me about this take is it's not as though Cloudflare is randomly inserting themselves in internet traffic. Cloudflare customers have choice in the marketplace and they chose Cloudflare for whatever reasons. If end-users take issue with accessing the site of a Cloudflare customer they should take it up with the owners of the site that chose Cloudflare. Theoretically the Cloudflare customer would take it up with them if it becomes problematic. Cloudflare has no obligation to the site end-users other than meeting the needs of their customer who does have obligation to their end-users (theoretically). Cloudflare is, ostensibly, providing a solution for their customers. How that impacts their customer's end-users is between Cloudflare and the customer.
- reaperman 3y agoIn general, I always seem to find comments along the lines of this are very easy to thoroughly disprove. There has been consistent criticism of Cloudflare for many years, ever since the majority of web traffic started going through their anti-DDOS and anti-bot gateways. Here's a HN post with lots of very critical comments[0] from 7 years ago, including a fairly scathing one from 'tptacek. Even way back then, you'd get the same comments you hear today like: > So rather than demand fixes for the fundamental issues that enable ddos attacks (preventing IP spoofing, allowing infected computers to remain connected, etc), we just continue down this path of massive centralization of services into a few big players that can afford the arms race against bonnets. Using services like Cloudflare as a 'fix' is wrecking the decentralized principles of the Internet. At that point we might as well just write all apps as Facebook widgets. 0: https://news.ycombinator.com/item?id=13718947 https://news.ycombinator.com/item?id=13718947
- tedunangst 3y ago> Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer. Ironic.
- denysvitali 3y agoMy only concern: 7:57 UTC: first reports coming in I noticed this issue quite quickly ("reported" at 7:54 UTC [1]), and I noticed I wasn't alone thanks to Twitter / X. I tried to get in touch with Cloudflare to report this issue - but I haven't found any meaningful contact other than Twitter. For such an important service, I'm impressed there is no contact email / form where you can get in touch with the engineers responsible for keeping the service up and running. Other than that, kudos for the well written blog post - as always! [1]: https://nitter.net/DenysVitali/status/1709476961523835246 https://nitter.net/DenysVitali/status/1709476961523835246
- hackmiester 3y agoGet what you pay for I guess.
- cj 3y agoThis sounds so foreign to my experience. Even when I just had 2 paid domains with Cloudflare, there was real time chat support that seemed to always have an available engineer when something came up. Maybe they don't provide support like that to free users.
- abraham 3y agoYou have to pay $200/month for access to live chat support. https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/
- denysvitali 3y agoI don't even have a Cloudflare account. You don't need a Cloudflare account to use their public DNS servers
- lopkeny12ko 3y agoDo you truly believe that the turnaround time for you contacting support, speaking to a first-tier support agent, having the request escalated to engineering, having a support engineer triaging it to the right team, and finally having the report being gauged so urgent that it pages the oncall for that team, is faster than automated infrastructure monitoring that the teams already have? For something as core as DNS, I'm sure engineers were aware about the issue within minutes. There's a lot of politics and processes between that and public acknowledgment of an incident.
- araes 3y agoI like how its a 42 joke. 4(0b10) 7:00 ends at 11:02 (4 hr 2 min) on a 4 sum 2x2. And refs to 1.1.1.1 vs 1.0.0.1
- robhlt 3y agoThe lack of additional alerts in the Remediation section is a little bit concerning. Adding an alert for serving stale root zone data is great, but I think a few more would be very useful too: - There's a clear uptick in SERVFAIL responses at 7:00 UTC but they don't start their response until an hour later after receiving external reports. This uptick should have automatically triggered an alert. It can't have been within the normal range because they got customer reports about it. - The resolver failed to load the root zone data on startup and resorted a fallback path. Even if this isn't an error for the resolver it should still be an alert for the static_zone service, because its only client is failing to consume its data. - The static_zone service should also alert when some percentage of instances fail to parse the root zone data, to get ahead of potential problems before the existing data becomes stale.
- RockRobotRock 3y agonot to be glib but you should consider working for them
- throwawaaarrgh 3y agoIf you can see shit falling apart and you're not even inside the org, it's probably a tire fire, and working there will just be stressful
- RockRobotRock 3y ago"shit falling apart" is a little dramatic. They're a big reputable org that always does writeups when they fuck up. tech people appreciate that. I have my own unrelated issues with Cloudflare as a company
- xcdzvyn 3y agoSure. CloudFlare recruiters, DM me :-)
- jrockway 3y agoSERVFAIL might not be a good enough signal for alerting. I definitely see third-party DNS providers returning SERVFAIL for their own reasons; if it's a popular host (looking at you, Route 53), then you'll proxy those through and end up alerting on AWS's issue instead of your own. They might just want a prober; ask every server for cloudflare.com every minute. If that errors, there are big problems. (I remember Google adding itself as malware many many years ago. Nice to have a continuous check for that sort of thing, which I am sure they do now.)
- homero 3y agoThis got me. I spent an hour trying to figure out why my Internet seemingly went down but not fully
- ktta 3y agoI'm expecting @tptacek to come in anytime reminding us of his thoughts on DNSSEC https://sockpuppet.org/blog/2015/01/15/against-dnssec/ https://sockpuppet.org/blog/2015/01/15/against-dnssec/
- dylukes 3y agoPeople who are actually correct on the basis of significant experience don't tend to feel it necessary to remind others at every possible occasion (unless not doing so would lead to significant harm to others). They're usually busy doing other things.
- sambazi 3y ago"Grant me the serenity to accept the things I cannot change, the courage to change the things I can and the wisdom to know the difference." but yea, repeatedly running into a wall of ignorance encourages shifting to "busy doing other things"
- tptacek 3y agounless not doing so would lead to significant harm to others Ding ding ding. At any rate: it's very funny that DNSSEC took 1.1.1.1 down, but this bug can't honestly be pinned on DNSSEC itself.
- dan15 3y agoIn a way they're lucky DNSSEC took it down, otherwise they may have not noticed the issue of using stale data for much longer.
- pornel 3y agoThey have expired the data correctly, which uncovered a bug in fetching of a new DNSSEC record. If the DNSSEC didn't add new unnecessary complexity to an otherwise working system, there would be no bug, and no stale data.
- jolux 3y agoThe buried lede here seems to be that this is yet another serious outage (indirectly) caused by using DNSSEC, though I understand why they don’t emphasize this part, given their strong advocacy for DNSSEC adoption.
- dc396 3y agoAs I read it, DNSSEC signature expiry was what triggered people noticing the root data was stale. That would seem to be a somewhat positive outcome.
- ipaddr 3y agoWould you rather an error or stale dns?
- mcesch 3y agoHaving had to troubleshoot a third-party service not so dissimilar to 1.1.1.1 and prove to them that their infrastructure was misbehaving in a similar manner, I'll take the error thank you.
- patmorgan23 3y agoWell it's a free public resolver service. There's plenty of other options and you can even run your own very easily.
- jolux 3y ago> DNSSEC signature expiry was what triggered people noticing the root data was stale If you noticed your brakes had failed because you ended up in a ditch I wouldn't really say that's a positive outcome. Frankly I can't believe they don't have better monitoring for a system as critical as that.
- mcesch 3y agoI don't know why Cloudflare, like Amazon, often get a free-pass on HN for their DNS implementation bugs. Regardless of DNSSEC's merits or otherwise, this bug isn't inherent to DNSSEC.
- icecap12 3y agoThese guys always have the best write-ups. I aspire to write postmortems like they do. In fact, maybe my next context for a ChatGPT query will be “you are a member of of CloudFlare’s public relations team…” All joking aside, it occurred to me that the vast majority of internet and even tech users know very little about DNS. For the longest time, I was in the same boat. After having been in a role where it was necessary to understand the record types and deploy DNS configs, I’m quite thankful I learned. Just remember…it’s always DNS. edit: typo
- indigodaddy 3y agoWould this have affected any authoritative DNS served by them (eg and also taking 1.1.1.1 out of the equation..) Thinking not correct?
- globular-toast 3y agoSome people on my team experienced random breakage around 3rd party 2FA yesterday during this time period. I wouldn't be at all surprised if this was the cause, somehow. I've no idea how, though. It's so easy to become dependent on single points of failure without realising.
- minusf 3y ago> 1.1.1.1 has a WebAssembly app called static_zone running on top of the main DNS logic that serves those new versions when they are available. webassembly? what is that word even doing in a post mortem about DNSSEC failures?
- spoiler 3y agoWasm runtimes are great for stuff like plugins! Seems like this static_zone thing was something like that (but they call them apps)
- Wallacy 3y agoCF Workers (that runs WebAssembly) are all over place. They may not run the main logic (not the actual Ngix, or DNSEC code) but they are used for several maintaince tasks.
- nilslice 3y agoworth a read! https://blog.cloudflare.com/big-pineapple-intro/ https://blog.cloudflare.com/big-pineapple-intro/
- vb-8448 3y agoSo, another monitoring issue here. It's incredible how far the SW industry gone in the last decades, but the way monitoring is done is the same as in the 90's.
- Charles_Stover 3y ago[dead]
- Charles_Stover 3y ago[dead]