9 ms·
Cloudflare outage – 24 hours now
- arp242 3y agoCloudflare Continuing to Experience Outages - https://news.ycombinator.com/item?id=38121370 https://news.ycombinator.com/item?id=38121370 - Nov 2023 (2 comments) Cloudflare Is Down (Again) - https://news.ycombinator.com/item?id=38116892 https://news.ycombinator.com/item?id=38116892 - Nov 2023 (2 comments) Cloudflare API Down - https://news.ycombinator.com/item?id=38112515 https://news.ycombinator.com/item?id=38112515 - Nov 2023 (141 comments) Cloudflare incident on October 30, 2023 - https://news.ycombinator.com/item?id=38100932 https://news.ycombinator.com/item?id=38100932 - Nov 2023 (29 comments)
- creshal 3y agoI'm really looking forward to the post-mortem to this.
- liotier 3y agoCloudflare's greatest product is arguably its blog !
- laluser 3y agoPower outage + data inconsistency issues.
- BillinghamJ 3y agoIsn't the real issue that the control plane isn't decentralized/redundant? Entirely dependent on PDX
- dogweather 3y agoI can't believe we haven't heard anything yet. AFAIK we've only been told, "power outage", which was resolved yesterday. What should our expectations be? The best assumption could be that this is the new normal.
- spacebacon 3y agoHmm... Who just changed their dns vs. riding it out?
- issafram 3y agoIt hasn't affected my home network at all. I use their DNS servers and nothing has resolving addresses has not stopped working
- 0x0000000 3y agoParent comment was likely referring to authoritative DNS, not Cloudflare's public resolvers.
- sirius87 3y agoI was in the midst of migrating my namecheap domain from Route53 to Cloudflare. Set up all the DNS records while ignoring the /api/ errors shown at the bottom of the Cloudflare dashboard thinking some ad block setting in my browser was messed up. Then I realised setting the NS in Namecheap to Cloudflare's nameservers was taking an inordinate amount of time to propagate, and that's when I checked X/Twitter. Set it back to Route53.
- burcs 3y agoWe did, we were slowly working towards migrating to AWS entirely and this just helped expedite it.
- dogweather 3y agoI'm planning my transition away for 10 or so subdomains and 30 records. The only feature I need to research in new providers is: access to Whois ASN numbers, which I insert into HTTP request headers. I use this to tailor my site for .gov and .edu users.
- tootie 3y agoI've heard tell of massive DDoS attacks against international news sources (AP, Reuters, NY Times). Not sure if this is related.
- andrewinardeer 3y agoAnymore info on that?
- tootie 3y agoI can't find any published details, it was just circling the media biz.
- nkcmr 3y agoIn this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg
- ta1243 3y agoIf you can't cope with the loss of a data centre you're not really running a resilient system.
- itslennysfault 3y agoOn yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.
- sangnoir 3y agoNot great to know an aspiring hyperscaler is one tornado away from insolvency.
- samlinnfer 3y agoBTW Cloudflare tunnels are not working (for the at least the last 16 hours), but it says "Operational" and "restored" on the ticket. Since Shopify's CLI uses Cloudflare tunnels by default to load local resources, Shopify partners are affected by this outage by unable to develop apps, unless they use another tunnel: [0] https://github.com/Shopify/cli/issues/3065 https://github.com/Shopify/cli/issues/3065 [1] https://github.com/Shopify/cli/issues/3060 https://github.com/Shopify/cli/issues/3060
- perryizgr8 3y agoData point of just one, but my tunnels are working just fine.
- samlinnfer 3y agoIf you've previously created a tunnel it will still work, just don't close it because you won't be able to open a new one.
- supertron 3y agoYeah, can confirm the same here. This seems to be the most frequent failure mode that I see with my use of Tunnels. Existing connections will continue working, but attempts to create new ones (or other changes) fail. Ie. the control plane fails more often than the core connectivity service. I certainly prefer that failure mode to the opposite, but I do find the status information on Cloudflare's page to be very confusing about this. Tunnels as a product is essentially heavily degraded (putting it lightly) and yet it's listed currently as "Cloudflare Tunnel: restored" [0] https://www.cloudflarestatus.com https://www.cloudflarestatus.com [0] Edit: also having used Shopify's CLI a little, one thing I noticed immediately is how opaque the whole thing is. They want to push you down a very specific path, and don't provide a lot of information if you want to take a bit more control over your dev process (as I always want to do) which directly leads to points of failure like this. From your GitHub links it looks like devs are struggling to figure out how to quickly switch to a different reverse proxy.
- 3y ago
- sanat 3y agoI run hirevire.com one way video interview SaaS - and we were pretty much dead in the water during the Cloudflare Stream outage. We moved out to BunnyCDN's stream after waiting for 20 hours. One side benefit is that our videos are now stored in EU instead of Cloudflare's <hand wavy> edge location around you.
- summarity 3y agoI've also been using Bunny's image and video delivery, while using CF for everything else. It's pretty neat - it just works. I like having both in my toolbelt, makes fallbacks like these easy.
- karlerss 3y agoHow much work was the migration? Were the APIs feature-compatible or did you lose functionality?
- sanat 3y agoThe migration work was only a couple of hours for our core process. Took us 4 in total to restart collecting video. We still have some accessory features to be moved to video on Bunny. Like transcriptions, downloads.
- epolanski 3y agoHonestly Cloudflare's PR pissed me off yesterday. Straight up going on LinkedIn and other socials telling everything was solved in one hour (actually 37 minutes), even though I and many other companies I know still had issues with their services *16 hours after* the post. Those are things that make me reconsider my position with Cloudflare. Straight up lying and not verifying whether your customers are able to operate on your platform while impacting their operations but making PR stunts about how good and fast they are at solving critical issues is something that erodes credibility. Especially after they used the Okta security failure to bash them on their blog for their lack of honest communication to their customers.
- corobo 3y agoIs it possible that you're referencing the other outage from the 30th? Just going by the 37 minutes number as it's very specific. This outage (not the current one) was 37 minutes long: https://blog.cloudflare.com/cloudflare-incident-on-october-30-2023/ https://blog.cloudflare.com/cloudflare-incident-on-october-3...
- anacrolix 3y agoThey are straight up scumbags.
- sailingparrot 3y agoI never experienced a longer than 12 hours outage with any service provider over my ~13 years career (maybe I was lucky). But thanks to Cloudflare I have been able to enjoy not just one, but two ~24h outages in not even a month! Jokes aside, it must be extremely stressful to be a SRE at CF recently. But something is clearly wrong over there. We have been burned so bad there is no chance we will touch CF ever again in the next decade once our migration off of it is complete.
- CodesInChaos 3y ago> two ~24h outages in not even a month Wasn't the previous outage on Oct 30 less than an hour?
- sailingparrot 3y agoYep, but on Oct 9 they were down for 22h.
- ta1243 3y agoTrying high for that 2-nines reliability. You just can't get that level of reliability if you do it yourself, no matter how hard you try.
- sailingparrot 3y agoWe won't do it ourselves, but we also won't do it with a provider that has accumulated 50+ hours of downtime in less than a month all the while having no communication or support. That's barely clearing the one nine availability for the last 30 days (93%) for our particular stack on CF, this is insane. Mind you last time we were hit by a 22h outage on Oct. 9 we didn't get so much as an email from CF either during or after the outage.
- samtho 3y agoThe no communication or support is the real killer here, imo. I can understand them having some catastrophic issues which I would be reasonably confident they could fix, it’s the uncertainty of the situation that makes me worry. Is this so easy to fix it will be back in less than an hour and they will communicate then? Are they going dark and I need to find a new provider asap?
- sph 3y agoLooking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.
- renonce 3y agoThe Internet has been decentralized from the beginning. Now I don't want to claim that Cloudflare made something worse (at least it's enabling a lot of websites to exist without fear of DDoS) but the fact is that Cloudflare made it more centralized, as there are lots of websites that cannot be accessed without going through Cloudflare.
- belthesar 3y agoI think you might have missed the joke on this one.
- joepie91_ 3y agoHonestly, the original post could have been a joke, or it could not have been. I regularly talk to people who seem to genuinely believe this sort of thing (on this topic and others).
- sph 3y agoI am from an era no one was serious on the Internet, so we didn't need to add /s every time we were sarcastic. In that era we also saw the last sysadmin configuring Apache with their bare hands without the help of Cloudflare.
- TeMPOraL 3y ago> decentralised global Internet, with packets being routed through alternative paths > I understand we do not have the technology for that just yet I looked at my router, remembered the term "packet-switched network", and wept.
- hipadev23 3y agoIs there a summary of what Cloudflare services are operational? Feels like it would be easier to track.
- kijin 3y agoBasic proxying seems to be working fine for me. Existing DNS records continue to be served. Existing files on R2 are accessible. Can't change anything without a bunch of API errors, though. Hope I don't need to turn on "I'm under attack" anytime soon.
- l5870uoo9y 3y agoWonder if this is related to the many product launches recently? Even though my general impression is that they test rigorously with long-running alpha and beta test phases.
- stpe 3y agoIt is the consequences of a power outage in Flexential PDX02 data center.
- dogweather 3y agoThat's the inference, but AFAIK there's been no direct assertion or explanation: Why has CF been knocked back to Alpha-status reliability across the board.
- dixie_land 3y agoA silver lining I take from this is at least we have incidents page hosted somewhere else :) I look fondly to earlier AWS outages where everything is Green on the status page because the Red icon hosted on S3 was down...
- deleted 3y ago[deleted]
- dogweather 3y agoHas Cloudflare said anything of substance yet? This is far beyond a simple power outage.
- gtirloni 3y agohttps://www.theregister.com/2023/11/02/cloudflare_outage/ https://www.theregister.com/2023/11/02/cloudflare_outage/ """ In a nutshell, Cloudflare rolled out a new KV build to production. It turned out that the deployment tool had a bug, and some traffic got diverted to the wrong destination, which triggered a rollback … which failed. The result was that engineers had to manually switch the production route to the previous working version of Workers KV. The problem is that an awful lot of Cloudflare products and services depend on Workers KV, meaning that when there is a problem with the platform, the blast radius can be impressive. """
- tux3 3y agoThe KV outage is the previous one, from Nov 1st. We're currently in the Nov 2-3 outage, soon to rollover into Nov 4 in my timezone. This one is the power outage — also mentioned in the article — but unrelated to KV.
- client4 3y agoThat are having issues with the new process spanning global MITM'd traffic to the NSA.
- sponaugle 3y agoCloudflare Postmortem: https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/ https://blog.cloudflare.com/post-mortem-on-cloudflare-contro... "On November 2 at 08:50 UTC Portland General Electric (PGE), the utility company that services PDX-04, had an unplanned maintenance event affecting one of their independent power feeds into the building. That event shut down one feed into PDX-04. The data center has multiple feeds with some level of independence that can power the facility. However, Flexential powered up their generators to effectively supplement the feed that was down. Counter to best practices, Flexential did not inform Cloudflare that they had failed over to generator power. None of our observability tools were able to detect that the source of power had changed. Had they informed us, we would have stood up a team to monitor the facility closely and move control plane services that were dependent on that facility out while it was degraded. It is also unusual that Flexential ran both the one remaining utility feed and the generators at the same time. It is not unusual for utilities to ask data centers to drop off the grid when power demands are high and run exclusively on generators. Flexential operates 10 generators, inclusive of redundant units, capable of supporting the facility at full load. It would also have been possible for Flexential to run the facility only from the remaining utility feed. We haven't gotten a clear answer why they ran utility power and generator power."