5 ms·
Cloudflare outage on February 20, 2026
- boarush 8mo agoWhile neither am I nor the company I work for directly impacted by this outage, I wonder how long can Cloudflare take these hits and keep apologizing for it. Truly appreciate them being transparent about it, but businesses care more about SLAs and uptime than the incident report.
- llama052 8mo agoI’ll take clarity and actual RCAs than Microsoft’s approach of not notifying customers and keeping their status page green until enough people notice. One thing I do appreciate about cloudflare is their actual use of their status page. That’s not to say these outages are okay. They aren’t. However I’m pretty confident in saying that a lot of providers would have a big paper trail of outages if they were more honest to the same degree or more so than cloudflare. At least from what I’ve noticed, especially this year.
- boarush 8mo agoAzure straight up refuses to show me if there's even an incident even if I can literally not access shit. But last few months has been quite rough for Cloudflare, and a few outages on their Workers platform that didn't quite make the headlines too. Can't wait for Code Orange to get to production.
- jacquesm 8mo agoBluntly: they expended that credit a while ago. Those that can will move on. Those that can't have a real problem. As for your last sentence: Businesses really do care about the incident reports because they give good insight into whether they can trust the company going forward. Full transparency and a clear path to non-repetition due to process or software changes are called for. You be the judge of whether or not you think that standard has been met.
- boarush 8mo agoI might be looking at it differently, but aren't decisions over a certain provider of service being made by the management. Incident reports don't ever reach there in my experience.
- samrus 8mo agoIn my experience, the gist of it does reach management when its an existing vendor. Especially if management is tech literate Becuase management wants to know why the graphs all went to zero, and the engineers have nothing else to do but relay the incident report. This builds a perception for management of the vendor, and if the perception is that the vendor doesnt tell them shit or doesnt even seem to know theres an outage, then management can decide to shift vendors
- jacquesm 8mo agoEvery company that relies on their suppliers and that has mature management maintains internal supplier score cards as part of their risk assessment, more so for suppliers that are hard to find replacements for. They will of course all have their of thresholds for action but what has happened in the last period with CF exceeds most of the thresholds for management comfort that I'm aware of. Incident reports themselves are highly technical, so will not reach management because they are most likely simply not equipped to deal with them. But the CTOs of the companies will take notice, especially when their own committed SLAs are endangered and their own management asks them for an explanation. CF makes them all look bad right now.
- CommonGuy 8mo agoInsufficient mock data in the staging environment? Like no BYOIP prefixes at all? Since even one prefix should have shown that it would be deleted by that subtask... From all the recent outages, it sounds like Cloudflare is barely tested at all. Maybe they have lots of unit tests etc, but they do not seem to test their whole system... I get that their whole setup is vast, but even testing that subtask manually would have surfaced the bug
- asciii 8mo agoIt was also merged 15 days prior to production release...however, you're spot on with the empty test. That's a basic scenario that if it returned all...is like oh no.
- dabinat 8mo agoI think Cloudflare does not sufficiently test lesser-used options. I lurk in the R2 Discord and a lot of users seem to have problems with custom domains.
- martinald 8mo agoJust crazy. Why does a staging environment matter? They should be running some integration tests against eg an in memory database for these kinds of tasks surely?
- suhputt 8mo agomy guess is the company is rotting from the inside and drowning in tech debt
- zmj 8mo agoTesting the "whole system" for a mature enterprise product is quite difficult. The combinatorial explosion of account configurations and feature usage becomes intractable on two levels: engineers can't anticipate every scenario they need their tests to cover (because the product is too big understand the whole of), and even if comprehensive testing was possible - it would be impractical on some combination of time, flakiness, and cost.
- atty 8mo agoI do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this sort of thing, is that just recency bias?
- Icathian 8mo agoIt is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.
- NinjaTrance 8mo agoEngineers have been vibe coding a lot recently...
- jsheard 8mo agoThe featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.
- dryarzeg 8mo agoDaaS - Downtime as a Service© Just joking, no offence :)
- logicchains 8mo agoDaaS is good ja
- blibble 8mo agois this blog post LLM generated? the explanation makes no sense: > Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion. client: resp, err := d.doRequest(ctx, http.MethodGet, `/v1/prefixes?pending_delete`, nil) server: if v := req.URL.Query().Get("pending_delete"); v != "" { // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx) if err != nil { api.RenderError(ctx, w, ErrInternalError) return } api.Render(ctx, w, http.StatusOK, renderIPPrefixAPIResponse(prefixes, nil)) return } even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block
- bstsb 8mo agodoesn't look AI-generated. even if they have made a mistake, it's probably just from the rush of getting a postmortem out prior to root cause analysis
- bretthoerner 8mo ago> even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block If they passed in any value, they would have entered the block and returned early with the results of FetchPrefixesPendingDeletion. From the post: > this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them. They expected to drop into the block of code above, but since they didn't, they returned all routes.
- blibble 8mo agookay so the code which returned everything isn't there actual explanation: the API server by default returns everything. the client attempted to make a request to return "pending_deletes", but as the request was malformed, the API instead went down the default path, which returned everything. then the client deleted everything. makes sense now but is that explanation is even worse because that means the code path was never tested?
- NinjaTrance 8mo agoThe irony is that the outage was caused by a change from the "Code Orange: Fail Small initiative". They definitely failed big this time.
- deleted 8mo ago[deleted]
- ssiddharth 8mo agoThe eternal tech outage aphorism: It's always DNS, except for when it's BGP.
- subscribed 8mo agoYou could argue BGP is like DNS for IPs :)
- himata4113 8mo agoThis blog post is inaccurate, the prefixes were being revoked over and over - to keep your prefixes advertised you had to have a script that would readd them or else it would be withdrawn again. The way they seemed to word it is really dishonest.
- henning 8mo ago[flagged]
- sp00chy 8mo agothat’s my feeling also. We will get this more and more in future.
- VirusNewbie 8mo agoIf you track large SaaS and Cloud uptime, it seem to correlate pretty highly with compensation for big companies. Is cloudflare getting top talent?
- bombcar 8mo agoBased on IPO date and lockups, I suspect top talent is moving on.
- jaboostin 8mo agoHindsight is 20/20 but why not dry run this change in production and monitor the logs/metrics before enabling it? Seems prudent for any new “delete something in prod” change.
- anurag 8mo agoThe one redeeming feature of this failure is staged rollouts. As someone advertising routes through CF, we were quite happy to be spared from the initial 25%.
- tokyobreakfast 8mo ago[flagged]
- alansaber 8mo agoWell I still appreciate a good postmortem even if I have no doubt it'll happen again imminently
- bdangubic 8mo agoand if they didn’t we’d posting about lack of transparency. damned if you do, damned if you don’t
- samrus 8mo agoJust seems like transparency. I agree that we should also judge them based on the frequency of these incidents and amwhether they provide a path to non-repeatability, but i wouldnt criticize them for the transparency per se
- NooneAtAll3 8mo agoagain?
- otar 8mo agoReliability was/is CF's label. It's alarming already. Too many outages in the past months. CF should fix it, or it becomes unacceptable and people will leave the platform. I really hope they will figure things out.
- argestes 8mo agoI have many things dependent on Cloudflare. That makes me root for Cloudflare and I think I'm not the only one. Instead of finding better options we're getting stuck on an already failing HA solution. I wonder what caused this.
- arcatech 8mo agoDo you not feel concern about you and everybody else deciding to put ALL of their eggs into one basket like this?
- ranger_danger 8mo agoI would bet money that most people who use CF now are already hosting their endpoints at a single provider. I don't think most people care until it actually becomes enough of a problem.
- esseph 8mo agoLike AWS/GCP/Azure?
- slothsarecool 8mo agoThere are no alternatives, and those alternatives that did exist back in the day, had to shut down due to either going out of business or not being able to keep a paygo model. Not everybody needs cloudflare, but those that need it and aren't major enterprises, have no other option.
- pocksuppet 8mo agoLots of people who think they need Cloudflare don't. What are you using it for?
- alansaber 8mo agoNot sure why everyone is complaining, new MCP features are more important than uptime
- dilyevsky 8mo ago> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. Lmao, iirc long time ago Google's internal system had the same exact bug (treating empty as "all" in the delete call) that took down all their edges. Surprisingly there was little impact as traffic just routed through the next set of proxies.
- wa008 8mo agoThis transparent report can earn my trust
- vimda 8mo agoOne has to wonder when the board realises Dane was a bad replacement for JGC. These outages are getting ridiculous
- user205738 8mo agoThey should have rewritten this code in Rust using these brilliant language models. /jk
- djfobbz 8mo agoI'm honestly amazed that a company CF's size doesn't have a neat little cluster of Mac Minis running OpenClaw and quietly taking care of this for them.
- kgeist 8mo agoIt's something we debated in our team: if there's an API that returns data based on filters, what's the better behavior if no filters are provided - return everything or return nothing? The consensus was that returning everything is rarely what's desired, for two reasons: first, if the system grows, allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching from the DB => OOM, and additional stress on the DB) and for the user (the same problem on their side). Second, it's easy to forget to specify filters, especially in cases like "let's delete something based on some filters." So the standard practice now is to return nothing if no filters are provided, and we pay attention to it during code reviews. If the user does really want all the data, you can add pagination to your API. With pagination, it's very unlikely for the user to accidentally fetch everything because they must explicitly work with pagination tokens, etc. Another option, if you don't want pagination, is to have a separate method named accordingly, like ListAllObjects, without any filters.
- MobileVet 8mo agoI like your thought process around the ‘empty’ case. While the opposite of a filter is no filter, to your point, that is probably not really the desire when it comes to data retrieval. We might have to revisit that ourselves.
- alemanek 8mo agoReturning an empty result in that case may cause a more subtle failure. I would think returning an error would be a bit better as it would clearly communicate that the caller called the API endpoint incorrectly. If it’s HTTP a 400 Bad Request status code would seem appropriate.
- pigpag 7mo ago[dead]
- Philip-J-Fry 8mo ago>allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching from the DB => OOM, and additional stress on the DB) You can limit stress on RAM by streaming the data. You should ideally stream rows for any large dataset. Otherwise, like you say you are loading the entire thing into RAM.
- abalone 8mo agoThe code they posted doesn't quite explain the root cause. This is a good study case for resilient API design and testing. They said their /v1/prefixes endpoint has this snippet: if v := req.URL.Query().Get("pending_delete"); v != "" { // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx) [..snip..] } What's implied but not shown here is that endpoint normally returns all prefixes. They modified it to return just those pending deletion when passing a pending_delete query string parameter. The immediate problem of course is this block will never execute if pending_delete has no value: /v1/prefixes?pending_delete <-- doesn't execute block This is because Go defaults query params to empty strings and the if statement skips this case. Which makes you wonder, what is the value supposed to be? This is not explained. If it's supposed to be: /v1/prefixes?pending_delete=true <--- executes block Then this would work, but the implementation fails to validate this value. From this you can infer that no unit test was written to exercise the value: /v1/prefixes?pending_delete=false <-- wrongly executes block The post explains "initial testing and code review focused on the BYOIP self-service API journey." We can reasonably guess their tests were passing some kind of "true" value for the param, either explicitly or using a client that defaulted param values. What they didn't test was how their new service actually called it. So, while there's plenty to criticize on the testing front, that's first and foremost a basic failure to clearly define an API contract and implement unit tests for it. But there's a third problem, in my view the biggest one, at the design level. For a critical delete path they chose to overload an existing endpoint that defaults to returning everything. This was a dangerous move. When high stakes data loss bugs are a potential outcome, it's worth considering more restrictive API that is harder to use incorrectly. If they had implemented a dedicated endpoint for pending deletes they would have likely omitted this default behavior meant for non-destructive read paths. In my experience, these sorts of decisions can stem from team ownership differences. If you owned the prefixes service and were writing an automated agent that could blow away everything, you might write a dedicated endpoint for it. But if you submitted a request to a separate team to enhance their service to returns a subset of X, without explaining the context or use case very much, they may be more inclined to modify the existing endpoint for getting X. The lack of context and communication can end up missing the risks involved. Final note: It's a little odd that the implementation uses Go's "if with short statement" syntax when v is only ever used once. This isn't wrong per se but it's strange and makes me wonder to what extent an LLM was involved.
- est 8mo agobitbucket was done for a while as well. Seems no one noticed.
- snowhale 8mo ago[dead]
- Bender 7mo agoOld tech could work around these outages. Set up GSLB at a DNS provider that does health checks or perform your own health checks to both origin and CDN's and use API's to change DNS. If the origin servers are OK and the CDN is not, automatically change DNS to a different CDN. There should be multiple probes that form a consensus. This process assumes one is managing the configurations of their CDN's through code and API so that one can set up and tear down any number of CDN's on a whim. That does mean having contracts with more than one CDN provider however the cost should be negotiated based on monthly volume. i.e. the CDN with the most uptime gets the most money. If an existing CDN under contract refuses to negotiate then move some non critical path services to them and let that contract expire. Instate a company wide policy to never return to a vendor if their contract was intentionally not renewed.
- fjoaasdfas 7mo agoyikes: https://github.com/golang/go/blob/master/src/net/url/url.go#L888 https://github.com/golang/go/blob/master/src/net/url/url.go#... maybe go can do (string v, ok bool) for this or add proper sum types...