14 ms·
Cloudflare outage on December 5, 2025
- mmmlinux 10mo agoMessing around on a Friday? Brave.
- orphea 10mo agoIf you're afraid of deploying on Friday, you're doing it wrong.
- roguecoder 10mo agoOr overworked. We can deploy on Fridays. We don't, because we aren't donating our time to the shareholders.
- chickensong 10mo agoYou don't really want security updates waiting around on a luxury schedule.
- jpeter 10mo agoUnwrap() strikes again
- throwawaymaths 10mo agothis time in lua. cloudflare can't catch a break
- RoyTyrell 10mo agoOr they're not thoroughly testing changes before pushing them out. As I've seen some others say, CloudFlare at this point should be considered critical infrastructure. Maybe not like power but dang close.
- esseph 10mo agoMy power goes out every Wednesday around noon and normally if the weather is bad. In a major US metro. I hope cloudflare is far more resilient than local power.
- gcau 10mo agoThe 'rewrite it in lua' crowd are oddly silent now.
- barbazoo 10mo agoHow do you know?
- infrcg 10mo ago[flagged]
- jcmfernandes 10mo agoDid you really go through the trouble of creating an account just to spit trash? Damn!
- rvz 10mo agoTime to use boring languages such as Java and Go.
- lexoj 10mo agoAnyone knows why lua? Or is it perhaps as a redis script in lua?
- lexoj 10mo agoFigured it, its prob a nginx lua module
- deleted 10mo ago[deleted]
- dap 10mo agoI guess you’re being facetious but for those who didn’t click through: > This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur.
- skywhopper 10mo agoThat bit may be true, but the underlying error of a null reference that caused a panic was exactly the same in both incidents.
- roguecoder 10mo agoYep: it is wild for them to claim that a strongly-typed language would have saved them when it didn't. Relying on language features instead of writing code well will always eventually backfire.
- dap 10mo agoYou're right that you have to "write code well" to prevent this sort of thing. It's also true that Rust's language features, if you use them, can make this sort of mistake a compile-time error rather than something that only blows up at runtime under the wrong conditions. The problem with their last outage was that somebody explicitly opted out of the tool provided by the language. As you say, that's "not writing code well". But I think you're dismissing the value of the language feature in helping you write code well.
- rossjudson 10mo agoRelying on code ninja ego backfires way sooner, and way more often. https://security.googleblog.com/2025/11/rust-in-android-move-fast-fix-things.html https://security.googleblog.com/2025/11/rust-in-android-move...
- barbazoo 10mo ago> Customers that did not have the configuration above applied were not impacted. Customer traffic served by our China network was also not impacted. Interesting.
- flaminHotSpeedo 10mo agoThey kinda buried the lede there, 28% failure rate for 100% of customers isn't the same as 100% failure rate for 28% of customers
- Scaevolus 10mo ago> Disabling this was done using our global configuration system. This system does not use gradual rollouts but rather propagates changes within seconds to the entire network and is under review following the outage we recently experienced on November 18. > As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following LUA exception: They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. > as part of this rollout, we identified an increase in errors in one of our internal tools which we use to test and improve new WAF rules Warning signs like this are how you know that something might be wrong!
- philipwhiuk 10mo ago> Warning signs like this are how you know that something might be wrong! Yes, as they explain it's the rollback that was triggered due to seeing these errors that broke stuff.
- Scaevolus 10mo agoThey saw errors and decided to do a second rollout to disable the component generating errors, causing a major outage.
- JesseJames210 10mo agoThat was their first mistake, if your deployment does not behave the way you expect to (or even give you bad smell) roll back, that how it used to be... when I was a kid...lol Or I don't know, maybe load test before you deploy.....?
- 8cvor6j844qw_d6 10mo agoWould be nice if the outage dashboards are directly linked to this instead of whatever they have now.
- testplzignore 10mo ago> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deployed and propagated to the network > 08:48: Change fully propagated > 08:50: Automated alerts > 09:11: Configuration change reverted and propagation start > 09:12: Revert fully propagated, all traffic restored 2 minutes for their automated alerts to fire is terrible. For a system that is expected to have no downtime, they should have been alerted to the spike in 500 errors within seconds before the changes even fully propagated. Ideally the rollback would have been automated, but even if it is manual, the dude pressing the deploy button should have had realtime metrics on a second display with his finger hovering over the rollback button. Ok, so they want to take the approach of roll forward instead of immediate rollback. Again, that's a valid approach, but you need to be prepared. At 08:48, they would have had tens of millions of "init.lua:314: attempt to index field 'execute'" messages being logged per second. Exact line of code. Not a complex issue. They should have had engineers reading that code and piecing this together by 08:49. The change you just deployed was to disable an "execute" rule. Put two and two together. Initiate rollback by 08:50. How disconnected are the teams that do deployments vs the teams that understand the code? How many minutes were they scratching their butts wondering "what is init.lua"? Are they deploying while their best engineers are sleeping?
- kachapopopow 10mo agowhy does this seem oddly familiar (fail-closed logic)
- deleted 10mo ago[deleted]
- xnorswap 10mo agoMy understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once". But a more important takeaway: > This type of code error is prevented by languages with strong type systems
- skywhopper 10mo agoThis is the exact same type of error that happened in their Rust code last time. Strong type systems don’t protect you from lazy programming.
- inejge 10mo agoIt's not remotely the same type of error -- error non-handling is very visible in the Rust code, while the Lua code shows the happy path, with no indication that it could explode at runtime. Perhaps it's the similar way of not testing the possible error path, which is an organizational problem.
- debugnik 10mo agoPrevented unless they assert the wrong invariant at runtime like they did last time.
- jsnell 10mo agoThat's a bizarre takeaway for them to suggest, when they had exactly the same kind of bug with Rust like three weeks ago. (In both cases they had code implicitly expecting results to be available. When the results weren't available, they terminated processing of the request with an exception-like mechanism. And then they had the upstream services fail closed, despite the failing requests being to optional sidecars rather than on the critical query path.)
- littlestymaar 10mo agoIn fairness, the previous bug (with the Rust unwrap) should never have happened: someone explicitly called the panicking function, the review didn't catch it and the CI didn't catch it. It required a significant organizational failure to happen. These happen but they ought to be rarer than your average bug (unless your organization is fundamentally malfunctioning, that is)
- deleted 10mo ago[deleted]
- flaminHotSpeedo 10mo agoWhat's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment system that caused their 11/18 outage. Pure speculation, but to me that sounds like there's more to the story, this sounds like the sort of cowboy decision a team makes when they've either already broken all the rules or weren't following them in the first place
- fidotron 10mo ago> This change was being rolled out using our gradual deployment system, and, as part of this rollout, we identified an increase in errors in one of our internal tools which we use to test and improve new WAF rules. As this was an internal tool, and the fix being rolled out was a security improvement, we decided to disable the tool for the time being as it was not required to serve or protect customer traffic. Come on. This PM raises more questions than it answers, such as why exactly China would have been immune.
- skywhopper 10mo agoChina is probably a completely separate partition of their network.
- fidotron 10mo agoOne that doesn't get proactive security rollouts, it would seem.
- roguecoder 10mo agoThe deploys are very unlikely to be managed from the same system.
- skywhopper 10mo agoI assume it was next on the checklist, or assigned to a different ops team.
- miyuru 10mo agoWhats going on with cloudflare's software team? I have seen similar bugs in cloudflare API recently as well. There is an endpoint for a feature that is available only to enterprise users, but the check for whether the user is on an enterprise plan is done at the last step.
- archon810 10mo agoI recently ran into an issue with the Cloudflare API feature that if you want to roll back requires contacting the support team because there's no way to roll it back with the API or GUI. Even when the exact issue was pointed out, it took multiple days to change the setting and to my knowledge there's still no API fix available. https://www.answeroverflow.com/m/1234405297787764816 https://www.answeroverflow.com/m/1234405297787764816
- LelouBil 10mo agoCan you elaborate? I'm not sure what you mean by "at the last step"
- Etheryte 10mo agoI'm not sure which endpoint gp meant, but as I understood it, as an example, imagine a three-way handshake that's only available to enterprise users. Instead of failing a regular user on the first step, they allow steps one and two, but then do the check on step three and fail there.
- miyuru 10mo agoThe API endpoint I am talking about needs a external verification. they allow to do the external verification before checking if the user is on the enterprise plan or not. The feature is only available to enterprise plans, it should not even allow external verification.
- 65 10mo agoMy guess? Code written by AI
- antiloper 10mo agoMake faster websites: > we started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications. Why is the Next.js limit 1 MB? It's not enough for uploading user generated content (photographs, scanned invoices), but a 1 MB request body for even multiple JSON API calls is ridiculous. There frameworks need to at least provide some pushback to unoptimized development, even if it's just a lower default request body limit. Otherwise all web applications will become as slow as the MS office suite or reddit.
- ramon156 10mo agoThe update was to update it to 3MB (paid 10MB)
- AmazingTurtle 10mo agoa) They serialize tons of data into requests b) Headers. Mostly cookies. They are a thing. They are being abused all over the world by newbies.
- websiteapi 10mo agoi wonder why they cannot partially rollout. like the other outage they have to do a global rollout.
- usrnm 10mo agoI really don't see how it would've helped. In go or Rust you'd just get a panic, which is in no way different.
- denysvitali 10mo agoThe article mentions that this Lua-based proxy is the old generation one, which is going to be replaced by the Rust based one (FL2) and that didn't fail on this scenario. So, if anything, their efforts towards a typed language were justified. They just didn't manage to migrate everything in time before this incident - which is ironically a good thing since this incident was cause mostly by a rushed change in response to an actively exploited vulnerability.
- websiteapi 10mo agoyes, but as the article states why are they doing global fast rollouts?
- denysvitali 10mo agoI think (would love to be corrected) that this is the nature of their service. They probably push multiple config changes per minute to mitigate DDOS attacks. For sure the proxies have a local list of IPs that, for a period of time, are blacklisted. For DDOS protection you can't really rely on multiple-hours rollouts.
- deleted 10mo ago[deleted]
- snafeau 10mo agoA lot of these kind of bugs feel like they could be caught be a simple review bot like Greptile... I wonder if Cloudlare uses an equivalent tool internally?
- nkmnz 10mo agoWhat makes greptile a better choice compared to claude code or codex, in your opinion?
- nish__ 10mo agoAny bot that runs an AI model should not be called "simple".
- roguecoder 10mo agoThat has not been my experience with those tools. Super-procedural code in particular is too complex for humans to follow, much less AI.
- denysvitali 10mo agoIronically, this time around the issue was in the proxy they're going to phase out (and replace with the Rust one). I truly believe they're really going to make resilience their #1 priority now, and acknowledging the release process errors that they didn't acknowledge for a while (according to other HN comments) is the first step towards this. HugOps. Although bad for reputation, I think these incidents will help them shape (and prioritize!) resilience efforts more than ever. At the same time, I can't think of a company more transparent than CloudFlare when it comes to these kind of things. I also understand the urgency behind this change: CloudFlare acted (too) fast to mitigate the React vulnerability and this is the result. Say what you want, but I'd prefer to trust CloudFlare who admits and act upon their fuckups, rather than trying to cover them up or downplaying them like some other major cloud providers. @eastdakota: ignore the negative comments here, transparency is a very good strategy and this article shows a good plan to avoid further problems
- trashburger 10mo agoI would very much like for him not to ignore the negativity, given that, you know, they are breaking the entire fucking Internet every time something like this happens.
- denysvitali 10mo agoThis is the kind of comment I wish he would ignore. You can be angry - but that doesn't help anyone. They fucked up, yes, they admitted it and they provided plans on how to address that. I don't think they do these things on purpose. Of course given their good market penetration they end up disrupting a lot of customers - and they should focus on slow rollouts - but I also believe that in a DDOS protection system (or WAF) you don't want or have the luxury to wait for days until your rule is applied.
- beanjuiceII 10mo agoI hope he doesn't ignore it, the internet has been forgiving enough toward cloudflares string of failures..its getting pretty old, and creates a ton of choas. I work with life saving devices, being impacted in any way in data monitoring has a huge impact in many ways. "sorry ma'am we can't give your child t1d readings on your follow app because our provider decided to break everything in the pursuit of some react bug." has a great ring to it
- deleted 10mo ago[deleted]
- gkoz 10mo agoI sometimes feel we'd be better off without all the paternalistic kitchensink features. The solid, properly engineered features used intentionally aren't causing these outages.
- ilkkao 10mo agoAgreed, I don't really like Cloudflare trying to magically fix every web exploit there is in frameworks my site has never used.
- nish__ 10mo agoHonestly. This feels outside of their domain.
- theideaofcoffee 10mo agoI’ve been downvoted enough with my comments on this blog post where I’m hesitant to add anything else, but here I agree with you. They’re trying to be everything to everyone, where does the accountability of their customers being responsible for running, you know, up-to-date packages come in? Like, you don’t take just a little bit of pride in your work that you’re continually watching CVE lists and exploits and just have a minimum of effort toward patching your own shit, rather than pawning it off on vendor? I simply can’t understand the mindset.
- venturecruelty 10mo agoThe good news is that you can have that right now. Just don't use Cloudflare.
- da_grift_shift 10mo agoIt's not an outage, it's an Availability Incident™. https://blog.cloudflare.com/5-december-2025-outage/#what-about-the-changes-being-made-after-the-incident-on-november-18-2025 https://blog.cloudflare.com/5-december-2025-outage/#what-abo...
- perching_aix 10mo agoYou jest, but recently I also felt compelled to stop using the word (planned) outage where I work, because it legitimately creates confusion around the (expected) character of impact. Outage is the nuclear wasteland situation, which given modern architectural choices, is rather challenging to manifest. To avoid it is face-saving, but also more correct.
- aw1621107 10mo agoFrom earlier in the very same blog post (emphasis added): > This system does not perform gradual rollouts, but rather propagates changes within seconds to the entire fleet of servers in our network and is under review following the outage we experienced on November 18.
- lapcat 10mo ago> This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur. Cloudflare deployed code that was literally never tested, not even once, neither manually nor by unit test, otherwise the straightforward error would have been detected immediately, and their implied solution seems to be not testing their code when written, or even adding 100% code coverage after the fact, but rather relying on a programming language to bail them out and cover up their failure to test.
- JohnMakin 10mo agoLarge scale infrastructure changes are often by nature completely untestable. The system is too large, there are too many moving parts to replicate with any kind of sane testing, so often, you do find out in prod, which is why robust and fast rollback procedures are usually desirable and implemented.
- lapcat 10mo ago> Large scale infrastructure changes are often by nature completely untestable. You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected. Theoretical outages could theoretically have other causes, but these two specific outages had specific causes that we know. > which is why robust and fast rollback procedures are usually desirable and implemented. Yes, nobody is arguing against that. It's a red herring with regard to my point about source code testing.
- JohnMakin 10mo agoI am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well. With all due respect, it sounds like you have not worked on these types of systems, but out of curiosity - what type of test do you think would have prevented this?
- paradite 10mo agoThe deployment pattern from Cloudflare looks insane to me. I've worked at one of the top fintech firms, whenever we do a config change or deployment, we are supposed to have rollback plan ready and monitor key dashboards for 15-30 minutes. The dashboards need to be prepared beforehand on systems and key business metrics that would be affected by the deployment and reviewed by teammates. I've never seen a downtime longer than 1 minute while I was there, because you get a spike on the dashboard immediately when something goes wrong. For the entire system to be down for 10+ minutes due to a bad config change or deployment is just beyond me.
- draw_down 10mo ago[dead]
- theideaofcoffee 10mo agoSame, my time at a F100 ecommerce retailer showed me the same. Every change control board justification needed an explicit back-out/restoration plan with exact steps to be taken, what was being monitored to ensure that was being held to, contacts of prominent groups anticipated to have an effect, emergency numbers/rooms for quick conferences if in fact something did happen. The process was pretty tight, almost no revenue-affecting outages from what I can remember because it was such a collaborative effort (even though the board presentation seemed a bit spiky and confrontational at the time, everyone was working together).
- prdonahue 10mo agoAnd you moved at a glacial pace compared to Cloudflare. There are tradeoffs.
- theideaofcoffee 10mo agoYes, of course, I want the organization that inserted itself into handling 20% of the world's internet traffic to move fast and break things. Like breaking the internet on a bi-weekly basis. Yep, great tradeoff there. Give me a break.
- hrimfaxi 10mo agoHaving their changes fully propagate within 1 minute is pretty fantastic.
- chatmasta 10mo agoThe coolest part of Cloudflare’s architecture is that every server is the same… which presumably makes deployment a straightforward task.
- jamesog 10mo agoThe bad change wasn't even a deployment as such, just an entry in the global KV store https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale/ https://blog.cloudflare.com/introducing-quicksilver-configur... Actual deployments take hours to propagate worldwide. (Disclosure: former Cloudflare SRE)
- denysvitali 10mo agoThis is most likely a strong requisite for such a big scale deployment if DDOS protection and detection - which explains their architectural choices (ClickHouse & co) and the need of a super low latency config changes. Since attackers might rotate IPs more frequently than once per minute, this effectively means that the whole fleet of servers should be able to quickly react depending on the decisions done centrally.
- reassess_blind 10mo agoWhy wasn’t the rollback fixed within the second minute after they saw the 500s?
- alwaysroot 10mo ago[flagged]
- dreamcompiler 10mo ago[flagged]
- rvz 10mo ago> Instead, it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability disclosed this week in React Server Components. Doesn't Cloudflare rigorously test their changes before deployment to make sure that this does not happen again? This better not have been used to cover for the fact that they are using AI to fix issues like this one. Better not be any presence of vibe coders or AI agents being used to be touching such critical pieces of infrastructure at all and I expected Cloudflare to learn from the previous outage very quickly. But this is quite a pattern but might need to consider putting the unreliability next to GitHub (which goes down every week).
- rany_ 10mo ago> As part of our ongoing work to protect customers using React against a critical vulnerability, CVE-2025-55182, we started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications. Why would increasing the buffer size help with that security vulnerability? Is it just a performance optimization?
- boxed 10mo agoI think the buffer size is the limit on what they check for malicious data, so the old 128k would mean it would be trivial to circumvent by just having 128k ok data and then put the exploit after.
- whs 10mo agoI got curious and I checked AWS WAF. Apparently AWS WAF default limit for CloudFront is 16KB and max is 64KB.
- redslazer 10mo agoIf the request data is larger than the limit it doesn’t get processed by the Cloudflare system. By increasing buffer size they process (and therefore protect) more requests.
- _pdp_ 10mo agoSo no static compiler checks and apparently no fuzzers used to ensure these rules work as intended?
- perching_aix 10mo agoSuch tooling exists for Lua? Didn't know.
- _pdp_ 10mo agoThey exist if you are worth 70 billion.
- deleted 10mo ago[deleted]
- theoldgreybeard 10mo agoThis is total amateur shit. Completely unacceptable for something as critical as Cloudflare.
- rachr 10mo agoTime for Cloudflare to start using the BOFH excuse generator. https://bofh.d00t.org/ https://bofh.d00t.org/
- deleted 10mo ago[deleted]
- iLoveOncall 10mo agoThe most surprising from this article is that CloudFlare handles only around 85M TPS.
- liampulles 10mo agoThe lesson presented by the last few big outages is that entropy is, in fact, inescapable. The comprehensibility of a system cannot keep up with its growing and aging complexity forever. The rate of unknown unknowns will increase. The good news is that a more decentralized internet with human brain scoped components is better for innovation, progress, and freedom anyway.
- hnthrowaway0328 10mo agoI'm not sure how decentralization helps though. People in a bazzar are going to care even less about sharing shadow knowledge. Linux IMO succeeds not because of the bazaar but because of Linus.
- liampulles 10mo agoWhat is the shadow knowledge in this case?
- thenthenthen 10mo agoAlso curious, since the bazaar seems to be where one acquires shadow knowledge (grey market items, support structures for unregistered people etc.). See Chungking Mansions in Hong Kong for a practical example.
- marcosdumay 10mo agoYou don't keep a bazaar running with shadow knowledge. Either the important things are published or it doesn't run.
- venturecruelty 10mo agoDecentralization is resilience; that's why the internet even works at all. That was the entire point of it, in fact.
- agentifysh 10mo agoyet my dedicated server has been up since 2015 with zero downtimes i dont think this is an entropy issue its human error bubbling up and cloudflare charges a premium for it my faith in cloudflare is shoook for sure two major outages weeks apart ad this wont be the last
- egorfine 10mo ago> provides customers with protection against malicious payloads, allowing them to be detected and blocked. To do this, Cloudflare’s proxy buffers HTTP request body content in memory for analysis. I have a mixed feeling about this. On the other hand, I absolutely don't want a CDN to look inside my payloads and decide what's good for me or. Today it's protection, tomorrow it's censorship. At the same time this is exactly what CloudFlare is good for - to protect sites from malicious requests.
- udev4096 10mo agoWe need a decentralized ddos mitigation network based on incentives. Donate X amount of bandwidth, get Y amount of protection from other peers. Yes, we gotta do TLS inspection on every end for effective L7 mitigation but at least filtering can be done without decrypting any packets
- mewpmewp2 10mo agoHow would that work for latency / reliability of the requests?
- jgalt212 10mo agoI do kind of like who they are blaming React for this.
- deleted 10mo ago[deleted]
- lionkor 10mo agoCloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.
- markus_zhang 10mo agoTBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic. I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy a starlink console.
- roguecoder 10mo agoCostco had a deal on solid-state UPS & solar panels a while back that I was happy to partake of
- DANmode 10mo agoFar more achievable pricing and logistics than even ten years ago.
- SoftTalker 10mo agoYeah, I'd guess I average a power drop once a month or so at home. Never calculated the nines of uptime average, but it's not that infrequent. I know when I need to reset the clock on my microwave oven.
- lionkor 10mo ago99.9 is like 9 hours of downtime a year.
- ryandvm 10mo agoIndeed. AWS too. I feel like the cloud hosting companies have lost the plot. "They can provide better uptime than us" is the entire rationale that a lot of small companies have when choosing to run everything in the cloud. If they cost more AND they're less reliable, what exactly is the reason to not self host?
- 8cvor6j844qw_d6 10mo agoIs there some underlying factors that resulted in the recent outages (e.g., new processes, layoffs, etc.) or just a series of pure coincidences?
- Elucalidavah 10mo agoSounds like their "FL1 -> FL2" transition is involved in both.
- Someone1234 10mo agoIt was involved in the previous one, but not in this latest one. All FL2 did was prevent the outage being even wider spread than it was. None of this had anything to do with migration.
- tetha 10mo agoIf FL2 didn't have the outage, and FL1 did, the pace of the migration did have an impact. Though this is showing the problem with these things: Migrating faster could have reduced the impact of this outage, while increasing the impact of the last outage. Migrating slower could have reduced the impact of the last outage, while increasing the impact of this outage. This is a hard problem: How fast do you rip old working infrastructure out and risk finding new problems in the new stack, yet, how long do you tolerate shortcomings of the old stack that caused you to build the new stack?
- gernigg 10mo ago[flagged]
- venturecruelty 10mo agoI'm sure everything slowly falling apart all at the same time is due to some strange coincidence, and not the regular and steady firing of thousands of people.
- rr808 10mo agoAI
- nish__ 10mo agoNo love lost, no love found.
- jakub_g 10mo agoThe interesting part: After rolling out a bad ruleset update, they tried a killswitch (rolled out immediately to 100%) which was a code path never executed before: > However, we have never before applied a killswitch to a rule with an action of “execute”. When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn’t evaluate the sub-ruleset pointed to by it. However, an error was then encountered while processing the overall results of evaluating the ruleset > a straightforward error in the code, which had existed undetected for many years
- 8cvor6j844qw_d6 10mo ago> have never before applied a killswitch to a rule with an action of “execute” One might think a company on the scale of Cloudflare would have a suite of comprehensive tests to cover various scenarios.
- robryan 10mo agoYeah the example they gave does feel like pretty isolated unit test territory, or at least an integration test on a subset of the system that could be ran in isolation.
- hnthrowaway0328 10mo agoI kinda think most companies out there are like that. Moving fast is the motto I heard the most. They are probably OK with occasional breaks as long as customers don't mind.
- deleted 10mo ago[deleted]
- blibble 10mo agoamateur level stuff again
- nish__ 10mo agoIs it crazy to anyone else that they deploy every 5 minutes? And that it's not just config updates, but actual code changes with this "execute" action.
- kccqzy 10mo agoConfig updates are not so clear cut from code changes. Once I worked with a team in the anti-abuse space where the policy is that code deployments must happen over 5 days and config updates can take a few minutes. Then an engineer on the team argued that deploying new Python code doesn’t count as a code change because the CPython interpreter did not change; it didn’t even restart. And indeed given how dynamic Python is, it is totally possible to import new Python modules that did not exist when the interpreter process is launched.
- nish__ 10mo agolol man... If your "config" is Turing-complete, that's a code change. Full stop. Bro's just lazy.
- kccqzy 10mo agoYeah sure. Now tell the people who are designing config languages to make them both powerful and non-Turing-complete. You see, it is a very hard problem for a powerful language to be non-Turing-complete.
- roguecoder 10mo agoNo: I've been at plenty of places where we get to continuous deployment, where any given change is deployed on demand. What is wild is that they are deploying without first testing in a staging environment.
- uyzstvqs 10mo agoWhat I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately rolls the changes back. Languages with strong type systems won't save you, good procedure will.
- deleted 10mo ago[deleted]
- vouwfietsman 10mo ago> Languages with strong type systems won't save you Neither will seatbelts if you drive into the ocean, or helmets if you drink poison. I'm not sure what your point is.
- djmips 10mo agoI think you strengthened their point.
- vouwfietsman 10mo agoI don't think I did. The implication is that using languages with strong types (as discussed in the article) is not a solution. That's rubbish. It's at least part of the solution.
- bombcar 10mo agoThey have millions of “free” subscribers; said subscribers should be the test pigs for rollouts; paying (read: big) subscribers can get the breaking changes later.
- beardedetim 10mo agoThis feels like such a valid solution and is how past $dayjobs released things: send to the free users, rollout to Paying Users once that's proven to not blow up.
- Bender 10mo agoSuggestion for Cloudflare: Create an early adopter option for free accounts. Benefit: Earliest uptake of new features and security patches. Drawback: Higher risk of outages. I think this should be possible since they already differentiate between free, pro and enterprise accounts. I do not know how the routing for that works but I bet they could do this. Think crowd-sourced beta testers. Also a perk for anything PCI audit or FEDRAMP security prioritized over uptime.
- LelouBil 10mo agoI would for sure enable this, my personal server can handle being unreachable for a few hours in exchange for (potentially) interesting features.
- ectospheno 10mo agoIf that meant free tier had WAF then sure, I’d enable that.
- rfmoz 10mo agoThey do in some way because the LaLiga blocking problems in Spain don’t affect the paid accounts=large websites. An other suggestion is to do it along night shift in every country, right now they only take into account EEUU night.
- deleted 10mo ago[deleted]
- kosolam 10mo agoSome nonsense again. The level of negligence there is astounding. This is frightening because this entity is daily exposed to a large portion of our personal data which goes over the wire. As well as business data. It’s just a matter of time before a disaster is going to occur. Some regulatory body must take control in their hands right now.
- kosolam 10mo agoGot furiously downvoted but no counter arguments presented. Says it all.
- cpncrunch 10mo agoI've noticed that in recent months, even apart from these outages, cloudflare has been contributing to a general degradation and shittification of the internet. I'm seeing a lot more "prove you're human", "checking to make sure you're human", and there is normally at the very least a delay of a few seconds before the site loads. I don't think this is really helping the site owners. I suspect it's mainly about AI extortion: https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/
- NooneAtAll3 10mo agoit can't even spy on us silently, damn
- james2doyle 10mo agoYou call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? I would say Cloudflare is giving these site owners an option to protect their content and as a byproduct, reduce their own costs of subsidizing their thieves. They can choose to turn off the crawl protection. If they aren't, that tells you that they want it, doesn’t it?
- cpncrunch 10mo ago>You call it extortion of the AI companies, but isn’t stealing/crawling/hammering a site to scrape their content to resell just as nefarious? You can easily block ChatGPT and most other AI scrapers if you want: https://habeasdata.neocities.org/ai-bots https://habeasdata.neocities.org/ai-bots
- james2doyle 10mo agoThis is just using robots.txt and asking "pretty please, don’t scrape me". Here is an article (from TODAY) about the case where Perplexity is being accused of ignoring robots.txt: https://www.theverge.com/news/839006/new-york-times-perplexity-lawsuit-copyright https://www.theverge.com/news/839006/new-york-times-perplexi... If you think a robots.txt is the answer to stopping the billion-dollar AI machine from scraping you, I don’t know what to say.
- deleted 10mo ago[deleted]
- markus_zhang 10mo agoI wonder anyone from internal could share the culture a bit. I'm mostly interested in the following part: If someone messes up royally, is there someone who says "if you break the build/whatever super critical, then your ass is the grass and I'm the lawn mower"?
- guluarte 10mo agois it me or critical software bugs are more and more common?
- w10-1 10mo agoKudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineering. It's just that I think of CloudFlare as an infrastructure place, where that shouldn't be true. I had a manager who came from defense electronics in the 1980's. He said in that context, the quality engineering team was always in charge, and always more skilled. For him, software is backwards.
- braiamp 10mo agoThis is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.
- mopsi 10mo ago... until the extended-range version is ordered and no one remembers to fix the leak. :]
- wizzwizz4 10mo agoThey will remember, because it'll have been measured and documented, rigorously.
- SketchySeaBeast 10mo agoI've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.
- lloeki 10mo ago
- system2 10mo agoIs that me, or did CloudFlare outages increase since LLM "engineers" were hired remotely? Do you think there is a correlation?
- roguecoder 10mo agoThey've always been flakey. At least these only impacted their own customers instead of taking down the internet.
- MagicMoonlight 10mo agoIf you had a 99.99% availability requirement they will have already cost you a fortune
- rudedogg 10mo agoI’m really sick of constantly seeing cloudflare, and their bullshit captchas. Please, look at how much grief they’re causing trying to be the gateway to the internet. Don’t give them this power
- aeyes 10mo agoHow hard can it be for a company with 1000 engineers to create a canary region before blasting their centralized changes out to everyone. Every change is a deployment, even if its config. Treat it as such. Also you should know that a strongly typed language won't save you from every type of problem. And especially not if you allow things like unwrap(). It is just mind boggling that they very obviously have completely untested code which proxies requests for all their customers. If you don't want to write the tests then at least fuzz it.
- Uptrenda 10mo agoCan't believe one shitty website can take down most of the mainstream web.
- deleted 10mo ago[deleted]
- dznodes 10mo agoWhen should we just give up on Cloudflare? Seems like this just keeps happening. Like some kind of backdoor triggered willy nilly, Hmmm?
- venturecruelty 10mo agoNow. Right now. Seriously, stop using this terrible service. We also need to change the narrative that step 1 in every tutorial is "sign up for Cloudflare". This is partly a culture problem.
- bradly 10mo agoDang… I don’t even use React and it still brings down my sites. Good beats I guess.
- roguecoder 10mo agoI notice that this is the kind of thing that solid sociable tests ought to have caught. I am very curious how testable that code is (random procedural if-statements don't inspire high confidence.)
- borplk 10mo agoEvery time they screw up they write an elaborate postmortem and pat themselves on the back. Don't get me wrong, better have the postmortem than not. But at this point it seems like the only thing they are good at is writing incident postmortem blog posts.
- jchip303 10mo ago[dead]
- stego-tech 10mo agoThe problem that irks me isn’t that Cloudflare is having outages (everyone does and will at some point, no matter how many 9’s your SLA states), it’s that the internet is so damn centralized that a Cloudflare issue can take out a continent-sized chunk of the internet. Kudos to them on their success story, but oh my god that’s way too many eggs in one basket in general.
- ferat 10mo agoToday, after the Cloudflare outage, I noticed that almost all upload routes for my applications were being blocked. After some investigation, I realized that none of these routes passed through Cloudflare OWASP. The reported anomalies total 50, exceeding the pre-configured maximum of 40 (Medium). Despite being simple image or video uploads, the WAF is generating anomalies that make no sense, such as the following: Cloudflare OWASP Core Ruleset Score (+5) 933100: PHP Injection Attack: PHP Open Tag Found Cloudflare OWASP Core Ruleset Score (+5) 933180: PHP Injection Attack: Variable Function Call Found For now, I’ve had to raise the OWASP Anomaly Score Threshold to 60 and enable the JS Challenge, but I believe something is wrong with the WAF after today’s outage. This issue was still not solved to this moment.
- jacobgkau 10mo agoI noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.
- deleted 10mo ago[deleted]
- cnnlives265 10mo agoThis is the case with just about every status page I’ve ever seen. It takes them a while to realize there’s really a problem and then to update the page. One day these things will be automated, but until then, I wouldn’t expect more of Cloudflare than any other provider. What’s more concerning to me is that now we’ve had AWS, Azure, and CloudFlare (and CliudFlare twice) go down recently. My gut says: 1. developers and IT are using LLMs in some part of the process, which will not be 100% reliable. 2. Current culture of I have (some personal activity or problem) or we don’t have staff, AI will replace me, f-this. 3. Pandemic after effects. 4. Political climate / war / drugs; all are intermingled.
- mikkupikku 10mo agoManagement doesn't like when things like this are automated. They want to "manage" the outage/production/etc numbers before letting them out.
- Yeri 10mo ago100% — will never be automated :)
- hnuser123456 10mo agoStill room for someone to claim the niche of the Porsche horsepower method in outage reporting - underpromise, overdeliver.
- away0x01ct 10mo ago1.1.1.1 domain test server, whether a relay or endpoints including /cdn-cgi/trace is WAF testing error, for 500 HTTP network & Cloudflare managed R-W-X permissions
- AtNightWeCode 10mo agoNot missing working with LUA in proxies. I think this is no big thing. They rolled back the change fairly quickly. Still bad but that outage mid November was worse since it was many bad decisions stacking up and it took too long time to resolve.
- dwa3592 10mo agoI am not sure if it's just me or there have been too many outages this year to count. Is it the AI slop making into production?
- jokoon 10mo agoI still don't understand what is cloudflare's business model, yet they manage to make news. I don't see how their main product is ddos protection, yet cloudflare goes down for some reason. This company makes zero sense to me.
- OneDeuxTriSeiGo 10mo agoCloudflare protects against DDOS but also various forms of malicious traffic (bots, low reputation IP users, etc) and often with a DDOS or similar attacks, it's better to have the site go down from time to time than for the attackers to hammer the servers behind cloudflare and waste mass amounts of resources. i.e. it's the difference between "site goes down for a few hours every few months" and "an attacker slammed your site, and through in on-demand scaling or serverless component cloud fees blew your entire infrastructure budget for the year. Doubly so when your service is part of a larger platform and attacks on your service risk harming your reputation for the larger platform.
- j45 10mo agoCurious if there isn't a way to ingest the incoming traffic at scale, but route it to a secondary infrastructure to make sure it's resolving correctly, before pushing it to production?
- seanparsons 10mo ago"This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur." It's starting to sound like a broken record at this point, languages are still seen as equal and as a result, interchangeable.
- arjie 10mo agoClassic. Things always get worse before they get better. I remember when Netflix was going through their annus horribilis, and AWS before that, and Twitter before that, and so on. Everyone goes through this. Good luck to you guys getting to FL2 quickly enough that this class of error reduces.
- rubatuga 10mo agoHonestly a lot of these problems are because they don't test a staging environment, like isn't this software engineering basics?
- mixedbit 10mo agoThis is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outages. Rust won't help, people will always make mistakes, also in Rust. Robust architecture addresses this by not allowing a single mistake to bring down myriad of unrelated services at once.
- WD-42 10mo agoIn other words, the consolidation on Cloudflare and AWS makes the web less stable. I agree.
- amazingman 10mo agoUsually I am allergic to pithy, vaguely dogmatic summaries like this but you're right. We have traded "some sites are down some of the time" for "most sites are down some of the time". Sure the "some" is eliding an order of magnitude or two, but this framing remains directionally correct.
- PullJosh 10mo agoDoes relying on larger players result in better overall uptime for smaller players? AWS is providing me better uptime than if I assembled something myself because I am less resourced and less talented than that massive team. If so, is it a good or bad trade to have more overall uptime but when things go down it all goes down together?
- VorpalWay 10mo agoFrom a societal view it is worse when everything is down at once. Leads to a less resilient society: It is not great if I can't buy essentials from one store because their payment system is down (this happened to one super market chain in Sweden due to a hacker attack some years ago, took weeks to fully fix everything, and then there was that whole Crowdstrike debacle globally more recently). It is far worse if all of the competitors are down at once. To some extent you can and should have a little bit of stock at home (water, food, medicine, ways to stay warm, etc) but not everything is practical to do so with (gasoline for example, which could have knock on effects on delivery of other goods).
- RA_Fisher 10mo agoAs a reliability statistician (and web user!), I'd love to see Cloudflare investing in reliability statistics. :)
- teleforce 10mo agoInternet packet switching based architecture was originally design to withstand this type of outages [1]. Some people even go further by speculating that the original military DARPA network precursor to the modern Internet was originally designed to ensure the continuity of command and control (C&C) of the US military operation in the potential event of all out nuclear attack during the Cold War. This the time when Internet researchers need to redefine the Internet application and operation. The local-first paradigm is the first step in the right direction (pardon the pun) [2]. [1] The Real Internet Architecture: Past, Present, and Future Evolution: https://press.princeton.edu/books/paperback/9780691255804/the-real-internet-architecture https://press.princeton.edu/books/paperback/9780691255804/th... [2] Local-first software You own your data, in spite of the cloud: https://www.inkandswitch.com/essay/local-first/ https://www.inkandswitch.com/essay/local-first/
- scottlamb 10mo agoThere's a lot of bad karma in this discussion. It's hard to run large services. Careful when you set a precedent of pillorying after an outage. It could be you next! Yes, this is the second time in a month. Were folks expecting that to have been enough time for them to have made sweeping technical and organization changes? I say no—this doesn't mean they aren't trying or haven't learned any lessons from the last outage. It's a bit too soon to say that. I see this event primarily as another example of the #1 class of major outages: bad rapid global configuration change. (The last CloudFlare outage was too, but I'm not just talking about CloudFlare. Google has had many many such outages. There was an inexplicable multi-year gap between recognizing this and having a good, widely available staged config rollout system for teams to drop into their systems.) Stuff like DoS attack configurations needs to roll out globally quickly. But they really need make it not quite this quick. Imagine they deployed to one server for one minute, one region for one minute on success, then everywhere on success. Then this would have been a tiny blip rather than a huge deal. (It can be a bit hard to define "success" when you're doing something like blocking bad requests that may even be a majority of traffic during a DDoS attack, but noticing 100% 5xx errors for 38% of your users due to a parsing bug is doable!) As for the specific bug: meh. They should have had 100% branch coverage on something as critical (and likely small) as the parsing for this config. Arguably a statically typed language would have helped (but the `.unwrap()` error in the previous outage is a bit of a counterargument to that). But it just wouldn't have mattered that much if they caught it before global rollout.
- rurban 10mo agoOh oh, looks like agentzh's code. A living legend
- resonious 10mo ago> This type of code error is prevented by languages with strong type systems. True, as long as you don't call unwrap!
- IshKebab 10mo agoThat's a different kind of error. And even then unwrap is opt-in whereas this is opt-out if you're lucky. Kind of funny that we get something showing the benefits of Rust so soon after everyone was ragging on a out unwrap anyway!
- mr_windfrog 10mo agoIf I'm remembering correctly, there was another outage around 10 days ago. It still surprises me that there are basically no free alternatives comparable to Cloudflare. Putting everything on CF creates a pretty serious single point of failure. It's strange that in most industries you have at least two major players, like Coke vs. Pepsi or Nike vs. Adidas. But in the CDN/edge space, there doesn't seem to be a real free competitor that matches Cloudflare's feature set. It feels very unhealthy for the ecosystem. Does anyone know why this is the case?
- halapro 10mo agoI reckon AWS is "free enough" for most of its users, but it's not as easy nor as safe for the common user.
- mr_windfrog 10mo agoTotally agree, AWS's free tier is great for many users, but it can definitely be tricky and risky for the average person.
- qouteall 10mo agoIt's (at least) the second time Couldflage gets bitten by React. Last time an useEffect caused an incident. https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-12-dashboard-and-api-outage/ https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...
- blinded 10mo agoThey bypassed the gradual rollout system in order to meet a deadline for a cve. They put security above availability, tough tradeoff. Is there a non prod environment where that one off waf testing tool change could have been tested?
- egnehots 10mo agoFrom a customer perspective, I think there should be an option: - prioritize security: get patchs ASAP - prioritize availability: get patchs after a cooldown period Because ultimately, it's a tradeoff that cannot be handled by Cloudflare. It depends on your business, your threat model.
- eviks 10mo ago> This first change was being rolled out using our gradual deployment system. So they are aware of some basic mitigation tactics guarding against errors > This system does not perform gradual rollouts, They just choose to YOLO > Typical actions are “block”, “log”, or “skip”. Another type of action is “execute”, > However, we have never before applied a killswitch to a rule with an action of “execute”. Do they do no testing? These isn't even fuzzing with “infinite” variations, but a limited list of actions > existed undetected for many years. This type of code error is prevented by languages with strong type systems. So this solution is also well known, just ignored for years, because "if it’s not broken, don’t fix it?", right?
- gotekom952 10mo ago"They let the internet down"
- kune 10mo agoThe interesting aspect of the Cloudflare support, which is not clarified, is how they came to the risk assessment that it is ok to roll out a change non-gradual globally without testing the procedure first. The only justification I can see is that the React/Next.js remote command execution vulnerabilities are actively exploited. But if this is the case they should say so.
- sandos 10mo agoIs it just me, or should they have just reverted instead of making _another_ change as a result of the first one? ALSO, very very weird that they had not caught this seemingly obvious bug in proxy buffer size handling. This points to that the change nr 2, done in "reactive" mode to change nr 1 that broke shit, HAD NOT BEEN TESTED AT ALL! Which is the core reason they should never have deployed that, but rather revert to a known good state, then test BOTH changes combined.
- lofaszvanitt 10mo agoHow come that crumbling, rotten to the core lua is still used? :D
- ken47 10mo agoI have to wonder if there is a relation to the rising prevalence of coding LLMs.
- dev1ycan 10mo agoStop vibe coding on critical infrastructure :)
- Smalltalker-80 10mo agoSo Cloudflare: - Did a last minute, untested change to their change: "turning off our WAF rule testing tool". - Did an immediate global roll-out, instead of a staged one. . Is seems they would have enough leaning-cases now never to do that again...
- mxpxrocks10 10mo agoFirst, what Cloudflare does is hard and I want to start with that. That being said, I think it’s worth a discussion. How much of the last 3 outages were because of the JGC (the former CTO) retiring and Dane taking over? Did JGC have a steady hand that’s missing? Or was it just time for outages that would have happened anyway? Dane has maintained a culture of transparency which is fantastic, but did something get injected in the culture leading towards these issues? Will it become more or less stable since JGC left? Curious for anyone with some insight or opinions. (Also, if it wasn’t clear - huge Cloudflare fan and sending lots of good vibes to the team)
- cinericius 10mo agoLooking at Dane's career history on LinkedIn, it appears that he has only ever been in product and some variety of manager, and his degree is in 'Engineering Management System'. It's an odd choice given that the previous two CTOs (Lee and John) were extremely technical and how core technology is to Cloudflare. As with any organisation where the CTO is not technical, there will be someone who the 'CTO' has to ask to understand technical situations. In my opinion, that person being asked is the real CTO, for any given situation.
- dzonga 10mo agobefore Cloudflare suffered an outage due to React's useEffect, now again trying to mitigate security issues around React Server Pages. at one point in time - they've to admit this react thing ain't working. & just use classic server rendered pages, since their dashboards are simple toggle controls
- 0x9e3779b6 10mo agoApart from Cloudflare config system working too good to propagate failure modes: the code quality on very mission critical path powering “half the internets” could’ve been better. I’m not sure if Lua LSP / linting tools would’ve caught the issue (I also never used Lua myself), but tools and methods exist to test mission critical dynamically typed code. The company with genuinely impressive concentration of talent was expected to think about fuzzing this legacy crap somehow. As for `.unwrap()` related incident: normally code like this should never pass the review. You just (almost) never unwrap in production code. I’d start with code quality tooling but more important - the related processes before even thinking about the architecture changes. Changing architecture in global sense which, in general has served for years with 99.99(9)% uptime is not obviously smart thing to do. Architecture is doing great, it’s just impact which has been devastating because of the scale. Everyone makes errors and it’s fine, but there are ways not to roll shit in prod (reference to famous meme pic where bugs do that).