30 ms·

Cloudflare outage on November 18, 2025 post mortem

• 1970-01-01 11mo ago
  I would have been a bit cheeky and opened with 'It wasn't DNS.'
• chatmasta 11mo ago
  Wow, crazy disproportional drop in the stock price… good buying opportunity for $NET.
  • rvz 11mo ago
    Agree. Cloudflare is very cheap at these prices.
• nawgz 11mo ago
  > a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats > The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail A configuration error can cause internet-scale outages. What an era we live in Edit: also, after finishing my reading, I have to express some surprise that this type of error wasn't caught in a staging environment. If the entire error is that "during migration of ClickHouse nodes, the migration -> query -> configuration file pipeline caused configuration files to become illegally large", it seems intuitive to me that doing this same migration in staging would have identified this exact error, no? I'm not big on distributed systems by any means, so maybe I'm overly naive, but frankly posting a faulty Rust code snippet that was unwrapping an error value without checking for the error didn't inspire confidence for me!
  • jmclnx 11mo ago
    I have to wonder if AI was involved with the change.
    • norskeld 11mo ago
      I don't think this is the case with CloudFlare, but for every recent GitHub outage or performance issue... oh boy, I blame the clankers!
  • mewpmewp2 11mo ago
    It would have been caught only in stage if there was similar amount of data in the database. If stage has 2x less data it would have never occurred there. Not super clear how easy it would have been to keep stage database exactly as production database in terms of quantity and similarity of data etc. I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.
    • Aeolun 11mo ago
      > I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod. We’re like a millionth the size of cloudflare and we have automated tests for all (sort of) queries to see what would happen with 20x more data. Mostly to catch performance regressions, but it would work to catch these issues too. I guess that doesn’t say anything about how rare it is, because this is also the first company at which I get the time to go to such lengths.
      • mewpmewp2 11mo ago
        But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day. In this case it seems the database table in question seemed modest in size (the features for ML) so naively thinking they could have kept stage features always in sync with prod at the very least, but could be they didn't consider that 55 rows vs 60 rows or similar could be a breaking point given a certain specific bug. It is much easier to test with 20x data if you don't have the amount of data cloudflare probably handles.
• binarymax 11mo ago
  28M 500 errors/sec for several hours from a single provider. Must be a new record. No other time in history has one single company been responsible for so much commerce and traffic. I wonder what some outage analogs to the pre-internet ages would be.
  • captainkrtek 11mo ago
    Something like a major telco going out, for example the AT&T 1990 outage of long distance calling: > The standard procedures the managers tried first failed to bring the network back up to speed and for nine hours, while engineers raced to stabilize the network, almost 50% of the calls placed through AT&T failed to go through. > Until 11:30pm, when network loads were low enough to allow the system to stabilize, AT&T alone lost more than $60 million in unconnected calls. > Still unknown is the amount of business lost by airline reservations systems, hotels, rental car agencies and other businesses that relied on the telephone network. https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...
  • adventured 11mo ago
    > No other time in history has one single company been responsible for so much commerce and traffic. AWS very likely has Cloudflare beat in commerce responsibility. Amazon is equal to ~2.3% of US GDP by itself.
  • manquer 11mo ago
    Absolute volume maybe[1], as relative % of global digital communication traffic, the era of early telegraph probably has it beat. In the pre digital era, East India Company dwarfs every other company in any metric like commerce controlled, global shipping, communication traffic, private army size, %GDP , % of workforce employed by considerable margins. The default was large consolidated organization throughout history, like say Bell Labs, or Standard Oil before that and so on, only for a brief periods we have enjoyed benefits of true capitalism. [1] Although I suspect either AWS or MS/Azure recent down-times in the last couple of years are likely higher
  • nullbyte808 11mo ago
• SerCe 11mo ago
  As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.
  • bayesnet 11mo ago
    And a well-written one at that. Compared to the AWS port-mortem this could be literature.
    • hn-idiots 11mo ago
      [flagged]
      • tclancy 11mo ago
        I feel like your username really brings something extra to the party. Now go home.
        • eastdakota 11mo ago
          Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.
          • yahoozoo 11mo ago
            Based CEO
            • hn-idiots 11mo ago
              [flagged]
    • philipwhiuk 11mo ago
      Except it fails to document anything about the actions they made to Warp in London during the resolution.
• 0xbadcafebee 11mo ago
  So, to recap: - Their database permissions changed unexpectedly (??) - This caused a 'feature file' to be changed in an unusual way (?!) - Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query - Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly) - They hit an internal application memory limit and that just... crashed the app - The crashing did not result in an automatic backout of the change, meaning their deployments aren't blue/green or progressive - After fixing it, they were vulnerable to a thundering herd problem - Customers who were not using bot rules were not affected; CloudFlare's bot-scorer generated a constant bot score of 0, meaning all traffic is bots In terms of preventing this from a software engineering perspective, they made assumptions about how their database queries work (and didn't validate the results), and they ignored their own application limits and didn't program in either a test for whether an input would hit a limit, or some kind of alarm to notify the engineers of the source of the problem. From an operations perspective, it would appear they didn't test this on a non-production system mimicing production; they then didn't have a progressive deployment; and they didn't have a circuit breaker to stop the deployment or roll-back when a newly deployed app started crashing.
  • tptacek 11mo ago
    People jump to say things like "where's the rollback" and, like, probably yeah, but keep in mind that speculative rollback features (that is: rollbacks built before you've experienced the real error modes of the system) are themselves sources of sometimes-metastable distributed system failures. None of this is easy.
    • 0xbadcafebee 11mo ago
      How about where's the most basic test to check if your config file will actually run at all in your application? It was a hard-coded memory limit; a git-hook test suite run a MacBook would have caught this. But nooo, let's not run the app for 0.01 seconds with this config before sending it out to determine the fate of the internet? This is literally the CrowdStrike bug, in a CDN. This is the most basic, elementary, day 0 test you could possibly invent. Forget the other things they fucked up. Their app just crashes with a config file, and nobody evaluates it?! Not every bug is preventable, but an egregious lack of testing is preventable. This is what a software building code (like the electrical code's UL listings that prevent your house from burning down from untested electrical components) is intended to prevent. No critical infrastructure should be legal without testing, period.
• deleted 11mo ago
  [deleted]
• rawgabbit 11mo ago
  > The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database: SELECT name, type FROM system.columns WHERE table = 'http_requests_features' order by name; Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning “duplicates” of columns because those were for underlying tables stored in the r0 database.
  • rawgabbit 11mo ago
    Here is a bit more context in addition to the quote above. A ClickHouse permissions change made a metadata query start returning duplicate column metadata from an extra schema, which more than doubled the size and feature count of a Bot Management configuration file. When this oversized feature file was deployed to edge proxies, it exceeded a 200-feature limit in the bot module, causing that module to panic and the core proxy to return 5xx errors globally
• zzzeek 11mo ago
  > Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. And here is the query they used ** (OK, so it's not exactly): SELECT * from feature JOIN permissions on feature.feature_type_id = permissions.feature_type_id someone added a new row to permissions and the JOIN started returning two dupe feature rows for each distinct feature. ** "here is the query" is used for dramatic effect. I have no knowledge of what kind of database they are even using much less queries (but i do have an idea). more edits: OK apparently it's described later in the post as a query against clickhouse's table metadata table, and because users were granted access to an additional database that was actually the backing store to the one they normally worked with, some row level security type of thing doubled up the rows. Not sure why querying system.columns is part of a production level query though, seems overly dynamic.
  • captainkrtek 11mo ago
    I believe they mentioned ClickHouse
• gucci-on-fleek 11mo ago
  > This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions Also appreciate the honesty here. > On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. […] > Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal. Why did this take so long to resolve? I read through the entire article, and I understand why the outage happened, but when most of the network goes down, why wasn't the first step to revert any recent configuration changes, even ones that seem unrelated to the outage? (Or did I just misread something and this was explained somewhere?) Of course, the correct solution is always obvious in retrospect, and it's impressive that it only took 7 minutes between the start of the outage and the incident being investigated, but it taking a further 4 hours to resolve the problem and 8 hours total for everything to be back to normal isn't great.
  • eastdakota 11mo ago
    Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
    • tptacek 11mo ago
      Richard Cook #18 (and #10) strikes again! https://how.complexsystems.fail/#18 https://how.complexsystems.fail/#18 It'd be fun to read more about how you all procedurally respond to this (but maybe this is just a fixation of mine lately). Like are you tabletopping this scenario, are teams building out runbooks for how to quickly resolve this, what's the balancing test for "this needs a functional change to how our distributed systems work" vs. "instead of layering additional complexity on, we should just have a process for quickly and maybe even speculatively restoring this part of the system to a known good state in an outage".
• EvanAnderson 11mo ago
  It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it. The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but today's outage highlights that rapid deployment isn't all upside. The remediation section doesn't give me any sense that phased deployment, acceptance testing, and rapid rollback are part of the planned remediation strategy.
  • tptacek 11mo ago
    I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).
    • EvanAnderson 11mo ago
      That's why I likened it Crowdstrike. It's a signature database that blew up the consumer of said database. (You probably caught my post mid-edit, too. You may be replying to the snarky paragraph I felt better of and removed.) Edit: Similar to Crowdstrike, the bot detector should have fallen-back to its last-known-good signature database after panicking, instead of just continuing to panic.
    • eastdakota 11mo ago
      That’s correct.
      • tptacek 11mo ago
        Is it actually consul-template? (I have post-consul-template stress disorder).
        • threatofrain 11mo ago
• tristan-morris 11mo ago
  Why call .unwrap() in a function which returns Result<_,_>? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
  • sayrer 11mo ago
    Yes, can't have .unwrap() in production code (it's ok in tests)
    • orphea 11mo ago
      Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.
      • gishh 11mo ago
        To be fair, if you’re not “this tall” you really shouldn’t consider using goto in a c program. Most people aren’t that tall.
        • tucnak 11mo ago
          Nonsense. Linux kernel for one example, uses goto everywhere for error handling.
          • gishh 11mo ago
            How many codebases besides the Linux kernel had you seen an intentional amount of use of goto?
            • orphea 10mo ago
              The answer is one search away on GitHub. I will give you one example though: various .NET repositories (runtime, aspnetcore, orleans).
• otterley 11mo ago
  > work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditions across all core proxy modules Absent from this list are canary deployments and incremental or wave-based deployment of configuration files (which are often as dangerous as code changes) across fault isolation boundaries -- assuming CloudFlare has such boundaries at all. How are they going to contain the blast radius in the future? This is something the industry was supposed to learn from the CrowdStrike incident last year, but it's clear that we still have a long way to go. Also, enabling global anything (i.e., "enabling global kill switches for features") sounds like an incredibly risky idea. One can imagine a bug in a global switch that transforms disabling a feature into disabling an entire system.
  • nikcub 11mo ago
    They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy. I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more suited, but i'm sure they have their reasons [0] https://clickhouse.com/docs/guides/developer/deduplication https://clickhouse.com/docs/guides/developer/deduplication
    • HumanOstrich 11mo ago
      I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue. Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.
      • hedora 11mo ago
• lukan 11mo ago
  "Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page." Unfortunately they do not share, what caused the status page to went down as well. (Does this happen often? Otherwise a big coincidence it seems)
  • Aeolun 11mo ago
    I mean, that would require a postmortem from statuspage.io right? Is that a service operated by cloudflare?
    • verletzen 11mo ago
      [dead]
    • edoceo 11mo ago
      Atlasaian
  • eastdakota 11mo ago
    We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.
    • dnw 11mo ago
      Yes, probably a bunch of automated bots decided to check the status page when they saw failures in production.
    • reassess_blind 11mo ago
      The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.
• vsgherzi 11mo ago
  Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type. At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect"). The whole point of errors as values is preventing this kind of thing.... It wouldn't have stopped the outage but it would've made it easy to diagnose. If anyone at cloudflare is here please let me in that codebase :)
  • waterTanuki 11mo ago
    Not a cloudflare employee but I do write a lot of Rust. The amount of things that can go wrong with any code that needs to make a network call is staggeringly high. unwrap() is normal during development phase but there are a number of times I leave an expect() for production because sometimes there's no way to move forward.
    • vsgherzi 11mo ago
      I'm in a similar boat, at the very leas an expect can give hits to what happened. However this can also be problematic if your a library developer. Sometimes rust is expected to never panic especially in situations like WASM. This is a major problem for companies like Amazon Prime Video since they run in a WASM context for their TV APP. Any panic crashes everything. Personally I usually just either create a custom error type (preferred) or erase it away with Dyn Box Error (no other option). Random unwraps and expects haunt my dreams.
    • SchemaLoad 11mo ago
      Yeah it seems likely that even if there wasn't an unwrap, there would have been some error handling that wouldn't have panicked the process, but would have still left it inoperable if every request was instead going through an error path.
    • frumplestlatz 11mo ago
      At risk of sounding harsh, that’s a huge failure in your modeling of invariants that should not be permitted in development. Permitting it in development is why one ends up in the position of having to use an `expect()` in production code, because your API surfaces are wrong and can’t model your actual invariants.
• ed_mercer 11mo ago
  Wow. 26M/s 5xx error HTTP status codes over a span of roughly two hours. That's roughly 187 billion HTTP errors that interrupted people (and systems)!
  • watchful_moose 11mo ago
    Some of these would be retries that wouldn't have happened if not for earlier errors.
• moralestapia 11mo ago
  No publicity is bad publicity. Best post mortem I've read in a while, this thing will be studied for years. A bit ironic that their internal FL2 tool is supposed to make Cloudflare "faster and more secure" but brought a lot of things down. And yeah, as other have already pointed out, that's a very unsafe use of Rust, should've never made it to production.
• sigmar 11mo ago
  Wow. What a post mortem. Rather than Monday morning quarterbacking how many ways this could have been prevented, I'd love to hear people sound-off on things that unexpectedly broke. I, for one, did not realize logging in to porkbun to edit DNS settings would become impossible with a cloudflare meltdown
  • brandon272 11mo ago
    That's unfortunate. I'll need to investigate whether Porkbun plans on decoupling its auth from being reliant on CloudFlare, otherwise I will need to migrate a few domains off of that registrar.
• deleted 11mo ago
  [deleted]
• ojosilva 11mo ago
  This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is textbook "parse, don't validate" anti-pattern. I know, this is "Monday morning quarterbacking", but that's what you get for an outage this big that had me tied up for half a day.
  • wrs 11mo ago
    It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.
    • dist1ll 11mo ago
      > every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this. How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.
      • danielheath 11mo ago
        I mean... yeah, in general. That's what iterators are for.
      • tux3 11mo ago
        Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style. For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not. You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid manual iteration (not just for safety, but because it lets the compiler eliminate bounds checks), so it would actually be quite viable to write these comments, since you only need them when you're doing something unusual.
• trengrj 11mo ago
  Classic combination of errors: Having the feature table pivoted (with 200 feature1, feature2, etc columns) meant they had to do meta queries to system.columns to get all the feature columns which made the query sensitive to permissioning changes (especially duplicate databases). A Crowdstrike style config update that affects all nodes but obviously isn't tested in any QA or staged rollout strategy beforehand (the application panicking straight away with this new file basically proves this). Finally an error with bot management config files should probably disable bot management vs crash the core proxy. I'm interested here why they even decided to name Clickhouse as this error could have been caused by any other database. I can see though the replicas updating causing flip / flopping of results would have been really frustrating for incident responders.
  • tptacek 11mo ago
    Right but also this is a pretty common pattern in distributed systems that publish from databases (really any large central source of truth); it might be like the problem in systems like this. When you're lucky the corner cases are obvious; in the big one we experienced last year, a new row in our database tripped an if-let/mutex deadlock, which our system dutifully (and very quickly) propagated across our entire network. The solution to that problem wasn't better testing of database permutations or a better staging environment (though in time we did do those things). It was (1) a watchdog system in our proxies to catch arbitrary deadlocks (which caught other stuff later), (2) segmenting our global broadcast domain for changes into regional broadcast domains so prod rollouts are implicitly staged, and (3) a process for operators to quickly restore that system to a known good state in the early stages of an outage. (Cloudflare's responses will be different than ours, really I'm just sticking up for the idea that the changes you need don't follow obviously from the immediate facts of an outage.)
• nullbyte808 11mo ago
  I thought it was an internal mess-up. I thought an employee screwed a file up. Old methods are sometimes better than new. AI fails us again!
• ksajadi 11mo ago
  May I just say that Matthew Prince is the CEO of Cloudflare and a lawyer by training (and a very nice guy overall). The quality of this postmortem is great but the fact that it is from him makes one respect the company even more.
• deleted 11mo ago
  [deleted]
• dzonga 11mo ago
  > thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually wanna try it for industrial robot stuff or small controllers etc. there's no bad language - just occassional hiccups from us users who use those tools.
  • dzonga 11mo ago
    other people might say - why use unsafe rust - but we don't know the conditions of what the original code shipped under. why the pr was approved. could have been tight deadline, managerial pressure or just the occasional slip up.
  • tptacek 11mo ago
    What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.
    • olivia-banks 11mo ago
      What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than bot mitigation you'd still have the same symptom of getting 500'd.
      • tptacek 11mo ago
        Yeah, see, that's what I mean.
      • 11mo ago
• deleted 11mo ago
  [deleted]
• rvz 11mo ago
  Great write up. This is the first significant outage that has involved Rust code, and as you can see the .unwrap is known to carry the risk of a panic and should never be used on production code.
• nanankcornering 11mo ago
  Matt, Looking forward in regaining Elon's and his team trust to use CF again.
  • wilg 11mo ago
    I wish Elon would regain my trust!
• RagingCactus 11mo ago
  Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate. Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as the downtime starts. And the well defined clean crashes should in theory also stand out against all the random errors that start occuring all over the system as it begins to go down, precisely because it's always failing at the exact same point.
  • rixed 11mo ago
    Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire. The issue here is about the system as a whole not any line of code.
    • frumplestlatz 11mo ago
      > The issue here is about the system as a whole not any line of code. Unsoundness in the type system that leads to a systemic failure is about the system as a whole. Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.
      • rixed 11mo ago
        In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..." People here chatting about unwrap remind me of them :)
        • frumplestlatz 11mo ago
          Assuming software and people will fail is exactly what not using unwrap is about. If you depend on engineers not fucking up, you will fail. Using unwrap is assuming humans won’t get human-enforced invariants wrong. They will. They did here. As someone that works in formal verification of crypto systems, watching people like yourself advocate for hope-and-prayer development methodology is astonishing. However, I understand why we’re still having this debate. It’s the same debate that’s been occurring for the same reasons for decades. Doing things correctly is mentally more difficult, and so people jump through ridiculous rhetorical hoops to justify why they will not — or quite often, mentally cannot — perform that intellectual labor. It’s a disheartening lack of craftsmanship and industry accountability, but it’s nothing new.
• slyall 11mo ago
  Ironically just now I got a Cloudflare "Error code 524" page because blog.cloudflare.com was down
• jijji 11mo ago
  this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about
  • mercnz 11mo ago
    i think that is data rather than code which is where it falls short, in a way you need stringent code and more safeguarded code; it's like if everyone sends you 64k posts as that's all your proxy layer lets in, someone checked sending 128kb and it gave an error before reaching your app - and then someone sends 128kb and the proxy layer has changed - and your app crashes as it was more than 64kb and your app had an assert against that. to actually track issues with erraneous data that overflows well and stuff isn't so much code test but more like fuzz testing, brute force testing etc. which i think people should do; but that's more like we need strong test networks, and also those test networks may need to be more internet like to reflect real issues too, so the whole testing infrastructure in itself becomes difficult to get right - like they have their own tunneling system etc, they could segregate some of their servers and make a test system with better error diagnosis etc potentially. but to my mind, if they had better error propogation back that really identified what was happening and where then that would be a lot better in general. sure, start doing that on a test network. this is something i've beeen tihnking about in general - i made a simple rpc system for being able to send real time rust tracing logs (it allows to just use the normal tracing framework and use a thin rpc layer) back from multiple end servers but that's mostly for granular debugging. i've never quite understood why systems like systemd-journald aren't more network centric when they're going to be big and complex kitchensink approaches - apparently there's dbus support, but to my mind something inbetween debugging level of code and warning/info. like even if it's doing things like 1/20 of log info it's too much volume if things like large files getting close to limits is increasing etc and we can see this as things run, and can see if it's localised or common etc it'd help have more resilient systems. something may already exist in this line but i didn't come across anything in a reasonably passive way - i mean there's debugging tools like dtrace etc that have been around for ages.
  • 11mo ago
• yoyohello13 11mo ago
  People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue. I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.
  • SchemaLoad 11mo ago
    Yeah even if you handled this situation without unwrap() if you just went down an error path that didn't panic, the service would likely still be inoperable if every single request went down the error path.
    • HL33tibCe7 11mo ago
      Okay, but if you returned a wrapped error it’d at least be easier to debug.
      • VMG 11mo ago
        not necessarily, by default `Result` does not even carry a stack trace
  • samdoesnothing 11mo ago
    The reason why people are criticizing is because Rust evangelicals say stuff like "if it compiles it works" or talk about how Rust's type system is so much better than other languages that it catches logic errors like this. You won't see Go or Java developers making such strong claims about their preferred languages.
    • Patryk27 11mo ago
      > [...] it catches logic errors like this but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens > You won't see Go or Java developers making such strong claims about their preferred languages. yess no Java developer ever said that OOP will solve world hunger
• JamesJGoodwin 11mo ago
  >Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features. So they basically hardcoded something, didn't bother to cover the overflow case with unit tests, didn't have basic error catching that would fallback and send logs/alerts to their internal monitoring system and this is why half of the internet went down?
• testemailfordg2 11mo ago
  "Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero." This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL. I hope it was not because of AI driven efficiency gains.
  • lmm 11mo ago
    In most domains, silently returning 0 in a case where your logic didn't actually calculate the thing you were trying to calculate is far worse than giving a clear error.
• habibur 11mo ago
  On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures As of 17:06 all systems at Cloudflare were functioning as normal 6 hours / 5 years gives ~99.98% uptime.
  • TehShrike 11mo ago
    I'm feeling generous tonight, I'm willing to consider 0.99986 to round to 99.99%
• arjie 11mo ago
  Great post-mortem. Very clear. Surprised that num(panicking threads) didn't show up somewhere in telemetry.
• avereveard 11mo ago
  Question: customer having issues also couldn't switch their dns to bypass the service, why is the control plane updated along the data plane here it seem a lot of use could save business continuity if they could change their dns entry temporarily
• wildmXranat 11mo ago
  Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query. There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N" If you knew that this design is rigid, why not leverage the query to actually do it ? What am I missing ?
  • JuniperMesos 11mo ago
    Because nothing forced them to and they didn't think of it. Maybe the people writing the code that did the query knew that the tables they were working with never had more than 60 rows and figured "that's small" so they didn't bother with a limit. Maybe the people who wrote the file size limit thought "60 rows isn't that much data" and made a very small file size limit and didn't coordinate with the first people. Anyway regardless of which language you use to construct a SQL query, you're not obligated to put in a max rows
    • wildmXranat 11mo ago
      I imagine there's numerous ways to protect against it and protection should've been added by whoever decided on this optimization. In data layer, create some kind of view which never returns more than 200 rows from base table(s). In code, use some kind of iterator. I'm not a Rust guy, just a C defensive practices type of dude, but maybe they just missed a biggie during a code review.
• sema4hacker 11mo ago
  If you deploy a change to your system, and things start to go wrong that same day, the prime suspect (no matter how unlikely it might seem) should be the change you made.
  • wildmXranat 11mo ago
    My first question when faced with an unknown error is "What was the last change and when was it promoted?"
  • 1970-01-01 11mo ago
    This is an area where they are allowed to think yet another record setting DDoS attack first and bad config second.
• cvhc 11mo ago
  I don't get why that SQL query was even used in the first place. It seems it fetches feature names at runtime instead of using a static hardcoded schema. Considering this decides the schema of a global config, I don't think the dynamicity is a good idea.
• back_to_basics 11mo ago
  While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be: Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?
  • JuniperMesos 11mo ago
    Who's "we"? This is not a trick question, what specific people do you think acted wrongly here? I don't use Cloudflare personally. I don't run any of the sites that do use it. The people who did make the decision to put thier websites behind Cloudflare could stop, and maybe some will, but presumably they're paying for it because they think, perhaps accurately, that they get value out of it. Should some power compel them not to use Cloudflare?
• issafram 11mo ago
  I give them a pass on lots of things, but this is inexcusable
• alhirzel 11mo ago
  > I worry this is the big botnet flexing. Even worse - the small botnet that controls everything.
• ademarre 11mo ago
  I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open. Of course, some users were still blocked, because the Turnstile JS failed to load in their browser but the subsequent siteverify check succeeded on the backend. But overall the fail-open implementation lessened impact to our customers nonetheless. Fail-open with Turnstile works for us because we have other bot mitigations that are sufficient to fall back on in the event of a Cloudflare outage.
  • cj 11mo ago
    So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?
    • ademarre 11mo ago
      Only if they are able to block the siteverify check performed by our backend server. That's not the kind of attack we are trying to mitigate with Turnstile.
• pdimitar 11mo ago
  While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language. They are escape hatches. Without those your language would never take off. But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to lunch in a nearby restaurant. --- Cloudflare should likely invest in better linting and CI/CD alerts. Not to mention isolated testing i.e. deploy this change only to a small subset and monitor, and only then do a wider deployment. Hindsight is 20/20 and we can all be smartasses after the fact of course. But I am really surprised because lately I am only using Rust for hobby projects and even I know I should not use `unwrap` and `expect` beyond the first iteration phases. --- I have advocated for this before but IMO Rust at this point will benefit greatly from disallowing those unsafe APIs by default in release mode. Though I understand why they don't want to do it -- likely millions of CI/CD pipelines will break overnight. But in the interim, maybe a rustc flag we can put in our `Cargo.toml` that enables such a stricter mode? Or have that flag just remove all the panicky API _at compile time_ though I believe this might be a Gargantuan effort and is likely never happening (sadly). In any case, I would expect many other failures from Cloudflare but not _this_ one in particular.
  • duped 11mo ago
    This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable. Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do. If panicking is guaranteed because of some input mistake to the system your failure is in testing.
    • pdimitar 11mo ago
      I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit. I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all. And you will not believe how many alerts I got for the "impossible" errors. I do agree there was not too much that could have been done, yes. But they should have invested in more visibility and be more thorough. I mean, hobbyist Rust devs seem to do that better. It was just a bit disappointing for me. As mentioned above, I'd understand and sympathise with many other mistakes but this one stung a bit.
• deleted 11mo ago
  [deleted]
• lapcat 11mo ago
  It's unbelievable that the end of this postmortem is an advertisement for Cloudflare. The last thing we need here is for more of the internet to sign up for Cloudflare.
  • JuniperMesos 11mo ago
    It's in Cloudflare's interest and they wrote the blog post.
• ulfw 11mo ago
  The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al. This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does. Yesterday with Cloudflare down half the sites I tried gave me nothing but errors. The internet is dead.
  • rubatuga 11mo ago
    I mean you still depend on authoritative dns servers no?
  • samdoesnothing 11mo ago
    It's not that deep, if AWS or Cloudflare suddenly disappeared sites would move to different hosts, it wouldn't mean the internet would die.
• AtNightWeCode 11mo ago
  So they made a newbie mistake in SQL that would not even pass an AI review. They did not verify the change in a test environment. And I guess the logs are so full of errors it is hard to pinpoint which matters. Yikes.
• cvshane 11mo ago
  Would be nice if their Turnstile could be turned off on their login page when something like this happens, so we can attempt to route traffic away from Cloudflare during the outage. Or at least have a simple app where this can be modified from.
• snoppy45 11mo ago
  I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.
  • sophacles 11mo ago
    I think you should get exactly what the contract you signed said you'd get. Outages happen in all infrasturture. Planned and unplanned ones both. The SLA and SLO are literal acknowledgements of the fact that and part of the contract for that reason. Its fair to be upset at their decision making - use that to renegotiate your contract.
• nwellinghoff 11mo ago
  The real take away is that so much functionality depends on a few players. This is a fundamental flaw in design that is getting worse by the year as the winner takes all winners win. Not saying they didn’t earn their wins. But the fact remains. The system is not robust. Then again, so what. It went down for a while. Maybe we shouldn’t depend on the internet being “up” all the time.
• awesome_dude 11mo ago
  But but but muh rust makes EVERYTHING safer!!!! My dude, everything is a footgun if you hold it wrong enough
• homeonthemtn 11mo ago
  Another long day...
• weihz5138 11mo ago
  Glad it's fixed, keep going!
• bri3d 11mo ago
  Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random weird edge case #9999999?
  • spprashant 11mo ago
    I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched. Sometimes you have smart people in the room who dig deeper and fish it out, but you cannot always rely on that.
    • bri3d 11mo ago
      I have plenty of empathy, having been in plenty of similar situations. It's not a matter of "I can't BELIEVE it took that long" (although it is a bit surprising) so much as that I disagree with the key takeaways here in the HN comments section and in the blog itself, which focus strongly on fixing rare edge case issues (the bad ClickHouse query and a bad config file causing a panic via unwrap), rather than reducing MTTR for all issues by improving the debug and monitoring experience. I'm also suspicious that > Eliminating the ability for core dumps or other error reports to overwhelm system resources from the blog had a lot more to do with the issue than perhaps the narrative is letting on.
  • reassess_blind 11mo ago
    Once they figured it out they didn't have a way to load in a new feature file, had to figure that out, and then restart every machine.
    • bri3d 11mo ago
• themark 11mo ago
  “…and the fluctuation stabilized in the failing state.” Sounds like the ops team had one hell of a day.
• abigailphoebe 11mo ago
  kudos to getting this blog post out so fast, it’s well written and is appreciated. i’m a little confused on how this was initially confused for an attack though? is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the <bot checking logic>" error mapping that could have initially pointed you guys towards that over an attack. also a bit taken aback that .unwrap()’s are ever allowed within such an important context. would appreciate some insight!
  • pornel 11mo ago
    1. Cloudflare is in the business of being a lightning rod for large and targeted DoS attacks. A lot of cases are attacks. 2. Attacks that make it through the usual defences make servers run at rates beyond their breaking point, causing all kinds of novel and unexpected errors. Additionally, attackers try to hit endpoints/features that amplify severity of their attack by being computationally expensive, holding a lock, or trigger an error path that restarts a service — like this one.
    • abigailphoebe 11mo ago
      eh, i'm not convinced. this was in the middle of a scheduled maintenance, with all requests failing at a singular point - that being a .unwrap(). there should be internal visibility into the fact a large number of requests are failing all at the same LOC - and attention should be focused there instantly imo. or at the very least, it shouldn't take 4 hours for anyone to even consider it wasn't an attack. in situations such as this, where your entire infra is fucked, you should have multiple crisis teams working in parallel, under different assumptions. if even one additional team was created that worked under the assumption it was an infra issue rather than an attack, this situation could have been resolved many hours earlier. for a product as vital to the internet as cloudflare, it is unacceptable to not have this kind of crisis management.
• robofanatic 11mo ago
  hope no one was fired
• spprashant 11mo ago
  A lot of outages off late seem to be related to automated config management. Companies seem to place a lot of trust is configs being pushed automatically without human review into running systems. Considering how important these configs are, shouldn't they perhaps first be deployed to a staging/isolated network for a monitoring window before pushing to production systems? Not trying to pontificate here, these systems are more complicated than anything I have maintained. Just trying to think of best practices perhaps everyone can adopt.
• jdlyga 11mo ago
  We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.
  • smt88 11mo ago
    Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software. What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.
    • Pooge 11mo ago
      Costs to architect systems that serve millions of request daily have gone down. Not up. Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client. People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)
      • smt88 11mo ago
        > Costs to architect systems that serve millions of request daily have gone down. Not up. I never said serving millions of requests is more expensive. Protecting your servers is more expensive. > Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client. HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloudflare's CDN/proxy is giving to HN incredibly cheaply, then running HN at the edge with good performance (and protecting it from botnets) would costs hundreds of millions of dollars. > People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?) Again, attacking your own straw men here. Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.
        • 11mo ago
• keypusher 11mo ago
  The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raising alerts or at least easily findable via log aggregation. It seems like maybe there was some significant time lost in assuming it was an attack, but it also seems strange to me that nobody was asking "what just changed?", which is usually the first question I ask during an incident.
  • Fiadliel 11mo ago
    If one actually looks at the current pingora API, it has limited ability to initialize async components at startup - the current pattern seems to be to lazily initialize on first call. An obvious downside of this is that a service can startup in a broken state. e.g. https://github.com/cloudflare/pingora/issues/169 https://github.com/cloudflare/pingora/issues/169 I can imagine that this could easily lead to less visibility into issues.
  • eastdakota 11mo ago
    That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.
• Adam2025 11mo ago
  Cloudflare’s write-up is clear and to the point. A small change spread wider than expected, and they explained where the process failed. It’s a good reminder that reliability depends on strong workflows as much as infrastructure.
• vasuadari 11mo ago
  Wondering why they didn’t disable the bot management temporarily to recover. Websites could have survived temporarily without it compared to the outage itself.
• uecker 11mo ago
  So an unhandled error condition after an configuration update similar to Crowdstrike - if they had just used a programming language where this can't happen due to the superior type system such as Rust. Oh wait.
• nurettin 11mo ago
  I can never get used to the error happening at call site rather than within the function where the early return of Err happened. It is not "much cleaner", you have no idea which line and file caused it at call site. By default Returning should have a way of setting a marker which can then be used to map back to the line() and file(). 10+ years and still no ergonomics.
• aetherspawn 11mo ago
  Cloudflare Access is still experiencing weird issues for us (it’s asking users to SSO login to our public website even though our zone rules - set on a completely different zone - haven’t changed). I don’t think the infrastructure has been as fully recovered as they think yet…
• laurentiurad 11mo ago
  Reason for the failure: switched to Chad IDE to ship new features.
• sanjitb 11mo ago
  cloudflare: > Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. also cloudflare: > The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.
  • perching_aix 11mo ago
    I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing. Cloudflare's status page: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ Cloudflare Dashboard: https://dash.cloudflare.com/ https://dash.cloudflare.com/
    • sanjitb 11mo ago
      thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.
      • perching_aix 11mo ago
        Yes, that was missing (along with the London WARP thing). Other comments mentioned that their status page is an Atlassian Statuspage solution, hosted on AWS CloudFront. Unclear to me if it's an Atlassian-managed deployment they have, or if it's self-managed, I'm not familiar with Statuspage and their website isn't helping. Though if it's managed, I'm not sure how they can know for sure there's no interdependence. (Though I guess we could technically keep that rabbit hole going indefinitely.)
• slanterns 11mo ago
  https://x.com/guanlandai/status/1990967570011468071 https://x.com/guanlandai/status/1990967570011468071
• wileydragonfly 11mo ago
  Did some $300k chief of IT blame it all on some overworked secretary clicking a link in an email they should have run through a filter? Because that’s the MO.
  • mos87 11mo ago
    Hopefully that's the case.
• aspbee555 11mo ago
  unwraps are so very easy to use and they have bit me so many times because you can nearly never run into a problem and suddenly crashes from an unwrap that almost always was fine
  • NetMageSCW 11mo ago
    How was AI involved?
• nromiun 11mo ago
  Unbelievable. I guess it's time to grep for every .unwrap in our code.
  • zgk7iqea 11mo ago
    theres a clippy lint for that
• keiywuvfwofw 11mo ago
  This post was written by chatgpt???? https://blog.cloudflare.com/18-november-2025-outage/#:~:text=access%20their%20sites-,%E2%80%94,-or%20not. https://blog.cloudflare.com/18-november-2025-outage/#:~:text...
  • boarush 11mo ago
    Not all em dashes are ChatGPT. Good writers use it wherever required.
• keiywuvfwofw 11mo ago
  This post was written by chatgpt?? https://blog.cloudflare.com/18-november-2025-outage/#:~:text=access%20their%20sites-,%E2%80%94,-or%20not. https://blog.cloudflare.com/18-november-2025-outage/#:~:text...
  • oskarkk 11mo ago
    Here's a random post from their blog by the same author from 2017 with an em dash: > As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics. https://blog.cloudflare.com/patent-troll-battle-update-doubling-down-on-project-jengo/ https://blog.cloudflare.com/patent-troll-battle-update-doubl... ChatGPT didn't invent the em dash, some people were always using it. But yeah, it's often one of the signs of AI.
    • NetMageSCW 11mo ago
      Too many think AI instead of horses.
• dilyevsky 11mo ago
  Long time ago Google had a very similar incident where ddos protection system ingested a bad config and took everything down. Except it was auto resolved in like four minutes by an automatic rollback system before oncall was even able to do anything. Perhaps Cloudflare should invest in a system like that
• lofaszvanitt 11mo ago
  It is staggering to see that even large companies like CF have zero monitoring, so they would know what happened in t=0.
• zeroq 11mo ago
  But Rust was supposed to cure cancer and solve world hunger. Is this the end of the hello world but in Rust saga?
  • deleted 11mo ago
    [deleted]
• thatoneengineer 11mo ago
  The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they weren't. The slowness to root cause: sheer bad luck, with the status page down and Azure's DDoS yesterday all over the news. The broken SQL: this is the one that I'd be up in arms about if I worked for Cloudflare. For a system with the power to roll out config to ~all of prod at once while bypassing a lot of the usual change tracking, having this escape testing and review is a major miss.
  • Xunjin 11mo ago
    Share the same opinion, as others pointed out, the status page down probably caused by bots checking it.
  • watchful_moose 11mo ago
    It's probably not ok to silently run with a partial config, which could have undefined semantics. An old but complete config is probably ok (or, the system should be designed to be safe to run in this state).
  • vbezhenar 11mo ago
    IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts). So basically bad config should be explicitly processed and handled by rolling back to known working config.
    • jgilias 11mo ago
      You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.
• hbarka 11mo ago
  ClickHouse db was mentioned. Does this incident raise any critiques about it?
  • deleted 11mo ago
    [deleted]
• mmaunder 11mo ago
  tl;dr A permissions change in a ClickHouse database caused a query to return duplicate rows for a “feature file” used by Cloudflares Bot Management system, which doubled the file size. That oversized file was propagated to their core proxy machines, triggered an unhandled error in the proxy’s bot-module (it exceeded its pre-allocated limit), and as a result the network started returning 5xx errors. The issue wasn’t a cyber-attack — it was a configuration/automation failure.
• jeffrallen 11mo ago
  This is an excellent lesson learned: Harden loading of internally generated config files as though they were untrusted content. Gonna use that one at $WORK.
• kylegalbraith 11mo ago
  The outage sucked for everyone. The root cause also feels like something they could have caught much earlier in a canary rollout from my reading of this. All that said, to have an outage reported turned around practically the same day, that is this detailed, is quite impressive. Here's to hoping they make their changes from this learning, and we don't see this exact failure mode again.
  • agentifysh 11mo ago
    how would you build redundancy around cloudflare failing? i think this is happening way too frequently meanwhile VPS, dedicated servers hum along without any issues i dont want to use kubernetes but if we have to build mission critical systems doesn't seem like building on cloudflare is going to cut it
• makach 11mo ago
  Excellent write up. Cybersecurity professionals read the story and learn. It’s textbook lesson in post-mortem incident analysis - a mvp for what is expected from us all in a similar situation. Reputationally this is extremely embarrassing for Cloudflare, but imo they seem to get their feet back on the ground. I was surprised to see not just one, but two apologies to the internet. This just cements how professional and dedicated the Cloudflare team is to ensure stable resilient internet and how embarrassed they must have been. A reputational hit for sure, but outcome is lessons learned and hopefully stronger resilience.
• Martcpp 11mo ago
  deny (clippy:: unwrap_used)
• agonux 11mo ago
  Time to rewrite with golang, explicit error handling ;-)
• dev_l1x_be 11mo ago
  Was it DNS this time?
• MagicMoonlight 11mo ago
  Having a system which automatically deploys configuration files across a million servers every 5 minutes without testing it seems stupid to me.
• anal_reactor 11mo ago
  Honestly... everyone shit themselves that internet doesn't work, but next week this outage will be forgotten by 99% of population. I was doing something on my PC when I saw clear information that Cloudflare is down, so I decided to just go take a nap, then read a book, then go for a walk. Once I was done, the internet was working again. Panic was not necessary on my side. What I'm trying to say is that things would be much better if everyone took a chill pill and accepted the possibility that in rare instances, the internet doesn't work and that's fine. You don't need to keep scrolling TikTok 24/7. > but my use case is especially important Take a chill pill. Probably it isn't.
  • vultour 11mo ago
    It's funny how everyone seems to be having a meltdown over this. I didn't even notice anything was wrong until I read about it on Reddit 5 hours later, even though I was working all day. Sounds to me like people are too reliant on random websites.
• xlii 11mo ago
  Cloudflare rewrites Rust services to <next-cool-language> /joke ... (I'd pick Haskell, cause I'm having fun with it recently :P)
• deleted 11mo ago
  [deleted]
• HL33tibCe7 11mo ago
  An unwrap like that in production code on the critical path is very surprising to me. I haven’t worked in Rust codebases, but I have never worked in a Go codebase where a `panic` in such a location would make it through code review. Is this normal in Rust?
  • xnotcursed 11mo ago
    absolutely not normal, this is why in my opinion it took them so long to understand the core issue. instead of a nice error message and a backtrace saying something like "failed to parse config feature names" they thought they were under attack because the service was just crashing instead.
• jokoon 11mo ago
  I don't understand what's the business of cloudflare. They just sell proxies, to whoever. Why are they the only company doing ddos protection? I just don't get it.
  • BOOSTERHIDROGEN 11mo ago
    Momentum, I guess, pretty much like you wouldn’t get fired for using AWS or IBM (if that’s still the case now).
  • bskybp 11mo ago
    [dead]
• hnarn 11mo ago
  > That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. > The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail. I'm no FAANG 10x engineer, and I appreciate things can be obvious in hindsight, but I'm somewhat surprised that engineering at the level of Cloudflare does not: 1. Push out files A/B to ensure the old file is not removed. 2. Handle the failure of loading the file (for whatever reason) by automatically reloading the old file instead and logging the error. This seems like pretty basic SRE stuff.
  • watchful_moose 11mo ago
    Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.
    • hnarn 11mo ago
      Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.
      • kevincox 11mo ago
        I think they need to strongly evaluate if they need this level of rollout speed. Even spending a few minutes with an automated canary gives you a ton of safety. Even if the servers weren't crashing it is possible that a bet set of parameters results in far too many false positives which may as well be complete failure.
• 0x001D 11mo ago
  Configuration can be validated. https://cuelang.org https://cuelang.org
• lalam 11mo ago
  Hack Free fire 8000 5487 3565 644664 464664644 464646449
• lalam 11mo ago
  Hack Free fire
• lalam 11mo ago
  F
• deleted 11mo ago
  [deleted]
• niedbalski 11mo ago
  Sure this has been said but the issue here is not code is the ability to canary and rollback quickly from any arbitrary (config) change.
• zyngaro 11mo ago
  Catastrophique failure for failing to read a file bigger that expected? Wow. This is really embarrassing.
• sachahjkl 11mo ago
  me af when there's a postmortem rubbing hands, impish smile on my face
• gkoz 11mo ago
  Given this was triggered by an old school configuration change across multiple servers, there's too little discussion of that particular process. It sounds like the change could've been rolled out more slowly, halted when the incident started and perhaps rolled back just in case.
• 130R 11mo ago
  If the software has a limit on the size of the feature file then the process that propagates the file should probably validate the size before propagating ..
• xyst 11mo ago
  A fucking unhandled exception brought down a majority of the internet? Why do we continue to let these clowns run a large portion of the internet? Big tech is a fucking joke.
  • NetMageSCW 11mo ago
    20% sure seems to grow with every commenter piling on. It’s like an Internet game of telephone.
• elAhmo 11mo ago
  Timely post mortem. Sucks to have this happened, but at least they are quite transparent and detailed in the writeup.
• cowsandmilk 11mo ago
  Blog post from less than a week ago on how Cloudflare avoids outages on configuration changes: https://blog.cloudflare.com/finding-the-grain-of-sand-in-a-heap-of-salt/ https://blog.cloudflare.com/finding-the-grain-of-sand-in-a-h... This has to sting a bit after that post.
• leonaves 11mo ago
  Why have a limit on the file size if the thing that happens when you hit the limit is the entire network goes down? Surely not having a limit can't be worse?
• __alexs 11mo ago
  Is dual sourcing CDNs feasible these days? Seems like having the capability to swap between CDN providers is good both from a negotiating perspective and a resiliency one.
• arkanovicz 11mo ago
  Interesting technical insight, but I would be curious to hear firsthand accounts from the teams on the ground, particularly regarding how the engineers felt the increasing pressure, frantically refreshing their dashboards, searching for phantom DDoS, scrolling codes updates...
• l___l 11mo ago
  > The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail. What could have prevented this failure? Cloudflare's software could have included a check that refused to generate the feature file if it's size was higher than the limit. A testcase could have caught this.
• markhandoff 11mo ago
  Dear Matthew Prince, don't you think we (the ones affected by your staff's mistake) should get some sort of compensation??? Yours truly, a Cloudflare client who lost money during the November 18th outage.
  • NetMageSCW 11mo ago
    What does your contract with Cloudflare say?
• darksideofthem 11mo ago
  Speaking of resiliency, the entire Bot Management module doesn't seems to be a critical part of the system, so for example, what happens if that module goes down for an hour? the other parts of the system should work. So I would rank every module and it's role in the system, and would design it in a way that when a non-critical module fails, other parts still can function.
• kjgkjhfkjf 11mo ago
  Seems like a substantial fraction of the web was brought down because of a coding error that should have been caught in CI by a linter. These folks weren't operating for charity. They were highly paid so-called professionals. Who will be held accountable for this?
• igornadj 11mo ago
  Any feature failing should still allow the traffic to continue. This should be the first bullet in the future actions list.
• BrtByte 11mo ago
  This incident feels like a strong argument for stricter guardrails around internal config propagation
• kqr 11mo ago
  One of the remediations listed is > Eliminating the ability for core dumps or other error reports to overwhelm system resources but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.
  • janpio 11mo ago
    I understood this to be related to this section: > As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information. (Just above https://blog.cloudflare.com/18-november-2025-outage/#how-cloudflare-processes-requests-and-how-this-went-wrong-today https://blog.cloudflare.com/18-november-2025-outage/#how-clo...)
    • kqr 11mo ago
      Ah, yes, of course. I must have missed that. Thanks!
• keepamovin 11mo ago
  That's interesting. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. It's like the issues with HOSTS.TXT needing to be copied among the network of the early internet to allow routing (taking days to download etc) and DNS having to be created to make that propagation less unwieldy.
• drc500free 11mo ago
  Makes me wonder which team is responsible for that feature generating query, and if they follow full engineering level QA. It might be deferred to an MLE team that is better than the data scientists but less rigorous than software needs to be.
  • mmaia 11mo ago
    Exactly. The post screams about all the issues I've seen in multiple companies between DS/MLE and SE/DevOps.
• cmilton 11mo ago
  How many changes to production systems does Cloudflare make throughout a day? Are they a part of any change management process? That would be the first place I would check after a random outage, recent changes.
• CSMastermind 11mo ago
  I'm honestly curious what culturally is going on inside Cloudflare given they've had a few outages this year.
• Forgeties79 11mo ago
  I’ll be honest, I only understand about 30% of what is being said in this thread and that is probably generous. But it is very interesting seeing so many people respond to each other “it’s so simple! what went wrong was…” as they all disagree on what exactly went wrong.
• baalimago 11mo ago
  Interesting. Although principle of least privilege is great, it should not be applied as a feature to filter data.
• baalimago 11mo ago
  Git blame disabled on the line which crashed it - Cowards!
• merobin_hood 11mo ago
  [dead]
• chaos_emergent 11mo ago
  Just a moment to reflect on how much freaking leverage computers give us today - a single permission change took down half the internet. Truly crazy times.
• zf00002 11mo ago
  As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actually smaller than our old building and has no room to grow.
  • rkomorn 11mo ago
    As a former CF employee, I'd say it's a mixed bag. There are plenty of resources , yet it's somehow never enough. You do tons of pretty amazing things with pretty amazing tools that also have notable shortcomings. You're surround by smart people who do lots of great work, but you also end up in incident reviews where you find facepalm-y stuff. Sometimes you even find out it was a known corner case that was deemed too unlikely to prioritize. The last incident for my team that I remember dealing with there ended up with my coworker and I realizing the staging environment we'd taken down hours earlier was actually the source of data for a production dashboard, so we'd lost some visibility and monitoring for a bit. I've also worked at Facebook (pre-Meta days) and at Datadog, and I'd say it was about the same. Most things are done quite well, but so much stuff is happening that you still end up with occasional incidents that feel like they shouldn't have happened.
• deleted 11mo ago
  [deleted]
• jamesblonde 11mo ago
  Cloudflare tried to build their own feature store, and get a grade F. I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility. Reference: https://www.oreilly.com/library/view/building-machine-learning/9781098165222/ https://www.oreilly.com/library/view/building-machine-learni...
• assbuttbuttass 11mo ago
  My website was down too, because a tree fell on my power line
• zhisme 11mo ago
  Thank you for being honest, all must learn from the mistakes.
• Chihuahua0633 11mo ago
  > The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35. I'm impressed they were able to corral people this quickly.
• throw7 11mo ago
  Is this true: from that core proxy diagram, I didn't realize cloudflare sees the full unencrypted packet between you and the server. If that's true, is there a way to tell (easily) whether a site is using cloudflare or not?
  • tempest_ 11mo ago
    It is pretty easy to see if cloudflare is proxying a site. Just ping the host and see if the ip belongs to CF. https://www.cloudflare.com/en-ca/ips/ https://www.cloudflare.com/en-ca/ips/
  • finally7394 11mo ago
    The NSA has to see the data somehow, right?
• Barry-Perkins 11mo ago
  [flagged]
  • jhiggins777 11mo ago
    ...ai
• jjice 11mo ago
  There's (obviously) a lot of discussion around the use of `unwrap` in production code. I feel like I'm watching comments speak past each other right now. I'd agree that the use of `unwrap` could possibly make sense in a place where you do want the system to fail hard. There's lot of good reasons to make the system fail hard. I'd lean towards an `expect` here, but whatever. That said, the function already returns a `Result` and we don't know what the calling code looks like. Maybe it does do an `unwrap` there too, or maybe there is a save way for this to log and continue that we're not aware of because we don't have enough info. Should a system as critical as the CF proxy fail hard? I don't know. I'd say yes if it was the kind of situation that could revert itself (like an incremental rollout), but this is such an interesting situation since it's a config being rolled out. Hindsight is 20:20 obviously, but it feels like there should've been better logging, deployment, rollback, and parsing/validation capabilities, no matter what the `unwrap`/`Result` option is. Also, it seems like the initial Clickhouse changes could've been testing much better, but I'm sure the CF team realizes that. On the bright side, this is a very solid write up so quickly after the outage. Much better than those times we get it two weeks later.
• Matthias247 11mo ago
  When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with. The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad". The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.
• ogurechny 11mo ago
  > Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable. Excuse me, what you've just said? Who decided on “Cloudflare's importance in the Internet ecosystem”? Some see it differently, you know, there's no need for that self-assured arrogance of an inseminating alpha male.
• arifiqbal81 11mo ago
  Thanks for the detailed writeup and explaining the root cause in details However, I have a question from a release deployment process perspective. Why was this issue not detected during internal testing ? I didn't find the RCA analysis covering this aspect. Doesn't cloudflare have an internal test stage as part of its CICD pipeline. Looking the description of the issue, it should have been immediately detected in internal stage test environment.
• flerchin 11mo ago
  Based on this writeup it seems that Cloudflare defaults to a 0 score for bot prevention if there's a failure. Could instead it default to a passing score? Default open instead of default closed? This would have been a non-event to a lot of websites if that change was made.
• NetMageSCW 11mo ago
  It feels like their list of after actions is lacking a bit to me. How about 1. The permissions change project is paused or rolled back until 2. All impacted database interactions (SQL queries) are evaluated for improper assumptions or better 3. Their design that depends on database metainfo and schema is replaced with ones that use specific tables and rows in tables instead of using the meta info as part of their application. 4. All hard coded limits are centralized in a single global module and referenced from their users and then back propagated to any separate generator processes that validate against the limit before pushing generated changes
• kwar13 11mo ago
  Perils of a centralized chokepoint. Substantial amounts of the internet shouldn't go down because Cloudflare had a hiccup.
• realtyblocks 11mo ago
  our site realtyblocks.com went down but we redirected the traffic by bypassing the cloudflare DNS routes with few clicks. Thank you for resolving the issue Cloudflare.
• realtyblocks 11mo ago
  our site realtyblocks.com went down but we redirected the traffic by bypassing DNS entries with few clicks. Thank you for resolving the issue Cloudflare.
• realtyblocks 11mo ago
  our site realtyblocks.com went down but we redirected the traffic with few clicks. Thank you for resolving the issue Cloudflare.
• Diggsey 11mo ago
  There were two things I think went extremely poorly here: 1) Lack of validation of the configuration file. Rolling out a config file across the global network every 5 minutes is extremely high risk. Even without hindsight, surely one would see then need for very careful validation of this file before taking on that risk? There were several things "obviously" wrong with the file that validation should have caught: - It was much bigger than expected. - It had duplicate entries. - Most importantly, when loaded into the FL2 proxy, the proxy would panic on every request. At the very least, part of the validation should involve loading the file into the proxy and serving a request? 2) Very long time to identify and then fix such a critical issue. I can't understand the complete lack of monitoring or reporting? A panic in Rust code, especially from an unwrap, is the application screaming that there's a logic error! I don't understand how that can be conflated with a DDoS attack. How are your logs not filled with backtraces pointing to the exact "unwrap" in question? Then, once identified, why was it so hard to revert to a known good version of the configuration file? How did noone foresee the need to roll back this file when designing a feature that deploys a new one globally every 5 minutes?
• phyzome 11mo ago
  This is a small issue in the writeup (everything else made sense), but -- why does doubling 60 features exceed the 200 limit? Missing something.
• harivyom 11mo ago
  I am just going by the outage post mortem report. I could not read the article after I read the first few lines - "feature file". I am stuck at consuming the design idea here where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have made the insertions as just a put/replace the value. I think that was not the case here, where cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here cloudflare should rethink. If someone can educate me on the design, I can continue reading the next lines.
• harivyom 11mo ago
  I am just going by the outage post-mortem report. I could not read the article after I read the first few lines - "feature file" expansion/limits. I am stuck at consuming the design idea here, where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have allowed for insertions as simple as a put/replace of the value and not appending to the file. That was not the case here, where Cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here that Cloudflare should rethink. If someone can educate me on the design, I can continue reading the next few lines.
• mayank94 11mo ago
  Thanks for the detailed RCA. After going through the blog post as well, there’s a curiosity about whether there are opportunities to detect such scenarios earlier in the testing phase. It would be helpful to understand if there are any differences between the test environment and production that might have contributed to this. This could provide insights into strengthening the process going forward.
• cocaine_coder 11mo ago
  [dead]
• tete 11mo ago
  Bit funny. "Memory management" bug in Rust. (writing that as kind of a fan of Rust)
• amitkumary96 11mo ago
  As we know that "With great power comes great responsibility" the team should understand this because Cloudflare is used worldwide, and for many countries it was the peak working time when Cloudflare went down so this affected massively. We always want perfect results but it's not possible, I hope the team is not overworking to get the changes on prod.
• laotree 11mo ago
  Attempt to reproduce the Cloudflare 2025-11-18 outage. Cloudflare's incident report is written clearly and explicitly, so based on my own understanding, I’m going to try reproducing this outage. Already completed: CK cluster Permission change triggering data doubling Cache propagation Unaffected proxy services Proxy services with bot score errors TODO: unwrap panic during pre-allocation of cache Full demonstration of the entire outage process https://github.com/Laotree/reproduce_cf20251118 https://github.com/Laotree/reproduce_cf20251118
• pkumar00007 11mo ago
  If you knew the expected number of features , any input file with >100 should be discarded as bad input and you failback to the last good feature file received. This would have protected your service even though you are unable to get the newly populated features. I believe these features are not updated that frequently. Even if they were, you would have biased your system towards availability vs 'correctness'. What were the teams doing between 11 to 1300 hrs , no explanation of what investigations were going on to not being able to figure the root cause.
• pkumar00007 11mo ago
  Did your incident response team look at the last few changes that were executed? If they had , they could have just rolleback the change. or just looking at the changes executed, in the vicinity of the start of the outage could have pointed to the problem. Didn't the services that were crashing due to OOM raise any alerts? This is shitty at so many levels.
• metasnitch02 11mo ago
  It was ASSUREDLY a cyber attack. Go to jeffblearning on LinkedIn. I took it down with 253 copies of a text file delivered through a vulnerability in Novo’s systems. I’ve documented all of it. It’s not done yet…
• proverbs53 11mo ago
  What I read is that a non-critical feature (blocking / managing bot - access) was able to impact a critical feature (routing traffic). Shouldn't the architecture setup in such a way that subcomponents can fail without impacting the critical function of the component?
• atari_guy 11mo ago
  It actually wasn't resolved quite that fast. My site continued to have issues for hours afterward before things finally resolved completely.