15 ms·

Post Mortem on Cloudflare Control Plane and Analytics Outage

486 points by eastdakota 3y ago

matdehaast 3y ago
Some days things just go badly. The only thing you can change is how you respond. Well done to you and the team for getting through this. I for one am and will always be a cloudflare customer
nanankcornering 3y ago
At what time were you notified Matt?
- eastdakota 3y ago
  Of the incident? Someone on my team called me about 30 minutes after it started. It was challenging for me to stay on top of because it was also the same day as our Q3 earnings call. But team kept me informed throughout the day. I helped where I could. And they handled a very difficult situation very well. That said, lots we can learn from and improve.
  - chaz6 3y ago
    What I find bizarre is that the Cloudflare share price jumped when the outage happend! Having read the post mortem, I do not think it could have been handled any better. I think the decision to extend the outage in order to provide rest was absolutely correct. I always enjoy reading these reports from Cloudflare as they are the best in the business.
    - eastdakota 3y ago
      I was surprised we didn't get a single question about it from an analyst or investor, either formally on the Q3 call or on any callbacks we did after. One weird phenomenon we've seen — though not so much in this case because the impact wasn't as publicly exposed — is that investors after we've had a really bad outage say: "Oh, wow, I didn't fully appreciate how important you were until you took down most of the Internet." So… ¯\_(ツ)_/¯
    - dboreham 3y ago
      There's a class of investor (and their trade bots presumably) that sees outrage over a service outage as proof the provider is now mission critical, hence able to "extract value" from the market.
  - mercurialuser 3y ago
    Did you rebuild all the server from scratch?
  - iAMkenough 3y ago
    Coincidental timing?
divbzero 3y ago
Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would cause all three to be impacted, while still close enough that they could all run active-active redundant data clusters. If the three data centers are all around Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.
- nanankcornering 3y ago
  thats why DRC exists right?
- pests 3y ago
  > Hillsboro, Oregon, an earthquake could probably take out all three simultaneously. Is it west of I5? (yes) Oh yeah, they all gone. Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz https://pnsn.org/outreach/earthquakesources/csz
  - bell-cot 3y ago
    Wikipedia's entry for Hillsboro: "Elevation 194 ft (60 m)" Between that, and being ~50 miles inland - I'd say there's ~zero threat of Cascadia quakes or tsunamis directly knocking out those DC's. (Yeah, larger-scale infrastructure and social order could still be killers.) OTOH - Mt. St. Helens is about 60 miles NNE of Hillsboro. If that really went boom, and the wind was right...how many cm's of dry volcanic ash can the roofs of those DC's bear? What if rain wets that ash? How about their HVAC systems' filters?
    - pests 3y ago
      I was never worried about the tsunami. Okay, maybe not gone, but I wouldn't say it would be operational. https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercise_Scenario.pdf https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercis... 50% of roads and near 75% of bridges damaged on the west coast and the I5 corridor. Refer to PDF page #93 where over 70% of power generation is highly damaged on the I5 corridor and 60% in the coastal areas with 0% undamaged. Highly damaged - "Extensive damage to generation plants, substations, and buildings. Repairs are needed to regain functionality. Restoring power to meet 90% of demand may take months to one year." "In the immediate aftermath of the earthquake, cities within 100 miles of the Pacific coastline may experience partial or complete blackout. Seventy percent of the electric facilities in the I-5 corridor may suffer considerable damage to generation plants, and many distribution circuits and substations may fail, resulting in a loss of over half of the systems load capacity (see Table 22). Most electrical power assets on the coast may suffer damage severe enough as to render the equipment and structures irreparable"
tebbers 3y ago
I think overall Cloudflare did a decent job on this. Clearly the DC provider cocked up big time here, but Cloudflare kept running fine for the vast majority of customers globally. No system is perfect and it’s only apocalyptic scenarios like this where the vulnerabilities are exposed - and they will now be fixed. Hope the SRE guys got some rest after all that stress.
- vb-8448 3y ago
  > Clearly the DC provider cocked up big time here Actually, this is the CF version, maybe Flexential will come out with a different one. BTW, if you design a system to survive a DC failure, you cannot blame the DC failure.
austinkhale 3y ago
I love how thorough Cloudflare post mortem’s are. Reading the frank, transparent explanations are like a breath of fresh air compared to the obfuscation of nearly every other company comm’s strategy. We were affected but it’s blog posts like these that make me never want to move away. Everyone makes mistakes. Everyone has bad days. It’s how you react afterwards that makes the difference.
- w10-1 3y ago
  I agree, but I also think that for security purposes they should leave out extraneous detail. Also, I know they want to hold their suppliers accountable, but I would hold off pointing fingers. It doesn't really improve behavior, and it makes incentives worse. I really appreciate that they're going to fix the process errors here. But as they suggested, there's a tension between moving fast and being sure. This is typically managed like the weather, buying rain jackets afterwards (not optimal). I'd be curious to see how they can make reliability part of the culture without tying development up in process. Perhaps they can model the system in software, then use traffic analytics to validate their models. If they can lower the cost of reliability experiments by doing virtual experiments, they might be able to catch more before roll-out.
  - logifail 3y ago
    > I also think that for security purposes they should leave out extraneous detail Disagree completely, it's the frank detail that makes me trust their story.
    - ShadowRegent 3y ago
      Maybe, but I think that their "Informed Speculation" section was probably unnecessary. They may or may not be correct, but give Flexential an opportunity to share what actually happened rather than openly guessing on what might have happened. Instead, state the facts you know and move onto your response and lessons learned.
      - 3y ago
deleted 3y ago
[deleted]
TekMol 3y ago
Contrary to others here, I find the postmortem a bit lacking. The TLDR is that CF runs in multiple data centers, one went down, and the services that depend on it went down with it. The interesting question would be why those services did depend on a single data center. They are pretty vague about it Cloudflare allows multiple teams to innovate quickly. As such, products often take different paths toward their initial alpha. If I was the CEO, I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center. That would make an interesting blog post to me. Designing a highly available system and building a company fast leads to interesting tradeoffs. The details would be interesting.
- pests 3y ago
  Isn't this sentence a bit further down more clear? > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. and > It [PDX-04] is also the default location for services that have not yet been onboarded onto our high availability cluster.
- nanankcornering 3y ago
  I experienced it myself within the last 24 hours. New D1 & Hyperdrive deployment was not working. It would spew out internal errors & timeouts. Both are non-GA products, and the point is that non-GA are not part of the HA cluster (yet)
- kccqzy 3y ago
  > why they decided to make services depend on just one data center In my experience, no engineers really decided to make services depend on just one data center. It happened because the dependency was overlooked. Or it happened because the dependency was thought to be a "soft dependency" with graceful degradation in case of unavailability but the graceful degradation path had a bug. Or it happened because the engineers thought it had a dependency on one of multiple data centers, but then the failover process had a bug. Reminds me of that time when a single data center in Paris for GCP brought down the entire Google Cloud Console albeit briefly. Really the same thing.
solatic 3y ago
Not criticism, just remarks: > While, over time, our practice is to migrate the backend for these services to our best practices, we did not formally require that before products were declared generally available (GA). I really like the model where a single team in a company, with Product + Dev, can quickly ship, iterate on a new product, and prove market demand without going through layers and layers of internal bureaucracy (Ops/Infra, Security, Privacy/Legal, Finance approval for production-scale), with the main stipulation being that such work is marked as alpha/beta/preview, and only going through the layers of internal bureaucracy once it's ready to go GA. But most companies really struggle with this, especially with ensuring that customers are never exposed to a/b/p software by default, requiring opt-in from the customer, allowing the customer to easily opt-out, and ensuring that using a/b/p software never endangers GA features they depend on. Building that out, if it's even on a company's internal Platform/DevX backlog, is usually super far down as a "wishlist" item. So I'm super interested to see what Cloudflare can build here and whether that can ever get exposed as part of their public Product portfolio as well. > We need to use the distributed systems products that we make available to all our customers for all our services so they continue to function mostly as normal even if our core facilities are disrupted. Super excited to see this. Cloudflare Workers is still too much of an "edge" platform and not a "main datacenter" platform, at least because D1 is still in beta and even if it wasn't, Postgres is far more feature-ful, and that pulls more software into a traditional single-datacenter model. So if Cloudflare can really succeed at this, then it'll be a much stronger statement in favor of building out software in an edge-only model. Between the Pages outage and the API outage happening in one week, I was considering selling my NET stock, but reading a postmortem like this reminds me why I invested in NET in the first place. Thanks Matt.
- throwaway6920 3y ago
  > I really like the model where a single team in a company, with Product + Dev, can quickly ship, iterate on a new product, and prove market demand without going through layers and layers of internal bureaucracy (Ops/Infra, Security, Privacy/Legal, Finance approval for production-scale), with the main stipulation being that such work is marked as alpha/beta/preview, and only going through the layers of internal bureaucracy once it's ready to go GA. Speaking from personal experience, what you're claiming as 'good', for CF meant SRE- usually core, but edge also suffered- got stuck with trying to fix a fundamentally broken design that was known faulty- and called faulty repeatedly- but forced through. Nothing about this is desirable or will end well. This reckoning was known and raised by multiple SRE near a decade before this occurred, and there were multiple near misses in the last few years that were ignored. The part that's probably funny- and painful- for ex-CF SRE is that the company will do a hard pivot and try to rectify this mess. It's always harder to fix after, rather than building for, and they've ignored this for a long while.
pests 3y ago
> It is not unusual for utilities to ask data centers to drop off the grid when power demands are high and run exclusively on generators. Are the data centers compensated or anything for this? I'd imagine generator-only might cost more in terms of fuel and wear-and-tear/maintinaince/inspections. edit: > DSG allows the local utility to run a data center's generators to help supply additional power to the grid. In exchange, the power company helps maintain the generators and supplies fuel Interesting.
- nijave 3y ago
  I'm not very well versed in this space but I've been told Progressive Insurance in Cleveland, OH has a similar (sounding) agreement. According to PGE's website, they basically pay for everything https://portlandgeneral.com/save-money/save-money-business/dispatchable-standby-generators https://portlandgeneral.com/save-money/save-money-business/d...
Dunedan 3y ago
> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04. > A handful of products did not properly get stood up on our disaster recovery sites. These tended to be newer products where we had not fully implemented and tested a disaster recovery procedure. So the root cause for the outage was that they relied on a single data center. I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet.
- SushiHippie 3y ago
  And the top comment on the other HN post called it: https://news.ycombinator.com/item?id=38113503 https://news.ycombinator.com/item?id=38113503
- thelastparadise 3y ago
  > I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Absolute lack of faith in cloudflare rn. This is amateur hour stuff. It's especially egregious that these are new services that were rolled out without HA.
  - NicoJuicy 3y ago
    ? Tbh. As far as I can see, their data plane worked at the edge. Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush. Their control plane was bad though. But since most products worked, that's more redundancy than most products. The proposed solution is simple: - GA requires to be in the high availability cluster - test entire DC outages
    - 3y ago
iot_devs 3y ago
Why the very first step was not to fail over Europe?
- pests 3y ago
  They did after two hours. After the first they assumed the generators would be back but then ran into the breaker issue which caused the full day delay.
- ahoka 3y ago
  My question too, although possibly it seemed as a greater risk first to fail over. BTW, is there any unexpected GDPR implication of that? Assuming that fail over means restoring US backups in EU.
  - MRtecno98 3y ago
    iirc the GDPR prohibits storing EU data in non-EU servers, not vice-versa
    - philipwhiuk 3y ago
      But it does mean all that data is now required to be handled in a compliant fashion
pseudocoder_sp 3y ago
Some one should make a webseries about this indecent. It will be a nice story to tell. Name: Mordern Day Disaster. Directed by : Mathew prince Releasing on : 25th December at Netflix Based on a true story.
__turbobrew__ 3y ago
> However, we had never tested fully taking the entire PDX-04 facility offline. That is a painful lesson, but unless you are physically powering off the dc or at least disconnecting the network from the outside world you are not testing a real disaster. You can point fingers at the facility operators, but at the end of the day you have to be able to recover from a dc going completely offline and maybe never coming back. Mother Nature may wipe it off the face of the earth.
- martinald 3y ago
  This is a fair point. Imagine there had been a serious fire like OVH suffered or flooding that destroyed the data center. Would Cloudflare have been able to recover?
  - creshal 3y ago
    Most likely, yes. They have enough customer lock-in that enough customers would stick with them even if it took them a week to rebuild everything from in other DCs.
  - NicoJuicy 3y ago
    That's not what happened here. Their edge worked fine. Business was mostly running as usual. The OVH outage was immediate downtime.
sidcool 3y ago
They really threw the electricity power provider under the bus there.
- tux3 3y ago
  The electricity provider is fine, it's Flexential that looks incredibly opaque and non-communicative in a stressful situation. While Cloudflare should have been better prepared for this, it seems to be amateur hour in that particular Portland data-center. Other customers (Dreamhost, etc) were impacted too, and I can't imagine they don't also have some very pointed questions.
  - SentinelRosko 3y ago
    Sure, but DreamHost recovered fully within 12 hours [1], Cloudflare took almost 2 days [2] [1] https://www.dreamhoststatus.com/pages/incident/575f0f606826303142000510/65438de1df1f3405ae39b151 https://www.dreamhoststatus.com/pages/incident/575f0f6068263... [2] https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg
awesomebing1 3y ago
Somewhat amazed at the structure of this article: after first discussing the third-party for 75% of blog post, the first-party recovery efforts were detailed in considerably lesser paragraphs. It’s promising to see a path forward mentioned but I can’t help but wonder why this was published instead of currently acknowledging their failure/circumstances and later on publishing a complete post-mortem after the dust fully settles (i.e. without speculation).
- ahoka 3y ago
  It’s called “shifting the blame”.
- xyst 3y ago
  To make sure their stonk doesn’t drop at market open next week. Investors will read this (or get the sound bites) and shrug it off as some vendor issue rather than deep issue that will require months of rework (millions of dollars and thus impacting earnings)
lbriner 3y ago
A lot of mud-slinging on here about HA setup and CF's dealing with the problem but I can only assume people are armchair experts with no real experience of HA at the scale of CF. "So the root cause for the outage was that they relied on a single data center.". No. Root cause was that data centre operator didn't manage the outage properly and didn't have systems in place in which case they could have avoided it + some systems knowingly and unknowingly had dependencies on the centre that went down because CF did have systems in place to allow that centre to fail. "Cloudflare has a shit reputation in my eyes, because their terrible captchas". You don't like one product so they have a shit reputation? Enough said. "but unless you are physically powering off the dc or at least disconnecting the network from the outside world you are not testing a real disaster." If you have ever had to do this, you know that it is never a good feeling. On-paper, yes, you should try your DR but in reality, even if it works, you lose data, you get service blips, you get a tonne of support calls and if it doesn't work, it might not even rollback again. On top of that, it isn't a case of just disconnecting something, most problems are more complicated. System A is available but not system B. Routers get a bad update but are still online, and on top of all of that, you would need some way to know that everything is still working and some problems don't surface for hours or until traffic volume is at a certain level etc. If you trust that a data centre can stay online for long periods of time and that you would then be able to migrate things at a reasonable rate if it doesn't, then you have to trust that to an extend. All-in-all, CF are not attempting to blame someone, even though a lot is down to Flexential, the last paragraph of the first section says, "To start, this never should have happened...I am sorry and embarrassed for this incident and the pain that it caused our customers and our team." Well done CF
- menaerus 3y ago
  > some systems knowingly and unknowingly had dependencies on the centre that went down because CF did have systems in place to allow that centre to fail. I mean you're contradicting yourself in the same sentence. Had CloudFlare had such a system in place that would allow that particular center to fail, there would be no outages in the service. The truth is that they didn't account for it , and because they missed it, that center became a single point of failure which is what brought the whole CloudFlare service down. Power outage was just a trigger to discover a weakness in their system design and not a root cause.
logronoide 3y ago
Classic distraction maneuver. This postmortem is a prime example of tech porn that diverts attention from the main issue: many at Cloudflare didn't do their job properly.
- deleted 3y ago
  [deleted]
Tocra 3y ago
I am really upset about this situation on behalf of CF however why don't they think about generating their own electricity with renewable energy sources?
- usr1106 3y ago
  And why not run their own DC? It doesn't change anything fundamentally. A complex product is only as good as the weakest link. I have worked with various employers, some world leaders at the time. All of them had seriously weak links.
- duskwuff 3y ago
  > why don't they think about generating their own electricity with renewable energy sources? How exactly do you imagine that working while inside a data center operated by a third party? It's not like they let you stick some solar panels on the roof and run an extension cord to your rack.
RockRobotRock 3y ago
hot take: HN is way too biased and sympathetic towards the provider whenever an outage like this happens.
- kkielhofner 3y ago
  Not a lot of measured takes here. It seems to be either “eh we get it, comms could have been better” or “they’re idiots”. The first group of people have been to war. The second have not.
darkwater 3y ago
Well, this is definitely NOT a blameless post-mortem! Obviously I'm joking because they are blaming an external company (Flexential) to which they are surely paying big money for the DC space.
- xyst 3y ago
  I wonder if CF execs aiming to use this to get out of their long term contract with them?
minimaul 3y ago
> Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning. That decision delayed our full recovery, but I believe made it less likely that we’d compound this situation with additional mistakes. I liked this - the human element is underemphasised often in these kinds of reports, and trying to fix a major outage while overly tired is only going to add avoidable mistakes. I don’t know how it would work for an org of Cloudflare’s size, but I know we have plans for a significant outage for staff to work/sleep in shifts, to try to avoid that problem as well. Issue there is that you need a way to hand over the current state of the outage to new staff as they wake up/come online.
- kkielhofner 3y ago
  I’m curious, have these plans ever been tested in a real incident? Like Mike Tyson says, everyone has a plan until they get punched in the face.
  - bombcar 3y ago
    The biggest key to implementing these types of plans is that when the shit hits the fan, you send a third of the people home - so they can come back in 10-20 hours are relieve those who are still there. If you don't do that, you're still going to be scrambling.
yowai 3y ago
As someone who was slightly affected by this outage, I personally also find this post-mortem to be lacking. 75% of the post-mortem talks about the power outage at PDX-04 and blames Flexential. Okay, fair - it was a bit of a disaster what was happening there judging from the text. But by end of November 2 (UTC), power was fully restored. It still took ~30 hours according to the post-mortem for Cloudflare to fully recover service. This was longer than the outage, and the text just states that too many services were dependent from each other. But I'd wish they go into more detail here why the operation as a whole took that long. Are there any take-aways from the recovery process, too? Or was it really just syncing data from the edges back to the "brain" that took this long? Also one aspect I am missing here is the lack of communication - especially to Enterprise customers. Cloudflare support was basically radio silent during this outage except for the status page. Realistically, they couldn't do much anyway. But at least any attempt at communication would be appreciated - especially for Enterprise customers, and even more especially after the post-mortem blames Flexential for a lack of communication. While I like Cloudflare since it's a great product, I think there are still a few more things that should be taken as a conclusion for CF to take away from this incident. That being said, glad you managed to recover, and thanks for the post-mortem.
- vb-8448 3y ago
  > Also one aspect I am missing here is the lack of communication - especially to Enterprise customers. They blame Flexential for lack of communication, but were the first one not saying anything.
  - iAMkenough 3y ago
    Even "we don't know why our data center is failing, but we're sending a team over to physically investigate now" would have been A+ communication in the moment.
    - NicoJuicy 3y ago
      Everything was on the status page since the start? DC related updates: > Update - Power to Cloudflare’s core North America data center has been partially restored. Cloudflare has failed over some core services to a backup data center, which has partially remediated impact. Cloudflare is currently working to restore the remaining affected services and bring the core North America data center back online. Nov 02, 2023 - 17:08 UTC > Identified - Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. We will keep providing regular updates until the issue is resolved, thank you for your patience as we work on mitigating the problem. Nov 02, 2023 - 13:40 UTC
nickdothutton 3y ago
Consider modes of failure.
andyjohnson0 3y ago
> We are a relatively large customer of the facility, consuming approximately 10 percent of its total capacity. I'm surprised that CF are renting space in colocation facilities. I would have expected a business of their size to have their own DCs. Is this common practice for cloud providers?
- NicoJuicy 3y ago
  You thought that they build > 300 DC's? Colo is much more flexible, cheaper and quicker to start. Definitely since they sit close to the end-user on the data plane.
  - andyjohnson0 3y ago
    > You thought that they build > 300 DC's? I have no idea how many DCs they have or operate in. Where does "300" come from? > Colo is much more flexible, cheaper and quicker to start. Definitely since they sit close to the end-user on the data plane. I understand that, but it has the disadvantage of reduced control and observability - particularly in the event of an outage such as that described in the blog post. I kind of assumed that top-tier cloud platforms like AWS/Azure/GCP operate out of dedicated DCs, and that CF are similar because of their well-known scale of operations. Since my original comment has been downvoted†, someone presumably thinks this it was a naive or trivial question - although I don't understand why. († I don't much care about downvotes, but I do take them to be a signal.)
    - martinald 3y ago
      https://www.cloudflare.com/network/ https://www.cloudflare.com/network/
    - NicoJuicy 3y ago
      Probably most of us follow Cloudflare a bit more closely. They want DC's close to every big city. I think most of us knew that they can't launch > 300 DC's in such a short amount of time. The many amount of DC's is mentioned a lot ( social networks, blogs, here). There is a distinction between eg. AWS / Azure / ... Which work with a couple of big DC's, while cloudflare operates more spread across more locations. You're comment did made me realize it may may not be that clear from an outsider viewpoint though ( fyi, I'm an outsider too)
ShadowRegent 3y ago
Interesting choice to spend the bulk of the article publicly shifting blame to a vendor by name and speculating on their root cause. Also an interesting choice to publicly call out that you're a whale in the facility and include an electrical diagram clearly marked Confidential by your vendor in the postmortem. Honestly, this is rather unprofessional. I understand and support explaining what triggered the event and giving a bit of context, but the focus on your postmortem needs to be on your incident, not your vendor's. Clearly, a lot went wrong and Flexential needs to do their own postmortem, but Cloudflare doesn't need to make guesses and do it for them, much less publicly.
- creshal 3y ago
  Especially since it shouldn't matter why the DC failed — Cloudflare's entire business model is selling services allegedly designed to survive that. 99% of the fault lies with Cloudflare for not being able to do their core job.
  - corobo 3y ago
    In all fairness the rest of the article is about that
    - creshal 3y ago
      Slightly less than half, and the bottom half, so that people just skimming over it will mostly remember the DC operators' problems, not Cloudflare's own. This is very deliberately manipulative.
      - corobo 3y ago
        It is of course possible they've shuffled things around since this was posted but it seems that the first part addresses their system failings. 5th paragraph to the 9th are Cloudflare's "we buggered up" before they get to the power segment. They then continue with the "this is our fault for not being fully HA" after the power bit. Each to their own, I'm going to read it as a regular old post mortem on this one.
kurok 3y ago
So well known fails. Same was with one of our DC 3 years ago when they powered city and failed whole DC
technotarek 3y ago
True, not snippy: I found interesting that their automated billing emails seemed to arrive right on time.
- nijave 3y ago
  HA, maximum redundancy
thomasdeml 3y ago
Poor doc: You had a high availability 3 data center setup that utterly failed. Why spend the first third of the document blaming your data center operator? The management of the data center facility is outside of your control. You gambled that not appropriately testing your high-availability setup (under your control) would not have consequences. You should absolutely discuss the DC management with your operator, but that's between you and them and doesn't belong in this post mortem.
nabla9 3y ago
They don't always know it, but all large systems are moving gradually towards dependency management system with logic rules that covers "everything", physical, logical, human and administrative dependencies. Every time something new not covered is discovered, new rules and conditions are added. You can do it with manual checklists, multiple rule checkers, or put everything together. I suspect that in end it's just easier to put everything into single declarative formal verification system and see if new change to the system passes, transition between configurations passes etc.
- lytedev 3y ago
  This is such an interesting way of putting it. I think this has been the subconscious reason I've been gravitating towards defining _everything_ I manage personally (and not yet at work) with Nix. It's not quite to the extent you're talking about here, of course, but in a similar vein at least.
marcinzm 3y ago
> Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning A minor point but this feels like not the most efficient way to manage an emergency. Having some form of staggered shifts or other approach versus just having everyone pile on. If a lot of knowledge resided in specific individuals so they are vital to an effort like this and cannot be substituted then that seems like a risk in it's own.
someonehere 3y ago
I asked a sales rep once about services going out and how that would affect CF For Teams. They said it would be virtually impossible for CF to go down because of all their data centers around the world. Paraphrasing, “if there’s an outage, there’s definitely something going wrong with the internet.” And here we are. My trust in them has hit zero.
- usr1106 3y ago
  Why would you trust a sales rep? Even honest engineers cannot foresee the exact cascading consequences effects of such outages. Sales reps are not paid to be either competent on such issues nor to be honest.
- cj 3y ago
  FWIW, I'm a Cloudflare Enterprise customer and we had zero downtime. Only thing that was temporarily unavailable was the cloudflare dashboard. I feel like a lot of people in this thread are commenting under the impression that all of Cloudflare was down for 24 hours when in reality I wouldn't be surprised if a lot of customers were unaffected and unaware of the incident. I wouldn't even have known of the outage had it not been for HN..
  - ianhawes 3y ago
    2nd this. We had zero downtime on anything in production. The only reason we knew is because we are actively standing up a transition to R2 and ran into errors configuring buckets.
- belter 3y ago
  Trust? Random sample from the last 60 days... Cloudflare outage – 24 hours now - https://news.ycombinator.com/item?id=38112515 https://news.ycombinator.com/item?id=38112515 Cloudflare Dashboard Logins Failing - https://news.ycombinator.com/item?id=38112230 https://news.ycombinator.com/item?id=38112230 Ask HN: Cloudflare Workers are down? - https://news.ycombinator.com/item?id=38074906 https://news.ycombinator.com/item?id=38074906 Cloudflare API, dashboard, tunnels down - https://news.ycombinator.com/item?id=38014582 https://news.ycombinator.com/item?id=38014582 Cloudflare Intermittent API Failures for Cloudflare Pages, Workers and Images - https://news.ycombinator.com/item?id=37819045 https://news.ycombinator.com/item?id=37819045 Cloudflare Issues with 1.1.1.1 public resolver and WARP - https://news.ycombinator.com/item?id=37762731 https://news.ycombinator.com/item?id=37762731 Cloudflare – Network Performance Issues - https://news.ycombinator.com/item?id=37604609 https://news.ycombinator.com/item?id=37604609 Cloudflare Issues Passing Challenge Pages - https://news.ycombinator.com/item?id=37336743 https://news.ycombinator.com/item?id=37336743
whoknowsidont 3y ago
>While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted. They're just going to straight up lie like that? We definitely weren't able to get "traffic through [their] network" through the outage at many different random points. So if the CF team is under the impression traffic was not impacted, dig deeper.
alberth 3y ago
Taking the positive outlook … Ensuring all of their services are fully distributed is now top of mind at CF. Ultimately, customers win if CF executes.
sackfield 3y ago
Its weird that upon reading this post, I have less confidence in Cloudflare. They basically browbeat Flexential for behaving unprofessional, which, yes, they probably did. However the fact that this causes entire systems that people rely on to go down is a massive redundancy failure on Cloudflares part, you should be able to nuke one of these datacentres and still maintain services. Very worrying is they start by stating their intended design: > Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon You need way more geographic dispersion than that, this control pane is used by people across the world. We are still on the intended design, not the flawed implementation by the way, which is wild to me. > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. I don't understand why this would ever be done in this way. If Cloudflare is making a new product for consumers shouldn't redundant design be at the forefront here? I am surprised that it was even an option. For the record I do use Cloudflare for certain systems and I use it because I assume it has great failovers if events like this occur making me not have to worry about these eventualities, but now I will be reconsidering this, how do I actually know my cloudflare workers are safe from these design decisions? > When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. Yeh I'll bet, its because Cloudflares core design is not redundant. Really disappointed in this blog post trying to shift the blame to Flexential when this slapdash architecture should be the main problem on show. As a customer I don't care if Flexential disappears in an earthquake tomorrow, I expect Cloudflare to handle it gracefully.
- onionisafruit 3y ago
  Is the Hillsboro thing is about latency?
  - DylanSp 3y ago
    That may well be part of it, some people were talking about the impact of latency in the outage thread [1]. [1] https://news.ycombinator.com/item?id=38113952 https://news.ycombinator.com/item?id=38113952
channer 3y ago
appreciate the status updates and the quick report. as a person that handles the tiniest datacenter, it's impossible to predict every potential event. best you can do is to recover as quickly as possible and learn the lesson. believing that this doesn't or can't happen to another vendor is being naive. it has happened to all of them and it'll happen again. can only hope it's super rare.
weird-eye-issue 3y ago
Wow they REALLY buried this important part didn't they! This took a ton of scrolling: "Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04." Bingo, there we have it.
- dmix 3y ago
  What does PDX-04 mean here? Not familiar with how data centers work.
  - weird-eye-issue 3y ago
    Read the damn article! It's explained at the top.
    - deleted 3y ago
      [deleted]
  - edaemon 3y ago
    PDX is the airport code for Portland, Oregon, USA. It's the fourth Portland data center.
- dclowd9901 3y ago
  Yep, past the part where they spent a long time blaming the data center and power company.
- worksonmine 3y ago
  Nah, if only the data center would've stayed up this wouldn't have been a problem. It's clearly on the data center. /s
- indigomm 3y ago
  In my experience, power is the most common data center failure there is. Often it's the redundant systems that cause the failure.
burroisolator 3y ago
While debatably unprofessional to blame your vendor, I found this read to be fascinating. I'm sure there are blog posts that detail how data centers work and fail but it's rare to get that cross over from a software engineering context. It puts into perspective what it takes for an average data center of this class to fail: power outage, generator failure, and then battery loss.
- chippiewill 3y ago
  I think what it really does is emphasise how common it is for crap to hit the fan when things go wrong - even with the best laid plans. The DC almost certainly advertises the redundant power supplies, generator backups and battery failover in order to get the customers. But probably doesn't do the legwork or spend the money to make those things truly reliable. It's a bit like having automated backups - but never testing them and discovering they're empty when they're really needed.
stemlord 3y ago
I'm ultimately glad this happened because it very effectively helps illustrate how we are assigning a centralized gatekeeper to the internet at the infrastructure level and why it's a bad thing.
simple10 3y ago
This is good reminder to myself to transfer domains registered on Cloudflare to another provider and only use Cloudflare for DNS or vice versa. I was effectively locked out of making any changes to domains registered and DNS hosted on Cloudflare during the entire outage due to single point of failure (Cloudflare) on my part.
jtchang 3y ago
For me it's basically summed up as "we didn't test turning the power off" and making sure things worked the way we planned. Yes it is hard and very expensive to do these types of tests. And doing it regularly is even more $$$ and time. As most customers we seem to be okay with a cheap price hidden behind a facade of "high availability" since I don't really want to pay for true HA. Because if I knew the real cost it would be too expensive.
AzzyHN 3y ago
To preface: I am not qualified to talk about this in the slightest. > Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04 What do you mean you discovered? How could you not know? Surely when you were setting this high availability cluster up years ago and migrating services over, you double checked that all crucial dependencies had also been moved, right? And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right???
- gpm 3y ago
  > And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right??? They discussed this. They had been running tests were they disabled the high availability cluster in any of (and of two of) the three DCs. That test didn't involve disabling the rest of the (non-HA) services from PDX-04 DC (oops).
madcadmium 3y ago
This is karma for their passive-aggressive blog post about Okta's security incident one week earlier
milkglass 3y ago
After reading this, we will be moving off Cloudflare. The post shows a lack of maturity and professionalism.
donutshop 3y ago
So the tl:dr is that they took a risk and never tested their high availability setup.
markhahn 3y ago
Shockingly poor choices by CF. We should absolutely blame them, just as "victims" of ransomware should be blamed. Hardening against system failure is the same process as security hardening.
throwoutway 3y ago
> Throughout the incident, Cloudflare's network and security services continued to work as expected. This seems misleading. Their own status page said the data plane was impacted across many services.
play168 3y ago
How you
tamimio 3y ago
Whenever we design and build a data center (in addition to disaster recovery, business continuity, etc.), we always test all possible scenarios of power interruptions and never assume that the power utility will get in touch with us in advance for a pre-planned maintenance, including the outcome AND the time to restore etc., it’s kind of shocking that you just assumed UPS will last 10min but never put it on test.. especially with size of CF!
zx8080 3y ago
Cloudflare Inc share price has increased 15% as a result. How does it work?
rymiel 3y ago
I truly and genuinely hope that this incident and the many before will drive away customers from Cloudflare's monopoly
zX41ZdbW 3y ago
You can run ClickHouse cluster across multiple datacenters. It will survive the failure of a single datacenter while being available for writes and reads, and the failure of two out of three datacenters while being available for reads. It works well when RTT between datacenters is less than 30 ms. If they are more distant, it will still work, but you will notice a quite high latency on INSERTs due to the distributed consensus. I've run a ClickHouse cluster with hundreds of bare-metal machines distributed across three datacenters in two countries at my previous job. It survived power failures (multiple), a flood (once), and network connectivity issues (regular). This cluster was used for logging and analytics :)
nixgeek 3y ago
Why would any supplier want to do business with Cloudflare now? You have 1.8MW of datacenter space to lease, you have a few interested parties, how could you not view Cloudflare as a huge reputational risk? Why even do the business? Why not lease that space to someone else? Moreover, why renew the existing deals with Cloudflare? Does Cloudflare have a plan to move 200+ racks in Oregon if that supplier decides just not to renew that deal whenever it comes up next? Are Cloudflare claiming they were able to build a technical plan which gets their architecture away from this site being a SPOF before the deal is up to renew, or is the CEO making a gamble here again? Cloudflare have demonstrated their willingness to create reputational issues for suppliers by publicly shaming two of them recently, and here, in only about 2 days from incident. One interpretation of this blog would be Cloudflare are a very unreasonable customer and one who is willing to post incomplete or informal information from their suppliers. Cloudflare also chose to focus the first half of a lengthy postmortem on blaming the supplier and only then on their own culpability for the outage, despite it clearly being a shared responsibility. One of the diagrams Cloudflare have posted is clearly marked "Proprietary and Confidential". Do Cloudflare have permission to post that? It's not clearly stated that they do. Should other suppliers expect when the sh*t hits the fan that any sensitive information they've shared will be part of a blog? Most of the "Lessons and Remediation" section is stuff Cloudflare could have worked on at any point in advance of a major incident, and Cloudflare's senior management have quite clearly chosen not to prioritize that work until today, when forced to by this major incident. When signing large deals, Cloudflare will frequently have to complete 'Supplier Disclosures', and they are also making claims through industry-standard certifications [1] like ISO, SOC and FedRAMP. Most of those will ask questions about the disaster recovery and business continuity plans and Cloudflare will have (repeatedly) attested they are adequate, something that this blog clearly demonstrates was a misrepresentation of their true capabilities. Will there be an SEC disclosure coming out of this considering it could have material impacts on the business, which is publicly traded? Was there any requirement that the SEC disclosure come first, or be concurrent with a blog? [1] https://www.cloudflare.com/trust-hub/compliance-resources/ https://www.cloudflare.com/trust-hub/compliance-resources/
ta1243 3y ago
TLDR: Major flooding, riots, earthquake, asteroid, nuke goes off in Portland and Cloudflare is down because they decided they could put their entire control infrastructure in a single location for ease of use. Centralising the web is such a great move.
GoAWAYS 3y ago
Is it common for CloudFlare to publish documents or diagrams from their vendors clearly marked "Proprietary and Confidential"
deleted 3y ago
[deleted]
ArrayQueue 3y ago
The guy I genuinely feel sorry for is "an unaccompanied technician who had only been on the job for a week". Regardless of any and all corporate spin on the issue, a newbie was dumped into an event at the worst possible time. I really really really hope he gets a decent bit of counselling to make sure that is fully aware that the issue with the data centre had NOTHING to do with him unplugging the coffee maker to plug in his recharger for his iPhone. Absolutely nothing at all.

486 points by eastdakota 3y ago

Some days things just go badly. The only thing you can change is how you respond. Well done to you and the team for getting through this. I for one am and will always be a cloudflare customer

nanankcornering 3y ago

At what time were you notified Matt?

eastdakota 3y ago
Of the incident? Someone on my team called me about 30 minutes after it started. It was challenging for me to stay on top of because it was also the same day as our Q3 earnings call. But team kept me informed throughout the day. I helped where I could. And they handled a very difficult situation very well. That said, lots we can learn from and improve.
- chaz6 3y ago
  What I find bizarre is that the Cloudflare share price jumped when the outage happend! Having read the post mortem, I do not think it could have been handled any better. I think the decision to extend the outage in order to provide rest was absolutely correct. I always enjoy reading these reports from Cloudflare as they are the best in the business.
  - eastdakota 3y ago
    I was surprised we didn't get a single question about it from an analyst or investor, either formally on the Q3 call or on any callbacks we did after. One weird phenomenon we've seen — though not so much in this case because the impact wasn't as publicly exposed — is that investors after we've had a really bad outage say: "Oh, wow, I didn't fully appreciate how important you were until you took down most of the Internet." So… ¯\_(ツ)_/¯
  - dboreham 3y ago
    There's a class of investor (and their trade bots presumably) that sees outrage over a service outage as proof the provider is now mission critical, hence able to "extract value" from the market.
- mercurialuser 3y ago
  Did you rebuild all the server from scratch?
- iAMkenough 3y ago
  Coincidental timing?

divbzero 3y ago

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would cause all three to be impacted, while still close enough that they could all run active-active redundant data clusters. If the three data centers are all around Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.

nanankcornering 3y ago
thats why DRC exists right?
pests 3y ago
> Hillsboro, Oregon, an earthquake could probably take out all three simultaneously. Is it west of I5? (yes) Oh yeah, they all gone. Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz https://pnsn.org/outreach/earthquakesources/csz
- bell-cot 3y ago
  Wikipedia's entry for Hillsboro: "Elevation 194 ft (60 m)" Between that, and being ~50 miles inland - I'd say there's ~zero threat of Cascadia quakes or tsunamis directly knocking out those DC's. (Yeah, larger-scale infrastructure and social order could still be killers.) OTOH - Mt. St. Helens is about 60 miles NNE of Hillsboro. If that really went boom, and the wind was right...how many cm's of dry volcanic ash can the roofs of those DC's bear? What if rain wets that ash? How about their HVAC systems' filters?
  - pests 3y ago
    I was never worried about the tsunami. Okay, maybe not gone, but I wouldn't say it would be operational. https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercise_Scenario.pdf https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercis... 50% of roads and near 75% of bridges damaged on the west coast and the I5 corridor. Refer to PDF page #93 where over 70% of power generation is highly damaged on the I5 corridor and 60% in the coastal areas with 0% undamaged. Highly damaged - "Extensive damage to generation plants, substations, and buildings. Repairs are needed to regain functionality. Restoring power to meet 90% of demand may take months to one year." "In the immediate aftermath of the earthquake, cities within 100 miles of the Pacific coastline may experience partial or complete blackout. Seventy percent of the electric facilities in the I-5 corridor may suffer considerable damage to generation plants, and many distribution circuits and substations may fail, resulting in a loss of over half of the systems load capacity (see Table 22). Most electrical power assets on the coast may suffer damage severe enough as to render the equipment and structures irreparable"

tebbers 3y ago

I think overall Cloudflare did a decent job on this. Clearly the DC provider cocked up big time here, but Cloudflare kept running fine for the vast majority of customers globally. No system is perfect and it’s only apocalyptic scenarios like this where the vulnerabilities are exposed - and they will now be fixed. Hope the SRE guys got some rest after all that stress.

vb-8448 3y ago
> Clearly the DC provider cocked up big time here Actually, this is the CF version, maybe Flexential will come out with a different one. BTW, if you design a system to survive a DC failure, you cannot blame the DC failure.

austinkhale 3y ago

I love how thorough Cloudflare post mortem’s are. Reading the frank, transparent explanations are like a breath of fresh air compared to the obfuscation of nearly every other company comm’s strategy. We were affected but it’s blog posts like these that make me never want to move away. Everyone makes mistakes. Everyone has bad days. It’s how you react afterwards that makes the difference.

w10-1 3y ago
I agree, but I also think that for security purposes they should leave out extraneous detail. Also, I know they want to hold their suppliers accountable, but I would hold off pointing fingers. It doesn't really improve behavior, and it makes incentives worse. I really appreciate that they're going to fix the process errors here. But as they suggested, there's a tension between moving fast and being sure. This is typically managed like the weather, buying rain jackets afterwards (not optimal). I'd be curious to see how they can make reliability part of the culture without tying development up in process. Perhaps they can model the system in software, then use traffic analytics to validate their models. If they can lower the cost of reliability experiments by doing virtual experiments, they might be able to catch more before roll-out.
- logifail 3y ago
  > I also think that for security purposes they should leave out extraneous detail Disagree completely, it's the frank detail that makes me trust their story.
  - ShadowRegent 3y ago
    Maybe, but I think that their "Informed Speculation" section was probably unnecessary. They may or may not be correct, but give Flexential an opportunity to share what actually happened rather than openly guessing on what might have happened. Instead, state the facts you know and move onto your response and lessons learned.
    - 3y ago

deleted 3y ago

[deleted]

TekMol 3y ago

Contrary to others here, I find the postmortem a bit lacking. The TLDR is that CF runs in multiple data centers, one went down, and the services that depend on it went down with it. The interesting question would be why those services did depend on a single data center. They are pretty vague about it Cloudflare allows multiple teams to innovate quickly. As such, products often take different paths toward their initial alpha. If I was the CEO, I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center. That would make an interesting blog post to me. Designing a highly available system and building a company fast leads to interesting tradeoffs. The details would be interesting.

pests 3y ago
Isn't this sentence a bit further down more clear? > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. and > It [PDX-04] is also the default location for services that have not yet been onboarded onto our high availability cluster.
nanankcornering 3y ago
I experienced it myself within the last 24 hours. New D1 & Hyperdrive deployment was not working. It would spew out internal errors & timeouts. Both are non-GA products, and the point is that non-GA are not part of the HA cluster (yet)
kccqzy 3y ago
> why they decided to make services depend on just one data center In my experience, no engineers really decided to make services depend on just one data center. It happened because the dependency was overlooked. Or it happened because the dependency was thought to be a "soft dependency" with graceful degradation in case of unavailability but the graceful degradation path had a bug. Or it happened because the engineers thought it had a dependency on one of multiple data centers, but then the failover process had a bug. Reminds me of that time when a single data center in Paris for GCP brought down the entire Google Cloud Console albeit briefly. Really the same thing.

solatic 3y ago

Not criticism, just remarks: > While, over time, our practice is to migrate the backend for these services to our best practices, we did not formally require that before products were declared generally available (GA). I really like the model where a single team in a company, with Product + Dev, can quickly ship, iterate on a new product, and prove market demand without going through layers and layers of internal bureaucracy (Ops/Infra, Security, Privacy/Legal, Finance approval for production-scale), with the main stipulation being that such work is marked as alpha/beta/preview, and only going through the layers of internal bureaucracy once it's ready to go GA. But most companies really struggle with this, especially with ensuring that customers are never exposed to a/b/p software by default, requiring opt-in from the customer, allowing the customer to easily opt-out, and ensuring that using a/b/p software never endangers GA features they depend on. Building that out, if it's even on a company's internal Platform/DevX backlog, is usually super far down as a "wishlist" item. So I'm super interested to see what Cloudflare can build here and whether that can ever get exposed as part of their public Product portfolio as well. > We need to use the distributed systems products that we make available to all our customers for all our services so they continue to function mostly as normal even if our core facilities are disrupted. Super excited to see this. Cloudflare Workers is still too much of an "edge" platform and not a "main datacenter" platform, at least because D1 is still in beta and even if it wasn't, Postgres is far more feature-ful, and that pulls more software into a traditional single-datacenter model. So if Cloudflare can really succeed at this, then it'll be a much stronger statement in favor of building out software in an edge-only model. Between the Pages outage and the API outage happening in one week, I was considering selling my NET stock, but reading a postmortem like this reminds me why I invested in NET in the first place. Thanks Matt.

throwaway6920 3y ago
> I really like the model where a single team in a company, with Product + Dev, can quickly ship, iterate on a new product, and prove market demand without going through layers and layers of internal bureaucracy (Ops/Infra, Security, Privacy/Legal, Finance approval for production-scale), with the main stipulation being that such work is marked as alpha/beta/preview, and only going through the layers of internal bureaucracy once it's ready to go GA. Speaking from personal experience, what you're claiming as 'good', for CF meant SRE- usually core, but edge also suffered- got stuck with trying to fix a fundamentally broken design that was known faulty- and called faulty repeatedly- but forced through. Nothing about this is desirable or will end well. This reckoning was known and raised by multiple SRE near a decade before this occurred, and there were multiple near misses in the last few years that were ignored. The part that's probably funny- and painful- for ex-CF SRE is that the company will do a hard pivot and try to rectify this mess. It's always harder to fix after, rather than building for, and they've ignored this for a long while.

pests 3y ago

> It is not unusual for utilities to ask data centers to drop off the grid when power demands are high and run exclusively on generators. Are the data centers compensated or anything for this? I'd imagine generator-only might cost more in terms of fuel and wear-and-tear/maintinaince/inspections. edit: > DSG allows the local utility to run a data center's generators to help supply additional power to the grid. In exchange, the power company helps maintain the generators and supplies fuel Interesting.

nijave 3y ago
I'm not very well versed in this space but I've been told Progressive Insurance in Cleveland, OH has a similar (sounding) agreement. According to PGE's website, they basically pay for everything https://portlandgeneral.com/save-money/save-money-business/dispatchable-standby-generators https://portlandgeneral.com/save-money/save-money-business/d...

Dunedan 3y ago

> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04. > A handful of products did not properly get stood up on our disaster recovery sites. These tended to be newer products where we had not fully implemented and tested a disaster recovery procedure. So the root cause for the outage was that they relied on a single data center. I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet.

SushiHippie 3y ago
And the top comment on the other HN post called it: https://news.ycombinator.com/item?id=38113503 https://news.ycombinator.com/item?id=38113503
thelastparadise 3y ago
> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Absolute lack of faith in cloudflare rn. This is amateur hour stuff. It's especially egregious that these are new services that were rolled out without HA.
- NicoJuicy 3y ago
  ? Tbh. As far as I can see, their data plane worked at the edge. Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush. Their control plane was bad though. But since most products worked, that's more redundancy than most products. The proposed solution is simple: - GA requires to be in the high availability cluster - test entire DC outages
  - 3y ago

iot_devs 3y ago

Why the very first step was not to fail over Europe?

pests 3y ago
They did after two hours. After the first they assumed the generators would be back but then ran into the breaker issue which caused the full day delay.
ahoka 3y ago
My question too, although possibly it seemed as a greater risk first to fail over. BTW, is there any unexpected GDPR implication of that? Assuming that fail over means restoring US backups in EU.
- MRtecno98 3y ago
  iirc the GDPR prohibits storing EU data in non-EU servers, not vice-versa
  - philipwhiuk 3y ago
    But it does mean all that data is now required to be handled in a compliant fashion

pseudocoder_sp 3y ago

Some one should make a webseries about this indecent. It will be a nice story to tell. Name: Mordern Day Disaster. Directed by : Mathew prince Releasing on : 25th December at Netflix Based on a true story.

__turbobrew__ 3y ago

> However, we had never tested fully taking the entire PDX-04 facility offline. That is a painful lesson, but unless you are physically powering off the dc or at least disconnecting the network from the outside world you are not testing a real disaster. You can point fingers at the facility operators, but at the end of the day you have to be able to recover from a dc going completely offline and maybe never coming back. Mother Nature may wipe it off the face of the earth.

martinald 3y ago
This is a fair point. Imagine there had been a serious fire like OVH suffered or flooding that destroyed the data center. Would Cloudflare have been able to recover?
- creshal 3y ago
  Most likely, yes. They have enough customer lock-in that enough customers would stick with them even if it took them a week to rebuild everything from in other DCs.
- NicoJuicy 3y ago
  That's not what happened here. Their edge worked fine. Business was mostly running as usual. The OVH outage was immediate downtime.

sidcool 3y ago

They really threw the electricity power provider under the bus there.

tux3 3y ago
The electricity provider is fine, it's Flexential that looks incredibly opaque and non-communicative in a stressful situation. While Cloudflare should have been better prepared for this, it seems to be amateur hour in that particular Portland data-center. Other customers (Dreamhost, etc) were impacted too, and I can't imagine they don't also have some very pointed questions.
- SentinelRosko 3y ago
  Sure, but DreamHost recovered fully within 12 hours [1], Cloudflare took almost 2 days [2] [1] https://www.dreamhoststatus.com/pages/incident/575f0f606826303142000510/65438de1df1f3405ae39b151 https://www.dreamhoststatus.com/pages/incident/575f0f6068263... [2] https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg

awesomebing1 3y ago

Somewhat amazed at the structure of this article: after first discussing the third-party for 75% of blog post, the first-party recovery efforts were detailed in considerably lesser paragraphs. It’s promising to see a path forward mentioned but I can’t help but wonder why this was published instead of currently acknowledging their failure/circumstances and later on publishing a complete post-mortem after the dust fully settles (i.e. without speculation).

ahoka 3y ago
It’s called “shifting the blame”.
xyst 3y ago
To make sure their stonk doesn’t drop at market open next week. Investors will read this (or get the sound bites) and shrug it off as some vendor issue rather than deep issue that will require months of rework (millions of dollars and thus impacting earnings)

lbriner 3y ago

A lot of mud-slinging on here about HA setup and CF's dealing with the problem but I can only assume people are armchair experts with no real experience of HA at the scale of CF. "So the root cause for the outage was that they relied on a single data center.". No. Root cause was that data centre operator didn't manage the outage properly and didn't have systems in place in which case they could have avoided it + some systems knowingly and unknowingly had dependencies on the centre that went down because CF did have systems in place to allow that centre to fail. "Cloudflare has a shit reputation in my eyes, because their terrible captchas". You don't like one product so they have a shit reputation? Enough said. "but unless you are physically powering off the dc or at least disconnecting the network from the outside world you are not testing a real disaster." If you have ever had to do this, you know that it is never a good feeling. On-paper, yes, you should try your DR but in reality, even if it works, you lose data, you get service blips, you get a tonne of support calls and if it doesn't work, it might not even rollback again. On top of that, it isn't a case of just disconnecting something, most problems are more complicated. System A is available but not system B. Routers get a bad update but are still online, and on top of all of that, you would need some way to know that everything is still working and some problems don't surface for hours or until traffic volume is at a certain level etc. If you trust that a data centre can stay online for long periods of time and that you would then be able to migrate things at a reasonable rate if it doesn't, then you have to trust that to an extend. All-in-all, CF are not attempting to blame someone, even though a lot is down to Flexential, the last paragraph of the first section says, "To start, this never should have happened...I am sorry and embarrassed for this incident and the pain that it caused our customers and our team." Well done CF

menaerus 3y ago
> some systems knowingly and unknowingly had dependencies on the centre that went down because CF did have systems in place to allow that centre to fail. I mean you're contradicting yourself in the same sentence. Had CloudFlare had such a system in place that would allow that particular center to fail, there would be no outages in the service. The truth is that they didn't account for it , and because they missed it, that center became a single point of failure which is what brought the whole CloudFlare service down. Power outage was just a trigger to discover a weakness in their system design and not a root cause.

logronoide 3y ago

Classic distraction maneuver. This postmortem is a prime example of tech porn that diverts attention from the main issue: many at Cloudflare didn't do their job properly.

deleted 3y ago
[deleted]

Tocra 3y ago

I am really upset about this situation on behalf of CF however why don't they think about generating their own electricity with renewable energy sources?

usr1106 3y ago
And why not run their own DC? It doesn't change anything fundamentally. A complex product is only as good as the weakest link. I have worked with various employers, some world leaders at the time. All of them had seriously weak links.
duskwuff 3y ago
> why don't they think about generating their own electricity with renewable energy sources? How exactly do you imagine that working while inside a data center operated by a third party? It's not like they let you stick some solar panels on the roof and run an extension cord to your rack.

RockRobotRock 3y ago

hot take: HN is way too biased and sympathetic towards the provider whenever an outage like this happens.

kkielhofner 3y ago
Not a lot of measured takes here. It seems to be either “eh we get it, comms could have been better” or “they’re idiots”. The first group of people have been to war. The second have not.

darkwater 3y ago

Well, this is definitely NOT a blameless post-mortem! Obviously I'm joking because they are blaming an external company (Flexential) to which they are surely paying big money for the DC space.

xyst 3y ago
I wonder if CF execs aiming to use this to get out of their long term contract with them?

minimaul 3y ago

> Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning. That decision delayed our full recovery, but I believe made it less likely that we’d compound this situation with additional mistakes. I liked this - the human element is underemphasised often in these kinds of reports, and trying to fix a major outage while overly tired is only going to add avoidable mistakes. I don’t know how it would work for an org of Cloudflare’s size, but I know we have plans for a significant outage for staff to work/sleep in shifts, to try to avoid that problem as well. Issue there is that you need a way to hand over the current state of the outage to new staff as they wake up/come online.

kkielhofner 3y ago
I’m curious, have these plans ever been tested in a real incident? Like Mike Tyson says, everyone has a plan until they get punched in the face.
- bombcar 3y ago
  The biggest key to implementing these types of plans is that when the shit hits the fan, you send a third of the people home - so they can come back in 10-20 hours are relieve those who are still there. If you don't do that, you're still going to be scrambling.

yowai 3y ago

As someone who was slightly affected by this outage, I personally also find this post-mortem to be lacking. 75% of the post-mortem talks about the power outage at PDX-04 and blames Flexential. Okay, fair - it was a bit of a disaster what was happening there judging from the text. But by end of November 2 (UTC), power was fully restored. It still took ~30 hours according to the post-mortem for Cloudflare to fully recover service. This was longer than the outage, and the text just states that too many services were dependent from each other. But I'd wish they go into more detail here why the operation as a whole took that long. Are there any take-aways from the recovery process, too? Or was it really just syncing data from the edges back to the "brain" that took this long? Also one aspect I am missing here is the lack of communication - especially to Enterprise customers. Cloudflare support was basically radio silent during this outage except for the status page. Realistically, they couldn't do much anyway. But at least any attempt at communication would be appreciated - especially for Enterprise customers, and even more especially after the post-mortem blames Flexential for a lack of communication. While I like Cloudflare since it's a great product, I think there are still a few more things that should be taken as a conclusion for CF to take away from this incident. That being said, glad you managed to recover, and thanks for the post-mortem.

vb-8448 3y ago
> Also one aspect I am missing here is the lack of communication - especially to Enterprise customers. They blame Flexential for lack of communication, but were the first one not saying anything.
- iAMkenough 3y ago
  Even "we don't know why our data center is failing, but we're sending a team over to physically investigate now" would have been A+ communication in the moment.
  - NicoJuicy 3y ago
    Everything was on the status page since the start? DC related updates: > Update - Power to Cloudflare’s core North America data center has been partially restored. Cloudflare has failed over some core services to a backup data center, which has partially remediated impact. Cloudflare is currently working to restore the remaining affected services and bring the core North America data center back online. Nov 02, 2023 - 17:08 UTC > Identified - Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. We will keep providing regular updates until the issue is resolved, thank you for your patience as we work on mitigating the problem. Nov 02, 2023 - 13:40 UTC

nickdothutton 3y ago

Consider modes of failure.

andyjohnson0 3y ago

> We are a relatively large customer of the facility, consuming approximately 10 percent of its total capacity. I'm surprised that CF are renting space in colocation facilities. I would have expected a business of their size to have their own DCs. Is this common practice for cloud providers?

NicoJuicy 3y ago
You thought that they build > 300 DC's? Colo is much more flexible, cheaper and quicker to start. Definitely since they sit close to the end-user on the data plane.
- andyjohnson0 3y ago
  > You thought that they build > 300 DC's? I have no idea how many DCs they have or operate in. Where does "300" come from? > Colo is much more flexible, cheaper and quicker to start. Definitely since they sit close to the end-user on the data plane. I understand that, but it has the disadvantage of reduced control and observability - particularly in the event of an outage such as that described in the blog post. I kind of assumed that top-tier cloud platforms like AWS/Azure/GCP operate out of dedicated DCs, and that CF are similar because of their well-known scale of operations. Since my original comment has been downvoted†, someone presumably thinks this it was a naive or trivial question - although I don't understand why. († I don't much care about downvotes, but I do take them to be a signal.)
  - martinald 3y ago
    https://www.cloudflare.com/network/ https://www.cloudflare.com/network/
  - NicoJuicy 3y ago
    Probably most of us follow Cloudflare a bit more closely. They want DC's close to every big city. I think most of us knew that they can't launch > 300 DC's in such a short amount of time. The many amount of DC's is mentioned a lot ( social networks, blogs, here). There is a distinction between eg. AWS / Azure / ... Which work with a couple of big DC's, while cloudflare operates more spread across more locations. You're comment did made me realize it may may not be that clear from an outsider viewpoint though ( fyi, I'm an outsider too)

ShadowRegent 3y ago

Interesting choice to spend the bulk of the article publicly shifting blame to a vendor by name and speculating on their root cause. Also an interesting choice to publicly call out that you're a whale in the facility and include an electrical diagram clearly marked Confidential by your vendor in the postmortem. Honestly, this is rather unprofessional. I understand and support explaining what triggered the event and giving a bit of context, but the focus on your postmortem needs to be on your incident, not your vendor's. Clearly, a lot went wrong and Flexential needs to do their own postmortem, but Cloudflare doesn't need to make guesses and do it for them, much less publicly.

creshal 3y ago
Especially since it shouldn't matter why the DC failed — Cloudflare's entire business model is selling services allegedly designed to survive that. 99% of the fault lies with Cloudflare for not being able to do their core job.
- corobo 3y ago
  In all fairness the rest of the article is about that
  - creshal 3y ago
    Slightly less than half, and the bottom half, so that people just skimming over it will mostly remember the DC operators' problems, not Cloudflare's own. This is very deliberately manipulative.
    - corobo 3y ago
      It is of course possible they've shuffled things around since this was posted but it seems that the first part addresses their system failings. 5th paragraph to the 9th are Cloudflare's "we buggered up" before they get to the power segment. They then continue with the "this is our fault for not being fully HA" after the power bit. Each to their own, I'm going to read it as a regular old post mortem on this one.

kurok 3y ago

So well known fails. Same was with one of our DC 3 years ago when they powered city and failed whole DC

technotarek 3y ago

True, not snippy: I found interesting that their automated billing emails seemed to arrive right on time.

nijave 3y ago
HA, maximum redundancy

thomasdeml 3y ago

Poor doc: You had a high availability 3 data center setup that utterly failed. Why spend the first third of the document blaming your data center operator? The management of the data center facility is outside of your control. You gambled that not appropriately testing your high-availability setup (under your control) would not have consequences. You should absolutely discuss the DC management with your operator, but that's between you and them and doesn't belong in this post mortem.

nabla9 3y ago

They don't always know it, but all large systems are moving gradually towards dependency management system with logic rules that covers "everything", physical, logical, human and administrative dependencies. Every time something new not covered is discovered, new rules and conditions are added. You can do it with manual checklists, multiple rule checkers, or put everything together. I suspect that in end it's just easier to put everything into single declarative formal verification system and see if new change to the system passes, transition between configurations passes etc.

lytedev 3y ago
This is such an interesting way of putting it. I think this has been the subconscious reason I've been gravitating towards defining _everything_ I manage personally (and not yet at work) with Nix. It's not quite to the extent you're talking about here, of course, but in a similar vein at least.

marcinzm 3y ago

> Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning A minor point but this feels like not the most efficient way to manage an emergency. Having some form of staggered shifts or other approach versus just having everyone pile on. If a lot of knowledge resided in specific individuals so they are vital to an effort like this and cannot be substituted then that seems like a risk in it's own.

someonehere 3y ago

I asked a sales rep once about services going out and how that would affect CF For Teams. They said it would be virtually impossible for CF to go down because of all their data centers around the world. Paraphrasing, “if there’s an outage, there’s definitely something going wrong with the internet.” And here we are. My trust in them has hit zero.

usr1106 3y ago
Why would you trust a sales rep? Even honest engineers cannot foresee the exact cascading consequences effects of such outages. Sales reps are not paid to be either competent on such issues nor to be honest.
cj 3y ago
FWIW, I'm a Cloudflare Enterprise customer and we had zero downtime. Only thing that was temporarily unavailable was the cloudflare dashboard. I feel like a lot of people in this thread are commenting under the impression that all of Cloudflare was down for 24 hours when in reality I wouldn't be surprised if a lot of customers were unaffected and unaware of the incident. I wouldn't even have known of the outage had it not been for HN..
- ianhawes 3y ago
  2nd this. We had zero downtime on anything in production. The only reason we knew is because we are actively standing up a transition to R2 and ran into errors configuring buckets.
belter 3y ago
Trust? Random sample from the last 60 days... Cloudflare outage – 24 hours now - https://news.ycombinator.com/item?id=38112515 https://news.ycombinator.com/item?id=38112515 Cloudflare Dashboard Logins Failing - https://news.ycombinator.com/item?id=38112230 https://news.ycombinator.com/item?id=38112230 Ask HN: Cloudflare Workers are down? - https://news.ycombinator.com/item?id=38074906 https://news.ycombinator.com/item?id=38074906 Cloudflare API, dashboard, tunnels down - https://news.ycombinator.com/item?id=38014582 https://news.ycombinator.com/item?id=38014582 Cloudflare Intermittent API Failures for Cloudflare Pages, Workers and Images - https://news.ycombinator.com/item?id=37819045 https://news.ycombinator.com/item?id=37819045 Cloudflare Issues with 1.1.1.1 public resolver and WARP - https://news.ycombinator.com/item?id=37762731 https://news.ycombinator.com/item?id=37762731 Cloudflare – Network Performance Issues - https://news.ycombinator.com/item?id=37604609 https://news.ycombinator.com/item?id=37604609 Cloudflare Issues Passing Challenge Pages - https://news.ycombinator.com/item?id=37336743 https://news.ycombinator.com/item?id=37336743

whoknowsidont 3y ago

>While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted. They're just going to straight up lie like that? We definitely weren't able to get "traffic through [their] network" through the outage at many different random points. So if the CF team is under the impression traffic was not impacted, dig deeper.

alberth 3y ago

Taking the positive outlook … Ensuring all of their services are fully distributed is now top of mind at CF. Ultimately, customers win if CF executes.

sackfield 3y ago

Its weird that upon reading this post, I have less confidence in Cloudflare. They basically browbeat Flexential for behaving unprofessional, which, yes, they probably did. However the fact that this causes entire systems that people rely on to go down is a massive redundancy failure on Cloudflares part, you should be able to nuke one of these datacentres and still maintain services. Very worrying is they start by stating their intended design: > Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon You need way more geographic dispersion than that, this control pane is used by people across the world. We are still on the intended design, not the flawed implementation by the way, which is wild to me. > This is a system design that we began implementing four years ago. While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. I don't understand why this would ever be done in this way. If Cloudflare is making a new product for consumers shouldn't redundant design be at the forefront here? I am surprised that it was even an option. For the record I do use Cloudflare for certain systems and I use it because I assume it has great failovers if events like this occur making me not have to worry about these eventualities, but now I will be reconsidering this, how do I actually know my cloudflare workers are safe from these design decisions? > When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. Yeh I'll bet, its because Cloudflares core design is not redundant. Really disappointed in this blog post trying to shift the blame to Flexential when this slapdash architecture should be the main problem on show. As a customer I don't care if Flexential disappears in an earthquake tomorrow, I expect Cloudflare to handle it gracefully.

onionisafruit 3y ago
Is the Hillsboro thing is about latency?
- DylanSp 3y ago
  That may well be part of it, some people were talking about the impact of latency in the outage thread [1]. [1] https://news.ycombinator.com/item?id=38113952 https://news.ycombinator.com/item?id=38113952

channer 3y ago

appreciate the status updates and the quick report. as a person that handles the tiniest datacenter, it's impossible to predict every potential event. best you can do is to recover as quickly as possible and learn the lesson. believing that this doesn't or can't happen to another vendor is being naive. it has happened to all of them and it'll happen again. can only hope it's super rare.

weird-eye-issue 3y ago

Wow they REALLY buried this important part didn't they! This took a ton of scrolling: "Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04." Bingo, there we have it.

dmix 3y ago
What does PDX-04 mean here? Not familiar with how data centers work.
- weird-eye-issue 3y ago
  Read the damn article! It's explained at the top.
  - deleted 3y ago
    [deleted]
- edaemon 3y ago
  PDX is the airport code for Portland, Oregon, USA. It's the fourth Portland data center.
dclowd9901 3y ago
Yep, past the part where they spent a long time blaming the data center and power company.
worksonmine 3y ago
Nah, if only the data center would've stayed up this wouldn't have been a problem. It's clearly on the data center. /s
indigomm 3y ago
In my experience, power is the most common data center failure there is. Often it's the redundant systems that cause the failure.

burroisolator 3y ago

While debatably unprofessional to blame your vendor, I found this read to be fascinating. I'm sure there are blog posts that detail how data centers work and fail but it's rare to get that cross over from a software engineering context. It puts into perspective what it takes for an average data center of this class to fail: power outage, generator failure, and then battery loss.

chippiewill 3y ago
I think what it really does is emphasise how common it is for crap to hit the fan when things go wrong - even with the best laid plans. The DC almost certainly advertises the redundant power supplies, generator backups and battery failover in order to get the customers. But probably doesn't do the legwork or spend the money to make those things truly reliable. It's a bit like having automated backups - but never testing them and discovering they're empty when they're really needed.

stemlord 3y ago

I'm ultimately glad this happened because it very effectively helps illustrate how we are assigning a centralized gatekeeper to the internet at the infrastructure level and why it's a bad thing.

simple10 3y ago

This is good reminder to myself to transfer domains registered on Cloudflare to another provider and only use Cloudflare for DNS or vice versa. I was effectively locked out of making any changes to domains registered and DNS hosted on Cloudflare during the entire outage due to single point of failure (Cloudflare) on my part.

jtchang 3y ago

For me it's basically summed up as "we didn't test turning the power off" and making sure things worked the way we planned. Yes it is hard and very expensive to do these types of tests. And doing it regularly is even more $$$ and time. As most customers we seem to be okay with a cheap price hidden behind a facade of "high availability" since I don't really want to pay for true HA. Because if I knew the real cost it would be too expensive.

AzzyHN 3y ago

To preface: I am not qualified to talk about this in the slightest. > Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04 What do you mean you discovered? How could you not know? Surely when you were setting this high availability cluster up years ago and migrating services over, you double checked that all crucial dependencies had also been moved, right? And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right???

gpm 3y ago
> And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right??? They discussed this. They had been running tests were they disabled the high availability cluster in any of (and of two of) the three DCs. That test didn't involve disabling the rest of the (non-HA) services from PDX-04 DC (oops).

madcadmium 3y ago

This is karma for their passive-aggressive blog post about Okta's security incident one week earlier

milkglass 3y ago

After reading this, we will be moving off Cloudflare. The post shows a lack of maturity and professionalism.

donutshop 3y ago

So the tl:dr is that they took a risk and never tested their high availability setup.

markhahn 3y ago

Shockingly poor choices by CF. We should absolutely blame them, just as "victims" of ransomware should be blamed. Hardening against system failure is the same process as security hardening.

throwoutway 3y ago

> Throughout the incident, Cloudflare's network and security services continued to work as expected. This seems misleading. Their own status page said the data plane was impacted across many services.

play168 3y ago

How you

tamimio 3y ago

Whenever we design and build a data center (in addition to disaster recovery, business continuity, etc.), we always test all possible scenarios of power interruptions and never assume that the power utility will get in touch with us in advance for a pre-planned maintenance, including the outcome AND the time to restore etc., it’s kind of shocking that you just assumed UPS will last 10min but never put it on test.. especially with size of CF!

zx8080 3y ago

Cloudflare Inc share price has increased 15% as a result. How does it work?

rymiel 3y ago

I truly and genuinely hope that this incident and the many before will drive away customers from Cloudflare's monopoly

zX41ZdbW 3y ago

You can run ClickHouse cluster across multiple datacenters. It will survive the failure of a single datacenter while being available for writes and reads, and the failure of two out of three datacenters while being available for reads. It works well when RTT between datacenters is less than 30 ms. If they are more distant, it will still work, but you will notice a quite high latency on INSERTs due to the distributed consensus. I've run a ClickHouse cluster with hundreds of bare-metal machines distributed across three datacenters in two countries at my previous job. It survived power failures (multiple), a flood (once), and network connectivity issues (regular). This cluster was used for logging and analytics :)

nixgeek 3y ago

Why would any supplier want to do business with Cloudflare now? You have 1.8MW of datacenter space to lease, you have a few interested parties, how could you not view Cloudflare as a huge reputational risk? Why even do the business? Why not lease that space to someone else? Moreover, why renew the existing deals with Cloudflare? Does Cloudflare have a plan to move 200+ racks in Oregon if that supplier decides just not to renew that deal whenever it comes up next? Are Cloudflare claiming they were able to build a technical plan which gets their architecture away from this site being a SPOF before the deal is up to renew, or is the CEO making a gamble here again? Cloudflare have demonstrated their willingness to create reputational issues for suppliers by publicly shaming two of them recently, and here, in only about 2 days from incident. One interpretation of this blog would be Cloudflare are a very unreasonable customer and one who is willing to post incomplete or informal information from their suppliers. Cloudflare also chose to focus the first half of a lengthy postmortem on blaming the supplier and only then on their own culpability for the outage, despite it clearly being a shared responsibility. One of the diagrams Cloudflare have posted is clearly marked "Proprietary and Confidential". Do Cloudflare have permission to post that? It's not clearly stated that they do. Should other suppliers expect when the sh*t hits the fan that any sensitive information they've shared will be part of a blog? Most of the "Lessons and Remediation" section is stuff Cloudflare could have worked on at any point in advance of a major incident, and Cloudflare's senior management have quite clearly chosen not to prioritize that work until today, when forced to by this major incident. When signing large deals, Cloudflare will frequently have to complete 'Supplier Disclosures', and they are also making claims through industry-standard certifications [1] like ISO, SOC and FedRAMP. Most of those will ask questions about the disaster recovery and business continuity plans and Cloudflare will have (repeatedly) attested they are adequate, something that this blog clearly demonstrates was a misrepresentation of their true capabilities. Will there be an SEC disclosure coming out of this considering it could have material impacts on the business, which is publicly traded? Was there any requirement that the SEC disclosure come first, or be concurrent with a blog? [1] https://www.cloudflare.com/trust-hub/compliance-resources/ https://www.cloudflare.com/trust-hub/compliance-resources/

ta1243 3y ago

TLDR: Major flooding, riots, earthquake, asteroid, nuke goes off in Portland and Cloudflare is down because they decided they could put their entire control infrastructure in a single location for ease of use. Centralising the web is such a great move.

GoAWAYS 3y ago

Is it common for CloudFlare to publish documents or diagrams from their vendors clearly marked "Proprietary and Confidential"

deleted 3y ago

[deleted]

ArrayQueue 3y ago

The guy I genuinely feel sorry for is "an unaccompanied technician who had only been on the job for a week". Regardless of any and all corporate spin on the issue, a newbie was dumped into an event at the worst possible time. I really really really hope he gets a decent bit of counselling to make sure that is fully aware that the issue with the data centre had NOTHING to do with him unplugging the coffee maker to plug in his recharger for his iPhone. Absolutely nothing at all.