15 ms·

Cloudflare outage on June 21, 2022

703 points by jgrahamc 4y ago

psim1 4y ago
CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.
- capableweb 4y ago
  I think a lot of companies don't realize the whole "Acknowledging our problems in public" thing CF got going for it is a positive. Lots of companies don't want to publish public post-mortems as they think it'll make them look weak rather than showing that they care about transparency in the face of failures/downtimes.
- deleted 4y ago
  [deleted]
- systemvoltage 4y ago
  Nerds in the executive office (CEO & CTO, etc). People just like us.
- homero 4y ago
  I'm also a huge fan
wondernine 4y ago
Part of the blog says : "In this time, we’ve converted 19 of our data centers to this architecture, internally called Multi-Colo PoP (MCP): Amsterdam, Atlanta, Ashburn, Chicago, Frankfurt, London, Los Angeles, Madrid, Manchester, Miami, Milan, Mumbai, Newark, Osaka, São Paulo, San Jose, Singapore, Sydney, Tokyo." Is the term MCP synonymous with "tier 1 PoPs" (mentioned elsewhere in other cloudflare blogs from time to time) or are the two terms referring to different things?
loist 4y ago
Seems that after this outage a lot of website that are behind Cloudflare NSs now gained top positions on Google SERP with strange links like http://domain/XX/yyyyyyy http://domain/XX/yyyyyyy Really strange, a coincidence?
samwillis 4y ago
In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
- thejosh 4y ago
  Yep, look at heroku and their big incident, and the amount of downtime they've had lately.
- viraptor 4y ago
  I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.
  - hunter2_ 4y ago
    > I wonder how much those posts are calculated and how much organic/culture related. Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
    - viraptor 4y ago
      No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
    - 4y ago
johnklos 4y ago
...and yet they still push so hard for recentralization of the web...
- samwillis 4y ago
  CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web". If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.
  - viraptor 4y ago
    It's often mentioned about AWS, especially when us-east-1 fails. The others are not big enough to affect basically "the internet" when they go down, so don't get pointed out as centralisation issues as much. And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.
  - johnklos 4y ago
    I think you've already drunk the Flavor Aid. What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization? Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize. It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't. Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?
    - samwillis 4y ago
      I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't. What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that. I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.
testplzignore 4y ago
Are there any steps that can be taken to test these types of changes in a non-production environment?
- vnkr 4y ago
  It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process: "There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."
thejosh 4y ago
07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch
- jgrahamc 4y ago
  Well, the "we can't reach these data centers at all and need to go through the break glass procedure" was pretty "ouch" also.
  - exikyut 4y ago
    I can('t) imagine, yikes. Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/ https://rachelbythebay.com/w/2019/01/20/quiet/ Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha. I would be very interested to hear more about how the break-glass process worked.
- michaelmior 4y ago
  This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.
  - scottlamb 4y ago
    I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.
rocky_raccoon 4y ago
Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake. Or you could go the opposite direction and risk turning something like this into a PR death spiral.
- can16358p 4y ago
  Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
  - rocky_raccoon 4y ago
    > I really don't get what's wrong with accepting mistakes, learning from them, and moving on. Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.
    - dylan604 4y ago
      Sure, this can be a good thing when it's a rare occurrence. If it is a weekly event, then you just start to look incompetent
  - coob 4y ago
    The exception that proves the rule with Apple: https://appleinsider.com/articles/12/09/28/apple-ceo-tim-cook-apologizes-to-customers-for-maps-in-ios-6 https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...
    - 4y ago
gcau 4y ago
Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.
- nielsole 4y ago
  > Talking about the outage as if it's reason for us to all ditch CF at time of writing no comment has done that except you.
  - gcau 4y ago
    I'm referring to other posts and discussions outside this website. I don't expect as much criticism in this post.
- Deukhoofd 4y ago
  It was a bit of a thing as people in Europe started their office work, and found out a lot of their internet services were down, and they were unable to access the things they needed. It's rather dangerous that we all depend on this one service being online.
- simiones 4y ago
  It is kind of a big deal to discover just how much of the Internet and the WWW is now dependent on CloudFlare. For their part, they handled this very well, and are to be commended (quick fix, quick explanation of failure). But you also can't help but see that they have a dangerous amount of control over such important systems.
leetrout 4y ago
BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.
- vnkr 4y ago
  That's what is suggested in the blogpost as one of future prevention plans.
- addingnumbers 4y ago
  That's the "commit-confirm" process they mention they will use in the write-up: > Primarily, we will be concentrating on automation improvements ... and provide an automated “commit-confirm” rollback.
  - Melatonic 4y ago
    Surprised everyone has not switched to this already - great idea
    - neuronexmachina 4y ago
      I assume there's some non-trivial caveats when using this with a widely-distributed system.
- antod 4y ago
  There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like: $ apply new rules; sleep 10; apply original rules If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
- DRW_ 4y ago
  Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled. I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
ransom1538 4y ago
Still seeing failed network calls. https://i.imgur.com/xHqvOzj.png https://i.imgur.com/xHqvOzj.png
- jgrahamc 4y ago
  Feel free to email me (jgc) details but based on that error I don't think that's us.
  - ransom1538 4y ago
    One more? Ill email too. https://i.imgur.com/Cxwv58g.png https://i.imgur.com/Cxwv58g.png
- zinekeller 4y ago
  Yeah, that's not Cloudflare at all (it's unlikely that CF still uses nginx/1.14).
- Jamie9912 4y ago
  Is that actually coming from Cloudflare? iirc Cloudflare reports it self as Cloudflare not nginx in the 5xx error pages
  - samwillis 4y ago
    The outage this morning manifested itself as a Nginx error page, somewhat unusually for CF.
  - lenn0x 4y ago
    correct, i saw that too. the outage returned 500/nginx. no version number either on footer. @jgrahamc thought that was strange too as few commenters last night were caught off guard trying to determine if it was their systems or cloudflare. supposedly its been forwarded along.
    - kc10 4y ago
      yes, there is definitely an nginx service in the path. We don't have any nginx in our infrastructure, but this was the response we had for our urls during the outage. <html> <head><title>500 Internal Server Error</title></head> <body bgcolor="white"> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>
Belphemur 4y ago
It's interesting that in 2022 we still have network issues caused by wrong order of rules. Everybody at one time experiences the dreaded REJECT not being at the end of the rule stack but just too early. Kudos to CF for such a good explanation of what caused the issue.
- ec109685 4y ago
  I wonder what tool the engineers used to view that diff. With a side by side one, it’s a bit more obvious when lines are reordered. Even better if the tool was syntax aware so it could highlight the different types of rules in unique colors.
xiwenc 4y ago
I’m surprised they did not conclude roll outs should be executed over longer period with smaller batches. When a system is complicated as theirs with so much impact, the only sane strategy is slow rolling updates so that you can hit the brake when needed.
- jgrahamc 4y ago
  That's literally one of the conclusions.
nerdbaggy 4y ago
Really interesting that 19 cities handle 50% of the requests.
- jgrahamc 4y ago
  Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.
  - worldofmatthew 4y ago
    If you have an efficient website, you can get decent performance to most of the world with one pop on the West cost of the USA.
- JCharante 4y ago
  Well half of those cities were in Asia during business hours, so given that the majority of humans live in Asia it makes sense. CF data centers in Asia also seem to be less distributed than in the West (e.g. Vietnam traffic seems to go to Singapore) meanwhile CF has multiple centers distributed throughout the US.
Tsiklon 4y ago
This is a great concise explanation. Thank you for providing it so quickly If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system? I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.
terom 4y ago
TODO: use commit-confirm for automated rollbacks Sounds like a good idea!
- buggeryorkshire 4y ago
  Is that the equivalent of Cisco 'save running config' with a timer? It's been many years so can't remember the exact incantations...
trollied 4y ago
Sounds like Cloudflare need a small low-traffic MCP that they can deploy to first.
sidcool 4y ago
This is a very nice write up.
dpz 4y ago
really appreciate the speed, detail and transparency of this post-mortem. Really one of, if not the best in the industry
rubatuga 4y ago
Uh, shouldn’t there be a staging environment for these sort of changes?
- alanning 4y ago
  Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed. One of their future changes is to include MCPs in their testing environments.
  - nijave 4y ago
    Ahh the old "dev doesn't quite match prod" issue
sharps_xp 4y ago
who will make the abstraction as a service we all need to protect us from config changes
- pigtailgirl 4y ago
  -- how much you willing to pay for said system? --
  - sharps_xp 4y ago
    depends on how guaranteed is your solution?
    - richardwhiuk 4y ago
      100%. You can never roll out any changes.
      - sharps_xp 4y ago
        would not buy, doesn't protect against initial config deployment.
        richardwhiuk 4y ago
        No, you can't roll that out either.
        sharps_xp 4y ago
        where do i send the check
kurtextrem 4y ago
Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...) - Google Cloud 2020: https://www.theregister.com/2020/12/16/google_europe_outage/ https://www.theregister.com/2020/12/16/google_europe_outage/ - IBM Cloud 2020: https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/ https://www.bleepingcomputer.com/news/technology/ibm-cloud-g... - Cloudflare 2019: https://news.ycombinator.com/item?id=20262214 https://news.ycombinator.com/item?id=20262214 - Amazon 2018: https://www.techtarget.com/searchsecurity/news/252439945/BGP-routing-security-flaw-caused-Amazon-Route-53-incident https://www.techtarget.com/searchsecurity/news/252439945/BGP... - AWS: https://www.thousandeyes.com/blog/route-leak-causes-amazon-and-aws-outage https://www.thousandeyes.com/blog/route-leak-causes-amazon-a... (2015) - Youtube: https://www.infoworld.com/article/2648947/youtube-outage-underscores-big-internet-problem.html https://www.infoworld.com/article/2648947/youtube-outage-und... (2008) And then there are incidents caused by hijacking: https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20user%20requests.-,Public%20incidents,-%5Bedit%5D https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20us...
- jve 4y ago
  Came here to say exactly this... things that mess with BGP have the power to wipe you off the internet. Some more: - Google 2016, configuration management bug/BGP: https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - Valve 2015: https://www.thousandeyes.com/blog/steam-outage-monitor-data-center-connectivity https://www.thousandeyes.com/blog/steam-outage-monitor-data-... - Cloudflare 2013: https://blog.cloudflare.com/todays-outage-post-mortem-82515/ https://blog.cloudflare.com/todays-outage-post-mortem-82515/
- simiones 4y ago
  Google in Japan 2017: https://www.internetsociety.org/blog/2017/08/google-leaked-prefixes-knocked-japan-off-internet/ https://www.internetsociety.org/blog/2017/08/google-leaked-p...
edf13 4y ago
It's nearly always BGP when this level of failure occurs.
- jgrahamc 4y ago
  I dunno man, you can really fuck things up with DNS also.
  - ngz00 4y ago
    I was on a severely understaffed edge team fronting several thousand engineers at a fortune 500 - every deploy felt like a spacex launch from my cubicle. I have a lot of reverence for the engineers who take on that kind of responsibility.
- star-glider 4y ago
  Generally speaking: You broke half the internet: BGP You broke half of your company's ability to access the internet: DNS
Cloudef 4y ago
ElectronShak 4y ago
What's it like to be an engineer designing and working on these systems? Must be sooo fulfiling! #Goals; Y'all are my heores!!
- jgrahamc 4y ago
  https://www.cloudflare.com/careers/ https://www.cloudflare.com/careers/
  - ElectronShak 4y ago
    Thanks, unfortunately I live in Africa, no roles yet for my location. I'll wait as I use the products :)
  - Icathian 4y ago
    I'm currently waiting on a recruiter to get my panel interviews scheduled. You guys are in "dream gig" territory for me. Any tips? ;-)
minecraftchest1 4y ago
Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around. I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.
- dane-pgp 4y ago
  Yeah, there's got to be some sweet spot between "formally verify all the things" and "i guess this diff looks okay, yolo!". I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them. The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.
ggalihpp 4y ago
The dns resolver also impacted and seems still have issue. We change to google dns and it solved. The problem is, we couldn't tell all our client they should change this :(
ruined 4y ago
happy solstice everyone
sschueller 4y ago
Nodejs is still having issues. For example: https://nodejs.org/dist/v16.15.1/node-v16.15.1-darwin-x64.tar.xz https://nodejs.org/dist/v16.15.1/node-v16.15.1-darwin-x64.ta... doesn't download if you do "n lts"
mproud 4y ago
If I use Cloudflare, what can I do — if anything — to avoid disruption when they go down?
- meibo 4y ago
  On the enterprise plans, you are able to set up your own DNS server that can route users away from Cloudflare, either to your origin or to another CDN/proxy.
  - nickjj 4y ago
    This comes with a major caveat. Your DNS host needs to support being able to assign a CNAME record on your root domain to a domain provided by Cloudflare. AWS Route 53 does not let you do this which I imagine is a decent chunk of enterprise clients. AWS only lets you alias records to AWS resources not external domains. With that said, even with enterprise in this case you would need to go all-in with Cloudlfare's nameservers or run the risk of not having DDoS protection on your root domain (ie. example.com wouldn't be protected but you could protect www.example.com since a CNAME with subdomains is a standard thing). However it's kind of interesting because an attacker could get the real IP of your root domain's AWS load balancer which is probably the same load balancer used for the `www` version of your site too, but now that they know your load balancer's IP they can completely bypass Cloudflare and go straight to your infrastructure. I'm pretty sure AWS doesn't let you assign an external domain with their aliases because they want you to pay them for AWS Shield Advanced instead of using Cloudflare because AWS knows getting an enterprise client to change their nameservers and all of their DNS records (potentially dozens of domains and multiple hundreds of records) is kind of a pain. It can be done but it's a friction point.
kache_ 4y ago
shit dawg i just woke up
ttul 4y ago
Every outage represents an opportunity to demonstrate resilience and ingenuity. Outages are guaranteed to happen. Might as well make the most of it to reveal something cool about their infrastructure.
weird-eye-issue 4y ago
One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudflare Teams], redirects, page rules, etc) I'll take these minor outages that make HN go apeshit. Of course it helped that most our traffic is from the US and this happened when it did but in the past week alone we served over 180 countries which Cloudflare helps make sure is nice and fast :D
- algo_trader 4y ago
  Could you possibly, kindly, mention which tools you use to track/buy/calculate conversions/revenue? Many thanks (Or DM the puppet email in my profile)
  - harrydehal 4y ago
    Not OP, but my team really, really enjoys using a combination of Segment.io for event tracking and piping that data into Amplitude for data viz, funnel metrics, A/B tests, conversion, etc.
  - weird-eye-issue 4y ago
    Our revenue is reported to us by our ad network and Amazon Associates. Basic events in Google Analytics let's us see various types of conversions
- herpderperator 4y ago
  I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?
  - jgrahamc 4y ago
    It wasn't a global outage.
_8j50 4y ago
Having been on the other side of similar outages, I am very impressee at their response timeline.
thesuitonym 4y ago
Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?
- ThaDood 4y ago
  DN42 <https://dn42.eu/Home https://dn42.eu/Home> gets mentioned a lot. Its basically a big dynamic VPN that you can do BGP stuff with. Pretty cool but I could never get my node working properly.
  - bpye 4y ago
    I started setting that up and totally forgot, maybe I should actually try and peer with someone.
- jamal-kumar 4y ago
  Nah Cisco has labs you can download and learn for their networking certifications, which are kinda the standard. Networking talent is kind of hard to find and if you learn that your chances of employment get pretty high.
- _whiteCaps_ 4y ago
  https://github.com/Exa-Networks/exabgp https://github.com/Exa-Networks/exabgp They've got some Docker examples in the README.
- nonameiguess 4y ago
  You can learn BGP with mininet: https://mininet.org/ https://mininet.org/ You can simulate arbitrarily large networks and internetworks with this, provided you have the hardware to run a large enough number of virtual appliances, but they are pretty lightweight.
  - Icathian 4y ago
    Mininet is what the Georgia Tech OMSCS Computer Networking labs use. It's not bad, the two labs that stood out to me were using it to implement BGP and a Distance Vector Routing protocol.
junon 4y ago
Now this is a post mortem.
throwaway_uke 4y ago
i'm gonna go with the less popular view here that overly detailed post mortems do little in the grand scheme of things other than satisfy tech p0rn for a tiny, highly technical audience. does wonders for hiring indeed. sure, transparency is better than "something went wrong, we take this very seriously, sorry." (although the non technical crowd couldn't care less) only people who dont do anything make no mistakes, but doing such highly impactful changes so quickly (inside one day!) for where 50% of traffic happens seems a huge red flag to me, no matter the procedure and safety valves.
kylegalbraith 4y ago
As others have said, this is a clear and concise write up of the incident. That is underlined even more when you take into account how quickly they published this. I have seen some companies take weeks or even months to publish an analysis that is half as good as this. Not trying to take the light away from the outage, the outage was bad. But the relative quickness to recovery is pretty impressive, in my opinion. Sounds like they could have recovered even quicker if not for a bit of toe stepping that happened.
- april_22 4y ago
  I think it's even better that they explained the backgorund of the outage in a really easy to understand way, so that not only experts can get a hang of what was happening.
malikNF 4y ago
off-topic-ish, this post on /r/ProgrammerHumor gave me a chuckle https://www.reddit.com/r/ProgrammerHumor/comments/vh9peo/just_pushed_my_first_pr_for_my_new_job_at/ https://www.reddit.com/r/ProgrammerHumor/comments/vh9peo/jus...
- jgrahamc 4y ago
  That made me smile.
thomashabets2 4y ago
tl;dr: Another BGP outage due to bad config changes. Here's a somewhat old (2016) but very impressive system at a major ISP for avoiding exactly this: https://www.youtube.com/watch?v=R_vCdGkGeSk https://www.youtube.com/watch?v=R_vCdGkGeSk
CodeWriter23 4y ago
Gotta hand it to them, a shining example of transparency and taking responsibility for mistakes.
asadlionpk 4y ago
Been a fan of CF since they were an essential for DDOS protection for various Wordpress sites I deployed back then. I buy more NET every time I see posts like this.
- malfist 4y ago
  Hackernews isn't wallstreetbets.
philipwhiuk 4y ago
Is there no system to unit test a rule-set?
DustinBrett 4y ago
I wish computers could stop us from making these kinds of mistakes without turning into Skynet.
lilyball 4y ago
They said they ran a dry-run. What did that do, just generate these diffs? I would have expected them to have some way of simulating the network for BGP changes in order to verify that they didn't just fuck up their traffic.
xtat 4y ago
Feels a little disingenuous to use the first 3/4 of the report to advertise.
grenbys 4y ago
Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect? Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story with reverting reverts? Adding more automation, tests and fixing that specific ordering issue of course is an improvement. But that adds more complexity and any automation ultimately will fail some day. Technical details are all appreciated. But it is going to be something else next time. Would be great to learn more about human interactions. That's where the resilience of a socio-technical system happened and I bet there is some room for improvement there.
- systemvoltage 4y ago
  It would be fun to be a fly on the wall when shit hits the fan in general. From Nuclear meltdowns to 9/11 ATC recordings, it is fascinating to see how emergencies play out and what kind of things go on with boots-on-ground, all-hands-on-deck situations. Like, does Cloudflare have an emergency procedure for escalation? What does that look like? How does the CTO get woken up in the middle of the night? How to get in touch with critical and most important engineers? Who noticed Cloudflare down first? How do quick decisions get made and decided? Do people get on a giant zoom call? Or emails going around? What if they can't get hold of the most important people that can flip switches? Do they have a control room like the movies? CTO looking over the shoulder calling "Affirmative, apply the fix." followed by a progress bar painfully moving towards completion.
  - matsur 4y ago
    We spend a lot of time and thought building out our incident management processes and tooling. We were not making things up as we went last night. https://sre.google/resources/book-update/managing-incidents/ https://sre.google/resources/book-update/managing-incidents/ is Google focused, but our flavor of incident response is not too far off.
- nijave 4y ago
keyle 4y ago
How did no one at cloudflare think that this MCP thing should be part of the staging rollout? I imagine that was part of a // TODO. It sounds like it's a key architectural part of the system that "[...] convert all of our busiest locations to a more flexible and resilient architecture." 25 year experience and it's always the things that are supposed to make us "more flexible" and "more resilient" or robust/stable/safer <keyword> that ends up royally f'ing us where the light don't shine.
jiggawatts 4y ago
The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here. What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection. Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one. This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds. Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts! (I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)
- inferiorhuman 4y ago
  The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network. Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).
  - jiggawatts 4y ago
    Again, you are talking about safety systems and mitigations that device customers “should” implement bespoke themselves. I’m saying this is the problem — the device configuration approach should be safe by default.
    - inferiorhuman 4y ago
      Yes/no – something like a watchdog timer should be part of whatever OS the router is running. Remote power management would be closer to bespoke but ought not to be too exotic.
drfrank 4y ago
Naively, it seems to me that there should at least be a warning somewhere if there are declarations after a REJECT-THE-REST. I'm not familiar with whatever language this is, but wouldn't such a construct always indicate something was being ignored?
llama052 4y ago
We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense of urgency on updating their status page. (Looking at you Azure)
- dmix 4y ago
  > I wish more companies would take note of the transparency and sense of urgency on updating their status page. Looking at you Twilio...
  - sarathyweb 4y ago
    Looking at you, AWS
    - onphonenow 4y ago
      Looking at you zoom?
- daniel-grigg 4y ago
  Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.
  - paranoidrobot 4y ago
    > Here in Australia the status page was inaccessible. Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.
    - Maxious 4y ago
js2 4y ago
Ah, this is why iCloud Private Relay wasn't working this morning.
mikewang 4y ago
I read the blog twice and have some thoughts: The root cause seems is as: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes." And a dry-run: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure." And a Peer review: "Before it was allowed to go out, it was also peer reviewed by multiple engineers. " I would doubt the expertise of tech guys of cloudflare, reviewing the change. And there was a dry-run. But is it really OK to apply the change to a spine network which would affect 50% network traffic? Just out of peer review and a dry run? No green/blue, no gray release, maybe these are not proper for a small change here. But this "small" change really got big affect. I thougt it was worth it. And from my shallow experience, the dry-run would always have do nothing to the env. It is dry-run anyway. And at last the three lines are found out. So I wonder how did this re-order happen? And why? With these tiny changes, there should be some mechanism to verify their correctness, not just review and dry-run.
- matsur 4y ago
  We use a phased rollout process for all routine changes (like this one). Once a change has passed peer review and the "dry-run", changes are rolled out to progressively larger slices of our production environment, with monitoring systems and engineers watching for adverse effects. The specific network locations that were impacted by this change were amongst the last to see the change rolled out. One deficiency in our deployment strategy (which we will correct) is that no network locations in the affected "MCP" configuration received the change early in our rollout process. If that had been the case, we would have found the problem much earlier and the incident's impact would have been much reduced.
kebab-case 4y ago
I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company. Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases? One crude approach we were discussing is during an outage we could change the NS records in the registrar to point to for eg. Google Cloud DNS which would already be in sync in terms of the DNS records it has.
- juancb 4y ago
  Cedexis now part of Citrix offers a multi-cdn product, that allows you to load balance between CDNs, check them out. I'm not sure who else is in that space, but in general it sounds like you want a multi-cdn strategy.
- bandwidth 4y ago
  If you're okay with load balancing DNS queries across multiple providers you could do 2x Cloudflare primary NS, 2x GCP for example where each provider is in sync with each other. If not, a manual swap at the registrar level would be good enough. I should also mention this approach sort of breaks with Cloudflare's proxied records which dynamically assign anycast IPs for records placed on their CDN. So if using this approach the failover NS provider would probably need to also use a different CDN, preferably one that just gives you a CNAME.
  - deleted 4y ago
    [deleted]
AtNightWeCode 4y ago
Would be nice to have some automation that one could use for keeping track of health status of cloud services. Status API, webhook solution, something. Maybe even a standard for it. Or a service that monitors all major cloud services. We did get alarms. Our things partially worked though so CF was not the first thing to check.
devonkim 4y ago
Most of the criticisms seem to be around BGP and network management. What I’m seeing here that also is important is that the change was applied to a DC where the route change didn’t trigger the defect. In essence, this is also due to a very classic problem of a test dataset giving a false sense of security due to variation from other configurations. For this reason my team prefers to rollout changes to production using a test region that most customers don’t use yet will have some visible impact if there’s any error in our presumptions so far such as hard-coding regions and relying upon services not present or as capable across all regions. This practice has caught a number of rather serious errors for us that while customer impacting was nowhere near as bad as if we had rolled out simply randomly like many teams do essentially. This is even more important the more difficult it is to perform rollbacks of changes or for rollbacks to take effect such as DNS and CDN caching changes.

703 points by jgrahamc 4y ago

CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.

capableweb 4y ago
I think a lot of companies don't realize the whole "Acknowledging our problems in public" thing CF got going for it is a positive. Lots of companies don't want to publish public post-mortems as they think it'll make them look weak rather than showing that they care about transparency in the face of failures/downtimes.
deleted 4y ago
[deleted]
systemvoltage 4y ago
Nerds in the executive office (CEO & CTO, etc). People just like us.
homero 4y ago
I'm also a huge fan

wondernine 4y ago

Part of the blog says : "In this time, we’ve converted 19 of our data centers to this architecture, internally called Multi-Colo PoP (MCP): Amsterdam, Atlanta, Ashburn, Chicago, Frankfurt, London, Los Angeles, Madrid, Manchester, Miami, Milan, Mumbai, Newark, Osaka, São Paulo, San Jose, Singapore, Sydney, Tokyo." Is the term MCP synonymous with "tier 1 PoPs" (mentioned elsewhere in other cloudflare blogs from time to time) or are the two terms referring to different things?

loist 4y ago

Seems that after this outage a lot of website that are behind Cloudflare NSs now gained top positions on Google SERP with strange links like http://domain/XX/yyyyyyy http://domain/XX/yyyyyyy Really strange, a coincidence?

samwillis 4y ago

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

thejosh 4y ago
Yep, look at heroku and their big incident, and the amount of downtime they've had lately.
viraptor 4y ago
I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.
- hunter2_ 4y ago
  > I wonder how much those posts are calculated and how much organic/culture related. Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
  - viraptor 4y ago
    No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
  - 4y ago

johnklos 4y ago

...and yet they still push so hard for recentralization of the web...

samwillis 4y ago
CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web". If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.
- viraptor 4y ago
  It's often mentioned about AWS, especially when us-east-1 fails. The others are not big enough to affect basically "the internet" when they go down, so don't get pointed out as centralisation issues as much. And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.
- johnklos 4y ago
  I think you've already drunk the Flavor Aid. What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization? Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize. It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't. Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?
  - samwillis 4y ago
    I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't. What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that. I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.

testplzignore 4y ago

Are there any steps that can be taken to test these types of changes in a non-production environment?

vnkr 4y ago
It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process: "There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."

thejosh 4y ago

07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. Ouch

jgrahamc 4y ago
Well, the "we can't reach these data centers at all and need to go through the break glass procedure" was pretty "ouch" also.
- exikyut 4y ago
  I can('t) imagine, yikes. Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/ https://rachelbythebay.com/w/2019/01/20/quiet/ Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha. I would be very interested to hear more about how the break-glass process worked.
michaelmior 4y ago
This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.
- scottlamb 4y ago
  I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.

rocky_raccoon 4y ago

Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake. Or you could go the opposite direction and risk turning something like this into a PR death spiral.

can16358p 4y ago
Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF). At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it? I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
- rocky_raccoon 4y ago
  > I really don't get what's wrong with accepting mistakes, learning from them, and moving on. Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.
  - dylan604 4y ago
    Sure, this can be a good thing when it's a rare occurrence. If it is a weekly event, then you just start to look incompetent
- coob 4y ago
  The exception that proves the rule with Apple: https://appleinsider.com/articles/12/09/28/apple-ceo-tim-cook-apologizes-to-customers-for-maps-in-ios-6 https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...
  - 4y ago

gcau 4y ago

Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.

nielsole 4y ago
> Talking about the outage as if it's reason for us to all ditch CF at time of writing no comment has done that except you.
- gcau 4y ago
  I'm referring to other posts and discussions outside this website. I don't expect as much criticism in this post.
Deukhoofd 4y ago
It was a bit of a thing as people in Europe started their office work, and found out a lot of their internet services were down, and they were unable to access the things they needed. It's rather dangerous that we all depend on this one service being online.
simiones 4y ago
It is kind of a big deal to discover just how much of the Internet and the WWW is now dependent on CloudFlare. For their part, they handled this very well, and are to be commended (quick fix, quick explanation of failure). But you also can't help but see that they have a dangerous amount of control over such important systems.

leetrout 4y ago

BGP changes should be like the display resolution changes on your PC... It should revert as a failsafe if not confirmed within X minutes.

vnkr 4y ago
That's what is suggested in the blogpost as one of future prevention plans.
addingnumbers 4y ago
That's the "commit-confirm" process they mention they will use in the write-up: > Primarily, we will be concentrating on automation improvements ... and provide an automated “commit-confirm” rollback.
- Melatonic 4y ago
  Surprised everyone has not switched to this already - great idea
  - neuronexmachina 4y ago
    I assume there's some non-trivial caveats when using this with a widely-distributed system.
antod 4y ago
There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like: $ apply new rules; sleep 10; apply original rules If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
DRW_ 4y ago
Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled. I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.

ransom1538 4y ago

Still seeing failed network calls. https://i.imgur.com/xHqvOzj.png https://i.imgur.com/xHqvOzj.png

jgrahamc 4y ago
Feel free to email me (jgc) details but based on that error I don't think that's us.
- ransom1538 4y ago
  One more? Ill email too. https://i.imgur.com/Cxwv58g.png https://i.imgur.com/Cxwv58g.png
zinekeller 4y ago
Yeah, that's not Cloudflare at all (it's unlikely that CF still uses nginx/1.14).
Jamie9912 4y ago
Is that actually coming from Cloudflare? iirc Cloudflare reports it self as Cloudflare not nginx in the 5xx error pages
- samwillis 4y ago
  The outage this morning manifested itself as a Nginx error page, somewhat unusually for CF.
- lenn0x 4y ago
  correct, i saw that too. the outage returned 500/nginx. no version number either on footer. @jgrahamc thought that was strange too as few commenters last night were caught off guard trying to determine if it was their systems or cloudflare. supposedly its been forwarded along.
  - kc10 4y ago
    yes, there is definitely an nginx service in the path. We don't have any nginx in our infrastructure, but this was the response we had for our urls during the outage. <html> <head><title>500 Internal Server Error</title></head> <body bgcolor="white"> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>

Belphemur 4y ago

It's interesting that in 2022 we still have network issues caused by wrong order of rules. Everybody at one time experiences the dreaded REJECT not being at the end of the rule stack but just too early. Kudos to CF for such a good explanation of what caused the issue.

ec109685 4y ago
I wonder what tool the engineers used to view that diff. With a side by side one, it’s a bit more obvious when lines are reordered. Even better if the tool was syntax aware so it could highlight the different types of rules in unique colors.

xiwenc 4y ago

I’m surprised they did not conclude roll outs should be executed over longer period with smaller batches. When a system is complicated as theirs with so much impact, the only sane strategy is slow rolling updates so that you can hit the brake when needed.

jgrahamc 4y ago
That's literally one of the conclusions.

nerdbaggy 4y ago

Really interesting that 19 cities handle 50% of the requests.

jgrahamc 4y ago
Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.
- worldofmatthew 4y ago
  If you have an efficient website, you can get decent performance to most of the world with one pop on the West cost of the USA.
JCharante 4y ago
Well half of those cities were in Asia during business hours, so given that the majority of humans live in Asia it makes sense. CF data centers in Asia also seem to be less distributed than in the West (e.g. Vietnam traffic seems to go to Singapore) meanwhile CF has multiple centers distributed throughout the US.

Tsiklon 4y ago

This is a great concise explanation. Thank you for providing it so quickly If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system? I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.

terom 4y ago

TODO: use commit-confirm for automated rollbacks Sounds like a good idea!

buggeryorkshire 4y ago
Is that the equivalent of Cisco 'save running config' with a timer? It's been many years so can't remember the exact incantations...

trollied 4y ago

Sounds like Cloudflare need a small low-traffic MCP that they can deploy to first.

sidcool 4y ago

This is a very nice write up.

dpz 4y ago

really appreciate the speed, detail and transparency of this post-mortem. Really one of, if not the best in the industry

rubatuga 4y ago

Uh, shouldn’t there be a staging environment for these sort of changes?

alanning 4y ago
Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed. One of their future changes is to include MCPs in their testing environments.
- nijave 4y ago
  Ahh the old "dev doesn't quite match prod" issue

sharps_xp 4y ago

who will make the abstraction as a service we all need to protect us from config changes

pigtailgirl 4y ago
-- how much you willing to pay for said system? --
- sharps_xp 4y ago
  depends on how guaranteed is your solution?
  - richardwhiuk 4y ago
    100%. You can never roll out any changes.
    - sharps_xp 4y ago
      would not buy, doesn't protect against initial config deployment.
      - richardwhiuk 4y ago
        No, you can't roll that out either.
        sharps_xp 4y ago
        where do i send the check

kurtextrem 4y ago

Yet another BGP caused outage. At some point we should collect all of them: - Cloudflare 2022 (this one) - Facebook 2021: https://news.ycombinator.com/item?id=28752131 https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie - (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...) - Google Cloud 2020: https://www.theregister.com/2020/12/16/google_europe_outage/ https://www.theregister.com/2020/12/16/google_europe_outage/ - IBM Cloud 2020: https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/ https://www.bleepingcomputer.com/news/technology/ibm-cloud-g... - Cloudflare 2019: https://news.ycombinator.com/item?id=20262214 https://news.ycombinator.com/item?id=20262214 - Amazon 2018: https://www.techtarget.com/searchsecurity/news/252439945/BGP-routing-security-flaw-caused-Amazon-Route-53-incident https://www.techtarget.com/searchsecurity/news/252439945/BGP... - AWS: https://www.thousandeyes.com/blog/route-leak-causes-amazon-and-aws-outage https://www.thousandeyes.com/blog/route-leak-causes-amazon-a... (2015) - Youtube: https://www.infoworld.com/article/2648947/youtube-outage-underscores-big-internet-problem.html https://www.infoworld.com/article/2648947/youtube-outage-und... (2008) And then there are incidents caused by hijacking: https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20user%20requests.-,Public%20incidents,-%5Bedit%5D https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20us...

jve 4y ago
Came here to say exactly this... things that mess with BGP have the power to wipe you off the internet. Some more: - Google 2016, configuration management bug/BGP: https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - Valve 2015: https://www.thousandeyes.com/blog/steam-outage-monitor-data-center-connectivity https://www.thousandeyes.com/blog/steam-outage-monitor-data-... - Cloudflare 2013: https://blog.cloudflare.com/todays-outage-post-mortem-82515/ https://blog.cloudflare.com/todays-outage-post-mortem-82515/
simiones 4y ago
Google in Japan 2017: https://www.internetsociety.org/blog/2017/08/google-leaked-prefixes-knocked-japan-off-internet/ https://www.internetsociety.org/blog/2017/08/google-leaked-p...

edf13 4y ago

It's nearly always BGP when this level of failure occurs.

jgrahamc 4y ago
I dunno man, you can really fuck things up with DNS also.
- ngz00 4y ago
  I was on a severely understaffed edge team fronting several thousand engineers at a fortune 500 - every deploy felt like a spacex launch from my cubicle. I have a lot of reverence for the engineers who take on that kind of responsibility.
star-glider 4y ago
Generally speaking: You broke half the internet: BGP You broke half of your company's ability to access the internet: DNS

Cloudef 4y ago

ElectronShak 4y ago

What's it like to be an engineer designing and working on these systems? Must be sooo fulfiling! #Goals; Y'all are my heores!!

jgrahamc 4y ago
https://www.cloudflare.com/careers/ https://www.cloudflare.com/careers/
- ElectronShak 4y ago
  Thanks, unfortunately I live in Africa, no roles yet for my location. I'll wait as I use the products :)
- Icathian 4y ago
  I'm currently waiting on a recruiter to get my panel interviews scheduled. You guys are in "dream gig" territory for me. Any tips? ;-)

minecraftchest1 4y ago

Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around. I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.

dane-pgp 4y ago
Yeah, there's got to be some sweet spot between "formally verify all the things" and "i guess this diff looks okay, yolo!". I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them. The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.

ggalihpp 4y ago

The dns resolver also impacted and seems still have issue. We change to google dns and it solved. The problem is, we couldn't tell all our client they should change this :(

ruined 4y ago

happy solstice everyone

sschueller 4y ago

Nodejs is still having issues. For example: https://nodejs.org/dist/v16.15.1/node-v16.15.1-darwin-x64.tar.xz https://nodejs.org/dist/v16.15.1/node-v16.15.1-darwin-x64.ta... doesn't download if you do "n lts"

mproud 4y ago

If I use Cloudflare, what can I do — if anything — to avoid disruption when they go down?

meibo 4y ago
On the enterprise plans, you are able to set up your own DNS server that can route users away from Cloudflare, either to your origin or to another CDN/proxy.
- nickjj 4y ago
  This comes with a major caveat. Your DNS host needs to support being able to assign a CNAME record on your root domain to a domain provided by Cloudflare. AWS Route 53 does not let you do this which I imagine is a decent chunk of enterprise clients. AWS only lets you alias records to AWS resources not external domains. With that said, even with enterprise in this case you would need to go all-in with Cloudlfare's nameservers or run the risk of not having DDoS protection on your root domain (ie. example.com wouldn't be protected but you could protect www.example.com since a CNAME with subdomains is a standard thing). However it's kind of interesting because an attacker could get the real IP of your root domain's AWS load balancer which is probably the same load balancer used for the `www` version of your site too, but now that they know your load balancer's IP they can completely bypass Cloudflare and go straight to your infrastructure. I'm pretty sure AWS doesn't let you assign an external domain with their aliases because they want you to pay them for AWS Shield Advanced instead of using Cloudflare because AWS knows getting an enterprise client to change their nameservers and all of their DNS records (potentially dozens of domains and multiple hundreds of records) is kind of a pain. It can be done but it's a friction point.

kache_ 4y ago

shit dawg i just woke up

ttul 4y ago

Every outage represents an opportunity to demonstrate resilience and ingenuity. Outages are guaranteed to happen. Might as well make the most of it to reveal something cool about their infrastructure.

weird-eye-issue 4y ago

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudflare Teams], redirects, page rules, etc) I'll take these minor outages that make HN go apeshit. Of course it helped that most our traffic is from the US and this happened when it did but in the past week alone we served over 180 countries which Cloudflare helps make sure is nice and fast :D

algo_trader 4y ago
Could you possibly, kindly, mention which tools you use to track/buy/calculate conversions/revenue? Many thanks (Or DM the puppet email in my profile)
- harrydehal 4y ago
  Not OP, but my team really, really enjoys using a combination of Segment.io for event tracking and piping that data into Amplitude for data viz, funnel metrics, A/B tests, conversion, etc.
- weird-eye-issue 4y ago
  Our revenue is reported to us by our ad network and Amazon Associates. Basic events in Google Analytics let's us see various types of conversions
herpderperator 4y ago
I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?
- jgrahamc 4y ago
  It wasn't a global outage.

_8j50 4y ago

Having been on the other side of similar outages, I am very impressee at their response timeline.

thesuitonym 4y ago

Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?

ThaDood 4y ago
DN42 <https://dn42.eu/Home https://dn42.eu/Home> gets mentioned a lot. Its basically a big dynamic VPN that you can do BGP stuff with. Pretty cool but I could never get my node working properly.
- bpye 4y ago
  I started setting that up and totally forgot, maybe I should actually try and peer with someone.
jamal-kumar 4y ago
Nah Cisco has labs you can download and learn for their networking certifications, which are kinda the standard. Networking talent is kind of hard to find and if you learn that your chances of employment get pretty high.
_whiteCaps_ 4y ago
https://github.com/Exa-Networks/exabgp https://github.com/Exa-Networks/exabgp They've got some Docker examples in the README.
nonameiguess 4y ago
You can learn BGP with mininet: https://mininet.org/ https://mininet.org/ You can simulate arbitrarily large networks and internetworks with this, provided you have the hardware to run a large enough number of virtual appliances, but they are pretty lightweight.
- Icathian 4y ago
  Mininet is what the Georgia Tech OMSCS Computer Networking labs use. It's not bad, the two labs that stood out to me were using it to implement BGP and a Distance Vector Routing protocol.

junon 4y ago

Now this is a post mortem.

throwaway_uke 4y ago

i'm gonna go with the less popular view here that overly detailed post mortems do little in the grand scheme of things other than satisfy tech p0rn for a tiny, highly technical audience. does wonders for hiring indeed. sure, transparency is better than "something went wrong, we take this very seriously, sorry." (although the non technical crowd couldn't care less) only people who dont do anything make no mistakes, but doing such highly impactful changes so quickly (inside one day!) for where 50% of traffic happens seems a huge red flag to me, no matter the procedure and safety valves.

kylegalbraith 4y ago

As others have said, this is a clear and concise write up of the incident. That is underlined even more when you take into account how quickly they published this. I have seen some companies take weeks or even months to publish an analysis that is half as good as this. Not trying to take the light away from the outage, the outage was bad. But the relative quickness to recovery is pretty impressive, in my opinion. Sounds like they could have recovered even quicker if not for a bit of toe stepping that happened.

april_22 4y ago
I think it's even better that they explained the backgorund of the outage in a really easy to understand way, so that not only experts can get a hang of what was happening.

malikNF 4y ago

off-topic-ish, this post on /r/ProgrammerHumor gave me a chuckle https://www.reddit.com/r/ProgrammerHumor/comments/vh9peo/just_pushed_my_first_pr_for_my_new_job_at/ https://www.reddit.com/r/ProgrammerHumor/comments/vh9peo/jus...

jgrahamc 4y ago
That made me smile.

thomashabets2 4y ago

tl;dr: Another BGP outage due to bad config changes. Here's a somewhat old (2016) but very impressive system at a major ISP for avoiding exactly this: https://www.youtube.com/watch?v=R_vCdGkGeSk https://www.youtube.com/watch?v=R_vCdGkGeSk

CodeWriter23 4y ago

Gotta hand it to them, a shining example of transparency and taking responsibility for mistakes.

asadlionpk 4y ago

Been a fan of CF since they were an essential for DDOS protection for various Wordpress sites I deployed back then. I buy more NET every time I see posts like this.

malfist 4y ago
Hackernews isn't wallstreetbets.

philipwhiuk 4y ago

Is there no system to unit test a rule-set?

DustinBrett 4y ago

I wish computers could stop us from making these kinds of mistakes without turning into Skynet.

lilyball 4y ago

They said they ran a dry-run. What did that do, just generate these diffs? I would have expected them to have some way of simulating the network for BGP changes in order to verify that they didn't just fuck up their traffic.

xtat 4y ago

Feels a little disingenuous to use the first 3/4 of the report to advertise.

grenbys 4y ago

Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect? Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story with reverting reverts? Adding more automation, tests and fixing that specific ordering issue of course is an improvement. But that adds more complexity and any automation ultimately will fail some day. Technical details are all appreciated. But it is going to be something else next time. Would be great to learn more about human interactions. That's where the resilience of a socio-technical system happened and I bet there is some room for improvement there.

systemvoltage 4y ago
It would be fun to be a fly on the wall when shit hits the fan in general. From Nuclear meltdowns to 9/11 ATC recordings, it is fascinating to see how emergencies play out and what kind of things go on with boots-on-ground, all-hands-on-deck situations. Like, does Cloudflare have an emergency procedure for escalation? What does that look like? How does the CTO get woken up in the middle of the night? How to get in touch with critical and most important engineers? Who noticed Cloudflare down first? How do quick decisions get made and decided? Do people get on a giant zoom call? Or emails going around? What if they can't get hold of the most important people that can flip switches? Do they have a control room like the movies? CTO looking over the shoulder calling "Affirmative, apply the fix." followed by a progress bar painfully moving towards completion.
- matsur 4y ago
  We spend a lot of time and thought building out our incident management processes and tooling. We were not making things up as we went last night. https://sre.google/resources/book-update/managing-incidents/ https://sre.google/resources/book-update/managing-incidents/ is Google focused, but our flavor of incident response is not too far off.
nijave 4y ago

keyle 4y ago

How did no one at cloudflare think that this MCP thing should be part of the staging rollout? I imagine that was part of a // TODO. It sounds like it's a key architectural part of the system that "[...] convert all of our busiest locations to a more flexible and resilient architecture." 25 year experience and it's always the things that are supposed to make us "more flexible" and "more resilient" or robust/stable/safer <keyword> that ends up royally f'ing us where the light don't shine.

jiggawatts 4y ago

The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous. I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here. What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection. Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one. This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds. Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts! (I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)

inferiorhuman 4y ago
The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network. Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).
- jiggawatts 4y ago
  Again, you are talking about safety systems and mitigations that device customers “should” implement bespoke themselves. I’m saying this is the problem — the device configuration approach should be safe by default.
  - inferiorhuman 4y ago
    Yes/no – something like a watchdog timer should be part of whatever OS the router is running. Remote power management would be closer to bespoke but ought not to be too exotic.

drfrank 4y ago

Naively, it seems to me that there should at least be a warning somewhere if there are declarations after a REJECT-THE-REST. I'm not familiar with whatever language this is, but wouldn't such a construct always indicate something was being ignored?

llama052 4y ago

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense of urgency on updating their status page. (Looking at you Azure)

dmix 4y ago
> I wish more companies would take note of the transparency and sense of urgency on updating their status page. Looking at you Twilio...
- sarathyweb 4y ago
  Looking at you, AWS
  - onphonenow 4y ago
    Looking at you zoom?
daniel-grigg 4y ago
Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.
- paranoidrobot 4y ago
  > Here in Australia the status page was inaccessible. Me and a bunch of my colleagues were all able to access it (different ISPs) on the east coast.
  - Maxious 4y ago

js2 4y ago

Ah, this is why iCloud Private Relay wasn't working this morning.

mikewang 4y ago

I read the blog twice and have some thoughts: The root cause seems is as: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes." And a dry-run: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure." And a Peer review: "Before it was allowed to go out, it was also peer reviewed by multiple engineers. " I would doubt the expertise of tech guys of cloudflare, reviewing the change. And there was a dry-run. But is it really OK to apply the change to a spine network which would affect 50% network traffic? Just out of peer review and a dry run? No green/blue, no gray release, maybe these are not proper for a small change here. But this "small" change really got big affect. I thougt it was worth it. And from my shallow experience, the dry-run would always have do nothing to the env. It is dry-run anyway. And at last the three lines are found out. So I wonder how did this re-order happen? And why? With these tiny changes, there should be some mechanism to verify their correctness, not just review and dry-run.

matsur 4y ago
We use a phased rollout process for all routine changes (like this one). Once a change has passed peer review and the "dry-run", changes are rolled out to progressively larger slices of our production environment, with monitoring systems and engineers watching for adverse effects. The specific network locations that were impacted by this change were amongst the last to see the change rolled out. One deficiency in our deployment strategy (which we will correct) is that no network locations in the affected "MCP" configuration received the change early in our rollout process. If that had been the case, we would have found the problem much earlier and the incident's impact would have been much reduced.

kebab-case 4y ago

I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company. Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases? One crude approach we were discussing is during an outage we could change the NS records in the registrar to point to for eg. Google Cloud DNS which would already be in sync in terms of the DNS records it has.

juancb 4y ago
Cedexis now part of Citrix offers a multi-cdn product, that allows you to load balance between CDNs, check them out. I'm not sure who else is in that space, but in general it sounds like you want a multi-cdn strategy.
bandwidth 4y ago
If you're okay with load balancing DNS queries across multiple providers you could do 2x Cloudflare primary NS, 2x GCP for example where each provider is in sync with each other. If not, a manual swap at the registrar level would be good enough. I should also mention this approach sort of breaks with Cloudflare's proxied records which dynamically assign anycast IPs for records placed on their CDN. So if using this approach the failover NS provider would probably need to also use a different CDN, preferably one that just gives you a CNAME.
- deleted 4y ago
  [deleted]

AtNightWeCode 4y ago

Would be nice to have some automation that one could use for keeping track of health status of cloud services. Status API, webhook solution, something. Maybe even a standard for it. Or a service that monitors all major cloud services. We did get alarms. Our things partially worked though so CF was not the first thing to check.

devonkim 4y ago

Most of the criticisms seem to be around BGP and network management. What I’m seeing here that also is important is that the change was applied to a DC where the route change didn’t trigger the defect. In essence, this is also due to a very classic problem of a test dataset giving a false sense of security due to variation from other configurations. For this reason my team prefers to rollout changes to production using a test region that most customers don’t use yet will have some visible impact if there’s any error in our presumptions so far such as hard-coding regions and relying upon services not present or as capable across all regions. This practice has caught a number of rather serious errors for us that while customer impacting was nowhere near as bad as if we had rolled out simply randomly like many teams do essentially. This is even more important the more difficult it is to perform rollbacks of changes or for rollbacks to take effect such as DNS and CDN caching changes.