6 ms·

Here's our blog post detailing what happened today and the mitigations we've put in place to ensure it doesn't happen again in the future: https://blog.cloudfla

Here's our blog post detailing what happened today and the mitigations we've put in place to ensure it doesn't happen again in the future: https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/ https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...

deleted 6y ago
[deleted]
txcwpalpha 6y ago
I mean this in only a slightly judgemental way: What kind of change management/testing is going on over there at CF? This is not the first time that someone at CF has made a hasty config change that brought down a significant part of the network. Now I'll admit that proper change management won't catch every issue, but the issue described in this post seems like something that should have been caught. It's a little worrisome that a company that so much of the internet relies on is apparently playing fast and loose with major config changes. The changes you describe in the post-mortem sound like they will fix the immediate problem and possibly prevent future occurrences of this exact same problem, but what about making broader changes about how you deploy these things? Not only that, but this single config change not only brought down the CDN, but brought down both 1.1.1.1 and the secondary 1.0.0.1. Was this type of failure never tested, or...? What's the point in having both if they both go down at the same time?
- harikb 6y ago
  I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company
  - txcwpalpha 6y ago
    >I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. Even in an "emergency" situation like described in the post-mortem (which I'm actually confused about, because Prince's tweet said it was actually "routine maintenance", which makes it even less justifiable), there should be a standard playbook with pre-tested and pre-approved failover options to enact. There definitely should not be someone just manually fingering in a config file on the fly. >In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing This is fine for some startup still hacking their way through an MVP, but for a global, multi-billion dollar company that prides themselves on supporting a significant part of the global internet, "the operator better know what they are doing" is unacceptable and not good enough. There should be multiple layers of protection against this stuff, so that even if an operator is having a nutty, they don't break half the internet. >By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company I can't agree at all. We're talking about some basic change management and testing stuff here. I've been at plenty of organizations (even ones much smaller and with less resources than CF) that have gotten this right, and they certainly are not dinosaurs nor have they been "disrupted". Testing and change management is as fundamental as encryption or password hashing. If you don't have it, it's not because it's hard, it's just because you haven't bothered to implement it.
    - harikb 6y ago
      For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. If you have seen and work for organizations that know how to do this better, good for you.
      - txcwpalpha 6y ago
        >If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
- rohansingh 6y ago
  What makes you think that whoever would be responsible for approving the change would somehow know better and catch the issue?
  - teraflop 6y ago
    The fact that such a critical problem can only be detected by a human noticing it is, itself, part of the problem. Having multiple people look at the change instead of just one is a little bit better, but it's still just a band-aid. Ideally, there would be automated processes in place that could prevent and/or mitigate this kind of thing. (If we were talking about software instead of configuration, how would you view an organization that required every commit to be reviewed, but had no automated tests?) One possibility would be to parse the config file and do a rudimentary simulation, taking into account traffic volumes, and warn the user if the result would be expected to overload any routes. Another possibility would be to do something a bit smarter than just instantly, blindly propagating a change to every peer at the same time. If the bad routes had been incrementally rolled out to other routers, there might have been time for someone to notice that the metrics looked abnormal before Atlanta became overloaded. (I don't know whether that's feasible with BGP, or if it would require a smarter protocol, but it seems like it would be worth looking into.) Finally, it seems like if a config change is made to a router and it immediately crashes, there should be systems that help to correlate those two events, so that it doesn't take half an hour to identify and revert the change.
  - txcwpalpha 6y ago
    Ensuring that the appropriately-knowledgeable people for various types of changes, as well as appropriate testing processes, are part of the change management workflow is a part of good change management.
  - RKearney 6y ago
    It’s probably easy to say this as an outside party, but anyone with the most basic understanding of Junos would have known that by deactivating all of the entries in the from section of a policy or policy term, the accompanying from section would apply to all routes.
- dylan604 6y ago
  Can you name one large service/platform provider where this hasn't happened? We know the likes of Amazon, Google, Microsoft, et al have had similarly idiotic causes to go down. I'd either give them all of my business or start betting against them as their day is due.
tomklein 6y ago
I love your transparent communication and how quickly you guys respond with an explanation of what actually went wrong and how to mitigate these issues in the future. Such things happen.
- tomklein 6y ago
  Addition: CF just sent an email out to customers about the incident.
- heliodor 6y ago
  I beg to differ! They wrote a two-page essay and buried in the middle is the one-sentence root cause of "someone changed something and made a mistake". No explanation or details given. This is exactly how to pretend you have incident reports without actually having them. Luckily, the HN crowd is able to piece it together and explain in detail.
  - paco3346 6y ago
    But... they literally showed the config change and said what was wrong with it.
dang 6y ago
Ok, we've changed to that from https://www.cloudflarestatus.com/incidents/b888fyhbygb8 https://www.cloudflarestatus.com/incidents/b888fyhbygb8.
dancemethis 6y ago
Please let it happen to Discord forever. We don't need malicious software being protected by Cloudflare.
- andrewnicolalde 6y ago
  Many of us would appreciate some proof of Discord being malware
wenbin 6y ago
A similar outage happened in July last year: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
- almost_usual 6y ago
  How is a WAF regex similar to a route configuration error?
  - lima 6y ago
    Similar lack of peer review and change management process.
    - michalhosna 6y ago
      Nope, not at all. Read the damn thing. They tested it, the thing was correct, it overloded cpu, it wasn’t configuration mistake.
      - lima 6y ago
        I read the damn thing: - No performance benchmarks for rules despite using a backtracking regexp engine. - No canary deployments, allowing the mistake to propagate to their entire network at once. A mature engineering organization is very unlikely to make such basic mistakes, because it has in-depth peer review for both design and implementation. Of course, their openness about the incidents is commendable and is the only reason why my company hasn't dropped them a long time ago, but it doesn't excuse such an easily avoidable downtime.
- bawolff 6y ago
  I dont think these are all that similar - different components, different types of overloads

deleted 6y ago

[deleted]

txcwpalpha 6y ago

I mean this in only a slightly judgemental way: What kind of change management/testing is going on over there at CF? This is not the first time that someone at CF has made a hasty config change that brought down a significant part of the network. Now I'll admit that proper change management won't catch every issue, but the issue described in this post seems like something that should have been caught. It's a little worrisome that a company that so much of the internet relies on is apparently playing fast and loose with major config changes. The changes you describe in the post-mortem sound like they will fix the immediate problem and possibly prevent future occurrences of this exact same problem, but what about making broader changes about how you deploy these things? Not only that, but this single config change not only brought down the CDN, but brought down both 1.1.1.1 and the secondary 1.0.0.1. Was this type of failure never tested, or...? What's the point in having both if they both go down at the same time?

harikb 6y ago
I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company
- txcwpalpha 6y ago
  >I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. Even in an "emergency" situation like described in the post-mortem (which I'm actually confused about, because Prince's tweet said it was actually "routine maintenance", which makes it even less justifiable), there should be a standard playbook with pre-tested and pre-approved failover options to enact. There definitely should not be someone just manually fingering in a config file on the fly. >In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing This is fine for some startup still hacking their way through an MVP, but for a global, multi-billion dollar company that prides themselves on supporting a significant part of the global internet, "the operator better know what they are doing" is unacceptable and not good enough. There should be multiple layers of protection against this stuff, so that even if an operator is having a nutty, they don't break half the internet. >By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company I can't agree at all. We're talking about some basic change management and testing stuff here. I've been at plenty of organizations (even ones much smaller and with less resources than CF) that have gotten this right, and they certainly are not dinosaurs nor have they been "disrupted". Testing and change management is as fundamental as encryption or password hashing. If you don't have it, it's not because it's hard, it's just because you haven't bothered to implement it.
  - harikb 6y ago
    For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. If you have seen and work for organizations that know how to do this better, good for you.
    - txcwpalpha 6y ago
      >If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
rohansingh 6y ago
What makes you think that whoever would be responsible for approving the change would somehow know better and catch the issue?
- teraflop 6y ago
  The fact that such a critical problem can only be detected by a human noticing it is, itself, part of the problem. Having multiple people look at the change instead of just one is a little bit better, but it's still just a band-aid. Ideally, there would be automated processes in place that could prevent and/or mitigate this kind of thing. (If we were talking about software instead of configuration, how would you view an organization that required every commit to be reviewed, but had no automated tests?) One possibility would be to parse the config file and do a rudimentary simulation, taking into account traffic volumes, and warn the user if the result would be expected to overload any routes. Another possibility would be to do something a bit smarter than just instantly, blindly propagating a change to every peer at the same time. If the bad routes had been incrementally rolled out to other routers, there might have been time for someone to notice that the metrics looked abnormal before Atlanta became overloaded. (I don't know whether that's feasible with BGP, or if it would require a smarter protocol, but it seems like it would be worth looking into.) Finally, it seems like if a config change is made to a router and it immediately crashes, there should be systems that help to correlate those two events, so that it doesn't take half an hour to identify and revert the change.
- txcwpalpha 6y ago
  Ensuring that the appropriately-knowledgeable people for various types of changes, as well as appropriate testing processes, are part of the change management workflow is a part of good change management.
- RKearney 6y ago
  It’s probably easy to say this as an outside party, but anyone with the most basic understanding of Junos would have known that by deactivating all of the entries in the from section of a policy or policy term, the accompanying from section would apply to all routes.
dylan604 6y ago
Can you name one large service/platform provider where this hasn't happened? We know the likes of Amazon, Google, Microsoft, et al have had similarly idiotic causes to go down. I'd either give them all of my business or start betting against them as their day is due.

tomklein 6y ago

I love your transparent communication and how quickly you guys respond with an explanation of what actually went wrong and how to mitigate these issues in the future. Such things happen.

tomklein 6y ago
Addition: CF just sent an email out to customers about the incident.
heliodor 6y ago
I beg to differ! They wrote a two-page essay and buried in the middle is the one-sentence root cause of "someone changed something and made a mistake". No explanation or details given. This is exactly how to pretend you have incident reports without actually having them. Luckily, the HN crowd is able to piece it together and explain in detail.
- paco3346 6y ago
  But... they literally showed the config change and said what was wrong with it.

dang 6y ago

Ok, we've changed to that from https://www.cloudflarestatus.com/incidents/b888fyhbygb8 https://www.cloudflarestatus.com/incidents/b888fyhbygb8.

dancemethis 6y ago

Please let it happen to Discord forever. We don't need malicious software being protected by Cloudflare.

andrewnicolalde 6y ago
Many of us would appreciate some proof of Discord being malware

wenbin 6y ago

A similar outage happened in July last year: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...

almost_usual 6y ago
How is a WAF regex similar to a route configuration error?
- lima 6y ago
  Similar lack of peer review and change management process.
  - michalhosna 6y ago
    Nope, not at all. Read the damn thing. They tested it, the thing was correct, it overloded cpu, it wasn’t configuration mistake.
    - lima 6y ago
      I read the damn thing: - No performance benchmarks for rules despite using a backtracking regexp engine. - No canary deployments, allowing the mistake to propagate to their entire network at once. A mature engineering organization is very unlikely to make such basic mistakes, because it has in-depth peer review for both design and implementation. Of course, their openness about the incidents is commendable and is the only reason why my company hasn't dropped them a long time ago, but it doesn't excuse such an easily avoidable downtime.
bawolff 6y ago
I dont think these are all that similar - different components, different types of overloads