13 ms·

Today's Outage Post Mortem

216 points by buttscicles 14y ago

rdl 14y ago
Wow, that's pretty fast turnaround for a post-mortem (although it looks to have been a simple problem, so easier to figure out what to write)
- jgrahamc 14y ago
  Our customers deserve to know what happened as quickly as possible.
yRetsyM 14y ago
I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?
- eastdakota 14y ago
  Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.
  - yRetsyM 14y ago
    Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling. My 2cents: Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff) It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live. The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.
  - nathannecro 14y ago
    Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?
    - toast0 14y ago
      Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact. If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.
- jonaslejon 14y ago
  Good idea. There might be a number of rules that can limit their communication with the routers (and even OOB)
lazyjones 14y ago
So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.
- rdl 14y ago
  Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion). It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless you're building the space shuttle (and can accept vastly higher costs and lower performance), it's probably better to pick one hardware platform, at least now.
  - eastdakota 14y ago
    While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.
    - rdl 14y ago
      The thing which annoyed me the most was losing all DNS. You really need to have the DNS servers in separate infrastructure (ASN, netblock, while anycasted) so there is never a case where both of your DNS are out for a customer domain. The "CNAME" product looks pretty kludgey.
      - ams6110 14y ago
        By the same token you (the customer) should not have all your DNS eggs in one basket.
        rincebrain 14y ago
        CloudFlare's CDN bits require you to give them DNS delegation of your stuff, last I looked.
senthilnayagam 14y ago
So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites. Hope lessons are learnt and your next generation is less prone to these attacks
- packetbeats 14y ago
  If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.
  - opendomain 14y ago
    This is exactly my analysis. Why would the packet the attacker using be so large? The only logical reason is that they knew the Juniper bug. I am wondering if Juniper also knew this bug but did not disclose it.
  - rurounijones 14y ago
    The idea that the attacker knew about the bug is, I think, a remote but intriguing idea. Wonder if Cloudflare need to do some tests along the lines of: A) List up all the types of rules we usually use to mitigate these situations. B) Run those rules on a test router with wildly unusual input values, as was the case in this situation. C) Send test traffic using that wildly unexpected input to see what happens. Basically a bit of manual fuzzing Time-consuming and maybe not worthwhile, but it could save against another full system death.
  - senthilnayagam 14y ago
    even according to their admission, if they had not made any change the apps would all have run, just possibly some lag, but making a change for malicious user without knowing the consequence lead to this scenario.
dododo 14y ago
if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).
- dageshi 14y ago
  Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.
- rdl 14y ago
  That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense. You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.
  - windexh8er 14y ago
    Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with. People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.
jcr 14y ago
To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes might have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast roll-out is one of those really tough balancing acts in your situation. You did a great job with it; by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again.
- MichaelApproved 14y ago
  "by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again." Not sure what your timeline shows but the "cloudflare is down" post hit the #1 spot on the front page just a few minutes after they went down. About 40 minutes after that, the services started to come back online for me. That's a significant outage. That's not reflecting on the job they did bringing things back online but your statement made it seem like a minor outage.
  - druiid 14y ago
    Indeed. This was no few minute outage. There was very close to a full hour of zero traffic according to our MRTG.
    - gwern 14y ago
      Pingdom reported an hour downtime on my site too.
- GigabyteCoin 14y ago
  Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?
  - eastdakota 14y ago
ryguytilidie 14y ago
This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.
- jgrahamc 14y ago
  Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.
  - larrys 14y ago
    "The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?
    - TazeTSchnitzel 14y ago
      Uptime is a fundamental part of their business, of course it's a team.
      - larrys 14y ago
        What do you mean "of course it's a team" because "uptime is a fundamental part of their business"? That's an assumption on your part. Many negative events happen because of assumptions people make and things they take for granted. I ask a valid question. Do you know for a fact how many people monitor their network or what systems they have in place? Or whether they even simulate events like this to see if other members of the team are even reachable? Have you ever seen the constant testing that goes on at large organizations (such as the military) to make sure battle systems are ready to deploy when necessary? Note also that cloudflare charges $3000 per month for enterprise plans (in addition to free) and that their own website was taken offline.
      - 14y ago
brokentone 14y ago
Impressive response. 30 minute outage for something most of the hosts I've worked with in the past would have been mystified about for hours. Then a quick RFO and promise of proactive SLA adjustments? Next time I need a CDN or attack mitigation I'll be talking to Cloudflare
druiid 14y ago
As always I'm glad to see Cloudflare post such detailed outage reports. They are one of the few providers I know of that is willing to go into such depth and that is one of the things I appreciate about them. That said, the outage that occurred was one that was indeed fully preventable. We don't exactly have as many locations as they do, but for internal resources at least, not pushing configuration changes to all devices (network included) is pretty standard practice. Basically I imagine for them a good routine to follow might be to script changes so that they are 'rolled out', something along the lines of push manual changes to a scripted 'random' router set (one in country A,B,C), wait 15 minutes and then push to the remaining router sets. That wouldn't work for all situations, such as if the entire network is seeing a DDoS or what have you, but I imagine they could adapt a routine that would prevent this particular scenario. With all of that said, as a Cloudflare customer and also having a call with them tomorrow scheduled already over the WAF stuff, I find it a bit... frustrating that this is occurring now and such a kind of mistake. Edit: As an aside, I wonder if the Puppet module for Junos will be extended to support route statements. That would make this kind of deployment much easier.
- alexchamberlain 14y ago
  Although I agree, it would be rather hard to fight an attack if you didn't roll out fairly quickly...
  - eastdakota 14y ago
    You've identified the rock and the hard place.
  - saurik 14y ago
    The article, however, actually states that the expectation should be for the rule to do nothing, as the packets in question were much larger than the maximum packet size. You thereby have to examine this as some kind of rushed "let's try something, anything" reaction to a situation where an engineer didn't actually understand what was happening enough to make such a call: it is not surprising that the result was that they ended up landing squarely in "something even more confusing has now happened and everything is offline" territory.
powertower 14y ago
> CloudFlare currently runs 23 data centers worldwide. Shouldn't that always say - CloudFlare currently runs in 23 data centers worldwide? Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone. Just curious.
- DASD 14y ago
  You are correct. However, it is not as bad as the egregious use of the word "unlimited" by hosting providers.
- ChuckMcM 14y ago
  A bit nit picky yes? Even Google mentions places as "data centers" which really are co-location facilities hosting a bit of gear. It has come to be a way to identify unique numbers of building rather than ownership in most papers or articles that I've read. To be clear I share your desire for precise speech, perhaps the could have said 'Cloudflare currently uses 23 data centers worldwide' but given that the person writing this is doing it on a Sunday, probably after a long night and having missed all the things they normally would have been doing on Sunday, I'm willing to cut them some slack.
- tlrobinson 14y ago
  The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.
  - larrys 14y ago
    "As a customer you should care that their service is geographically distributed" Don't agree. If you own the data center you have more control over it. We had a case where the UPS systems in a data center had bad batteries and equipment went down because the batteries failed to kick in. Since we don't own the data center we have no realistic way to make inspections and make sure the right thing happens or that the batteries (or the generators) are cycled and maintained. We just have to trust. [1] Now this may or may not matter with the way they have their redundancy setup. But owning a data center does give you more control over more things. "not whether they own the buildings where the servers are kept." Owning the data center and owning the buildings are two different things. Owning the building is owning real estate. Owning the data center is owning the security setup, backup systems etc. Two different things. [1] So as not to contradict myself with other things I have said I should not say "trust" because you can always put some things in place to verify the right thing is happening (inspections, logs etc.) if you want. But if you own the place it's easier. If I own my home I can decide when to replace the HVAC so it doesn't fail in the middle of the summer. If I rent that's up to the landlord.
Flow 14y ago
I think you should have investigated why you got ~90kb packages despite having a max pkg size of ~4kb instead of putting in that rule. :)
- dubcanada 14y ago
  I was thinking the exact same thing.
rainsford 14y ago
That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events. One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn't have out-of-band remote power cycle capabilities. There may be some factor I'm not considering that would make that an unattractive option, but it does seem like it could cut down an already quick response time even further for any similar events in the future.
- rdl 14y ago
  I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though. I think it's mostly that the routers usually have both good OOB management and good watchdog (reboot on freeze) behavior, and that the PSUs in the bigger routers tend to exceed the per-port power limits of most of the external power cyclers. It may be a good idea, though.
  - justincormack 14y ago
    You would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.
    - mprovost 14y ago
      We just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.
onemorepassword 14y ago
> Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources. I know very little of networking, but this seems to be a recurring pattern that aggravates many major outages. What surprises me is that this so often seems to be a scenario not accounted for.
- jonknee 14y ago
  You can only account for it by having more hardware and then it's possible more of your hardware will fail which puts you right back to where you started.
  - Dylan16807 14y ago
    I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully served. I bet that most of the time the domino effect happens to internet services in general it's with nodes that are accepting most requests. They allow themselves to be overloaded. An active HTTP session uses orders of magnitude more resources than simply denying the initial packet and forgetting about it forever.
    - seanp2k2 14y ago
      You're vastly oversimplifying the problem here by only accounting for one class of problems. >". I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them." depends on the attack. >"You can send a tiny error page. You can let X% of requests get through and be fully served." Not usually that easy.
      - Dylan16807 14y ago
jaequery 14y ago
this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.
- danielpal 14y ago
  What is the alternative? Are you saying you can run a service without a DNS provider? You can always have multiple DNS service providers and CloudFlare is probably one of the best ones.
  - frankacter 14y ago
    As of January 2013, CloudFlare is number two, second only to Dyn: http://blog.cloudflare.com/cloudflare-fastest-free-dns-among-fastest-dns http://blog.cloudflare.com/cloudflare-fastest-free-dns-among...
- driverdan 14y ago
  You could say the same thing about almost any of your service providers. Your DNS provider goes out, everything goes out. The routers at the data center with your servers go out, all your servers go out. Your CDN goes out, all of the static assets on your site go out. There will always be potential SPOF.
  - saurik 14y ago
    While your example with the routers at your backend is truly problematic, DNS is designed with built-in redundancy and CDNs (which CloudFlare should not really count as) having a world-wide outage (as opposed to "people accessing from New York are currently having issues, as we lost one PoP") is nigh-unto unheard of... can you imagine Akamai (or CDNetworks or EdgeCast or even Amazon) saying "doh, all of our infrastructure everywhere just disappeared"? The core problem with CloudFlare is that they seem to have a highly-centralized take on what is normally a massively-decentralized solution-space, with large numbers of value-adds they encourage customers to use without making it clear that they treat in a haphazard manner, doing very little testing before deploying pushing-the-envelope features while simultaneously having very little in-house debugging expertise to handle serious issues. (As a concrete example of that last complaint, Cydia was crippled for an entire day due to ModMyi turning on CloudFlare's "preloader" transformation, which apparently caused many WebKit-based browsers--including both MobileSafari and Cydia--to entirely lock up; CloudFlare seemed to go the entire day without noticing, which I continue to be utterly shocked by, and it was only after I told them how to fix it that they were able to acknowledge the issue.) http://www.saurik.com/id/14 http://www.saurik.com/id/14 <- When "Dumb Pipes" Get Too Smart, an extensive analysis of this bug
TranceMan 14y ago
Just wondering if the source of the large packets were from a [large] range of hosts or maybe a single host? Ouch if a single host activity took down ~750k websites - whether deliberate and direct or not.
- dubcanada 14y ago
  It's probably even more then that. CloudFlare hosts cdnjs which a ton of people use. It could have "taken down" millions of sites. And by taken down I mean rendered unusable.
ralph 14y ago
Presumably Jupiter's Junos is closed-source, making investigation more difficult? Do they provide it to some of their bigger clients under an agreement?
noselasd 14y ago
> attack packets were between 99,971 and 99,985 bytes long. This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.
- naww 14y ago
  http://en.wikipedia.org/wiki/Jumbogram http://en.wikipedia.org/wiki/Jumbogram > An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB
  - mprovost 14y ago
    Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.
    - bdonlan 14y ago
      They could've been fragmented IPv6 packets. Or it could've been a bug in their profiler.
      - pyvpx 14y ago
        which is precisely why it seems like lunacy to roll out such an asinine firewall rule to every router. if there was ever a time to "spot check" a change, this was it. they didn't. and they paid the price. good on 'em for the quick and honest post-mortem. regardless, it was a dumb move.
    - cbsmith 14y ago
      You are joking right? The packet size at the higher layer is what they were matching against. The size of the layer 2 packets is irrelevant.
deleted 14y ago
[deleted]
contingencies 14y ago
Developing good software comes down to consistently carrying out fundamental practices (regardless of the technology) - Paul M. Duvall In this case: Development. Versioned change. Test or staging environment. Tests pass. Production.
- rurounijones 14y ago
  Meanwhile your customers are getting DDOS'ed while you are faffing about. Yes, I fully agree that for things like software and standard network maintenance the above is good. But as someone else mentioned in this thread. DDoSes that require quick resolution put you between a rock and a hard place in terms of doing things "right"
  - contingencies 14y ago
    That's true. However, look at what happens to all of your customers when you fail to test. If you haven't limited, or at least tested the extreme ranges of allowable input to a system automatically pushing out live configuration to all of your routers, there's nobody else to blame but yourself. Sorry. Are most people this diligent? No. Should we be? Yes.
    - rurounijones 14y ago
      I actually wondered about testing with extreme ranges in another comment, but this testing is done "offline" (Not on live routers and not in response to current circumstances). However, at least how I read it, your comment was about testing rules in consistently in dev -> staging -> prod when you create one which I think is not viable in this situation since you are on a very tight deadline with immediate impact on your customers.
random42 14y ago
OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?
- pyvpx 14y ago
  akamai is about 100x the size and probably 200x the price.
  - saurik 14y ago
    If you are going to go for the $3k/mo CloudFlare plan, you are already nearing the ballpark of Akamai, and would do good to look at the many CDNs that sit in the middle of that scale (such as CDNetworks or EdgeCast).
- acdha 14y ago
  Akamai is much larger – 100+K edge nodes, many collocated in ISP's local facilities – and has much, much better global coverage, particularly if you care about Asia or South America. Downside: priced accordingly.
- meh02 14y ago
  Cloudflare is not a replacement for Akamai. They've just admitted to having a single network ops guy monitoring their network at night. Cloudflare is what happens when amateurs attempt to run a CDN. What reason do you have to pitch Cloudflare?
BoyWizard 14y ago
Two things: 1. That video of the BGP routes disappearing is awesome, and 2. A 40 minute outage sounds bad, but consider the following timeline (based on the writeup): > T+0: route change made, propagates > T+10: Response team online, attempting local fixes > T+30: Routers across 23 data centres in 14 countries hard reset and networks coming back up.
graycat 14y ago
Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time management of server farms and networks. Of course, in that work, goals included 'lights out data centers', that is, don't need people walking around doing manual work but not the case of 'lights out' as in the CloudFlare outage, and very high reliability. Looking into reliability, that is, putting into a few, broad categories the causes of outages, a category causing a large fraction of the outages was humans doing system management, or as in the words of the HAL 9000, "human error". Yup. And the whole thing went down? Yup: One example we worked with was system management of a 'cluster'. Well, one of the computers in the cluster "went a little funny, a little funny in the head" and was throwing all its incoming work into its 'bit bucket'. So, the CPU busy metric on that computer was not very high, and the load leveling started sending nearly all the work to that one computer and, thus, into its bit bucket and, thus, effectively killed the work of the whole cluster. As one response I decided that real time monitoring of a cluster, or any system that is supposed to be 'well balanced' via some version of 'load leveling', should include looking for 'out of balance' situations. So, let's see: Such monitoring can have false positives (false alarms) and false negetives (missed detections). So, such monitoring is necessarily essentially a case of some statistical hypothesis testing, typically with the 'null hypothesis' that the system is healthy, applied continually in near real-time. So, for monitoring 'balancing', we will likely have to work with multi-dimensional data. Next, our chances of knowing the probability distribution of that data, even in the case of a healthy system, is from slim down to none. So we need a statistical hypothesis test that is both multi-dimensional and distribution-free. So, CloudFlare's problems are not really new! I went ahead and did some work, math, prototype software, etc. and maybe someday it will be useful, but it wouldn't have helped CloudFlare here if only because they needed no help noticing that all their systems around the world were crashing. In our work on AI, at times we visited some high end sites, and in some cases we found some extreme, high up off the tops of the charts, concern and discipline for who, what, or why any humans could take any system management actions. E.g., they had learned the lesson that can't let someone just type in a new rule in a production system. Why? Because it was explained that one outage in a year, and the CIO would lose his bonus. Two outages and he would lose his job. Net, we're talking very high concern. No doubt CloudFlare will install lots of discipline around humans taking system management actions on their production systems. Net, I can't blame CloudFlare. If my business gets big enough to need their services, they will be high on the list of companies I will call first!
AsifRahman 14y ago
DNS need to be more redundant. Still appreciate your explanation.
DigitalSea 14y ago
Rather unfortunate for the credibility of Cloudfare as a network provider, but you've got to admire them for their honesty and it'll work out better for them in the end. It's amazing how a few lines of code managed to bring down Cloudfare, they could have told us anything and nobody would have been able to question it; instead they gave us the truth and I really respect that. They didn't blame the intern, they didn't blame their hardware or make an excuse about a power outage. In terms of honesty Cloudfare seems to be leading the way regardless of their public credibility or image being tainted. Very impressive response time and resolution of the issue as well, good job Cloudfare!
tedchs 14y ago
What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static config, and there is a set of firewalls (e.g. Juniper SRX 5800) behind them that are doing the actual filtering and changing configs in response to threats.
- seanp2k2 14y ago
  So now you have two pieces of gear to test changes on and another interaction where stuff could break / go weird. I don't see how that would solve anything here.
  - tedchs 14y ago
    The thing it would solve is risking all their BGP peerings going down as a result of day-to-day service operations (i.e. every time they add a filter).
newman314 14y ago
Posterous seems to be down.
DoubleMalt 14y ago
Funny that now the post mortem is down ...
- tomvo 14y ago
  same here, ironic indeed.
  - Igalze 14y ago
    Ironic is one way of putting it... :)
- jgrahamc 14y ago
  Yes, ironic. Unfortunately, the CloudFlare blog is hosted on posterous and they seem to be down.
  - rdl 14y ago
    You may want to move off posterous before it goes down for good in a month, too :)
automaton 14y ago
I'm getting a -- Oh noes! Something went wrong. --- screen. Seems like the post regarding the outage is facing an outage :)
rschmitty 14y ago
Now we need a Post Mortem on the Post Mortem, as it is now down "Oh noes! Something went wrong."
Ecio78 14y ago
I got a "Oops there was a problem" page from Postereous trying to open the blog page/site...

216 points by buttscicles 14y ago

rdl 14y ago

Wow, that's pretty fast turnaround for a post-mortem (although it looks to have been a simple problem, so easier to figure out what to write)

jgrahamc 14y ago
Our customers deserve to know what happened as quickly as possible.

yRetsyM 14y ago

I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response. It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?

eastdakota 14y ago
Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.
- yRetsyM 14y ago
  Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling. My 2cents: Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff) It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live. The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.
- nathannecro 14y ago
  Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?
  - toast0 14y ago
    Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact. If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.
jonaslejon 14y ago
Good idea. There might be a number of rules that can limit their communication with the routers (and even OOB)

lazyjones 14y ago

So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.

rdl 14y ago
Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion). It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless you're building the space shuttle (and can accept vastly higher costs and lower performance), it's probably better to pick one hardware platform, at least now.
- eastdakota 14y ago
  While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.
  - rdl 14y ago
    The thing which annoyed me the most was losing all DNS. You really need to have the DNS servers in separate infrastructure (ASN, netblock, while anycasted) so there is never a case where both of your DNS are out for a customer domain. The "CNAME" product looks pretty kludgey.
    - ams6110 14y ago
      By the same token you (the customer) should not have all your DNS eggs in one basket.
      - rincebrain 14y ago
        CloudFlare's CDN bits require you to give them DNS delegation of your stuff, last I looked.

senthilnayagam 14y ago

So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites. Hope lessons are learnt and your next generation is less prone to these attacks

packetbeats 14y ago
If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.
- opendomain 14y ago
  This is exactly my analysis. Why would the packet the attacker using be so large? The only logical reason is that they knew the Juniper bug. I am wondering if Juniper also knew this bug but did not disclose it.
- rurounijones 14y ago
  The idea that the attacker knew about the bug is, I think, a remote but intriguing idea. Wonder if Cloudflare need to do some tests along the lines of: A) List up all the types of rules we usually use to mitigate these situations. B) Run those rules on a test router with wildly unusual input values, as was the case in this situation. C) Send test traffic using that wildly unexpected input to see what happens. Basically a bit of manual fuzzing Time-consuming and maybe not worthwhile, but it could save against another full system death.
- senthilnayagam 14y ago
  even according to their admission, if they had not made any change the apps would all have run, just possibly some lag, but making a change for malicious user without knowing the consequence lead to this scenario.

dododo 14y ago

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

dageshi 14y ago
Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.
rdl 14y ago
That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense. You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.
- windexh8er 14y ago
  Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with. People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.

jcr 14y ago

To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes might have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast roll-out is one of those really tough balancing acts in your situation. You did a great job with it; by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again.

MichaelApproved 14y ago
"by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again." Not sure what your timeline shows but the "cloudflare is down" post hit the #1 spot on the front page just a few minutes after they went down. About 40 minutes after that, the services started to come back online for me. That's a significant outage. That's not reflecting on the job they did bringing things back online but your statement made it seem like a minor outage.
- druiid 14y ago
  Indeed. This was no few minute outage. There was very close to a full hour of zero traffic according to our MRTG.
  - gwern 14y ago
    Pingdom reported an hour downtime on my site too.
GigabyteCoin 14y ago
Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?
- eastdakota 14y ago

ryguytilidie 14y ago

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

jgrahamc 14y ago
Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.
- larrys 14y ago
  "The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?
  - TazeTSchnitzel 14y ago
    Uptime is a fundamental part of their business, of course it's a team.
    - larrys 14y ago
      What do you mean "of course it's a team" because "uptime is a fundamental part of their business"? That's an assumption on your part. Many negative events happen because of assumptions people make and things they take for granted. I ask a valid question. Do you know for a fact how many people monitor their network or what systems they have in place? Or whether they even simulate events like this to see if other members of the team are even reachable? Have you ever seen the constant testing that goes on at large organizations (such as the military) to make sure battle systems are ready to deploy when necessary? Note also that cloudflare charges $3000 per month for enterprise plans (in addition to free) and that their own website was taken offline.
    - 14y ago

brokentone 14y ago

Impressive response. 30 minute outage for something most of the hosts I've worked with in the past would have been mystified about for hours. Then a quick RFO and promise of proactive SLA adjustments? Next time I need a CDN or attack mitigation I'll be talking to Cloudflare

druiid 14y ago

As always I'm glad to see Cloudflare post such detailed outage reports. They are one of the few providers I know of that is willing to go into such depth and that is one of the things I appreciate about them. That said, the outage that occurred was one that was indeed fully preventable. We don't exactly have as many locations as they do, but for internal resources at least, not pushing configuration changes to all devices (network included) is pretty standard practice. Basically I imagine for them a good routine to follow might be to script changes so that they are 'rolled out', something along the lines of push manual changes to a scripted 'random' router set (one in country A,B,C), wait 15 minutes and then push to the remaining router sets. That wouldn't work for all situations, such as if the entire network is seeing a DDoS or what have you, but I imagine they could adapt a routine that would prevent this particular scenario. With all of that said, as a Cloudflare customer and also having a call with them tomorrow scheduled already over the WAF stuff, I find it a bit... frustrating that this is occurring now and such a kind of mistake. Edit: As an aside, I wonder if the Puppet module for Junos will be extended to support route statements. That would make this kind of deployment much easier.

alexchamberlain 14y ago
Although I agree, it would be rather hard to fight an attack if you didn't roll out fairly quickly...
- eastdakota 14y ago
  You've identified the rock and the hard place.
- saurik 14y ago
  The article, however, actually states that the expectation should be for the rule to do nothing, as the packets in question were much larger than the maximum packet size. You thereby have to examine this as some kind of rushed "let's try something, anything" reaction to a situation where an engineer didn't actually understand what was happening enough to make such a call: it is not surprising that the result was that they ended up landing squarely in "something even more confusing has now happened and everything is offline" territory.

powertower 14y ago

> CloudFlare currently runs 23 data centers worldwide. Shouldn't that always say - CloudFlare currently runs in 23 data centers worldwide? Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone. Just curious.

DASD 14y ago
You are correct. However, it is not as bad as the egregious use of the word "unlimited" by hosting providers.
ChuckMcM 14y ago
A bit nit picky yes? Even Google mentions places as "data centers" which really are co-location facilities hosting a bit of gear. It has come to be a way to identify unique numbers of building rather than ownership in most papers or articles that I've read. To be clear I share your desire for precise speech, perhaps the could have said 'Cloudflare currently uses 23 data centers worldwide' but given that the person writing this is doing it on a Sunday, probably after a long night and having missed all the things they normally would have been doing on Sunday, I'm willing to cut them some slack.
tlrobinson 14y ago
The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.
- larrys 14y ago
  "As a customer you should care that their service is geographically distributed" Don't agree. If you own the data center you have more control over it. We had a case where the UPS systems in a data center had bad batteries and equipment went down because the batteries failed to kick in. Since we don't own the data center we have no realistic way to make inspections and make sure the right thing happens or that the batteries (or the generators) are cycled and maintained. We just have to trust. [1] Now this may or may not matter with the way they have their redundancy setup. But owning a data center does give you more control over more things. "not whether they own the buildings where the servers are kept." Owning the data center and owning the buildings are two different things. Owning the building is owning real estate. Owning the data center is owning the security setup, backup systems etc. Two different things. [1] So as not to contradict myself with other things I have said I should not say "trust" because you can always put some things in place to verify the right thing is happening (inspections, logs etc.) if you want. But if you own the place it's easier. If I own my home I can decide when to replace the HVAC so it doesn't fail in the middle of the summer. If I rent that's up to the landlord.

Flow 14y ago

I think you should have investigated why you got ~90kb packages despite having a max pkg size of ~4kb instead of putting in that rule. :)

dubcanada 14y ago
I was thinking the exact same thing.

rainsford 14y ago

That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events. One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn't have out-of-band remote power cycle capabilities. There may be some factor I'm not considering that would make that an unattractive option, but it does seem like it could cut down an already quick response time even further for any similar events in the future.

rdl 14y ago
I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though. I think it's mostly that the routers usually have both good OOB management and good watchdog (reboot on freeze) behavior, and that the PSUs in the bigger routers tend to exceed the per-port power limits of most of the external power cyclers. It may be a good idea, though.
- justincormack 14y ago
  You would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.
  - mprovost 14y ago
    We just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.

onemorepassword 14y ago

> Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources. I know very little of networking, but this seems to be a recurring pattern that aggravates many major outages. What surprises me is that this so often seems to be a scenario not accounted for.

jonknee 14y ago
You can only account for it by having more hardware and then it's possible more of your hardware will fail which puts you right back to where you started.
- Dylan16807 14y ago
  I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully served. I bet that most of the time the domino effect happens to internet services in general it's with nodes that are accepting most requests. They allow themselves to be overloaded. An active HTTP session uses orders of magnitude more resources than simply denying the initial packet and forgetting about it forever.
  - seanp2k2 14y ago
    You're vastly oversimplifying the problem here by only accounting for one class of problems. >". I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them." depends on the attack. >"You can send a tiny error page. You can let X% of requests get through and be fully served." Not usually that easy.
    - Dylan16807 14y ago

jaequery 14y ago

this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.

danielpal 14y ago
What is the alternative? Are you saying you can run a service without a DNS provider? You can always have multiple DNS service providers and CloudFlare is probably one of the best ones.
- frankacter 14y ago
  As of January 2013, CloudFlare is number two, second only to Dyn: http://blog.cloudflare.com/cloudflare-fastest-free-dns-among-fastest-dns http://blog.cloudflare.com/cloudflare-fastest-free-dns-among...
driverdan 14y ago
You could say the same thing about almost any of your service providers. Your DNS provider goes out, everything goes out. The routers at the data center with your servers go out, all your servers go out. Your CDN goes out, all of the static assets on your site go out. There will always be potential SPOF.
- saurik 14y ago
  While your example with the routers at your backend is truly problematic, DNS is designed with built-in redundancy and CDNs (which CloudFlare should not really count as) having a world-wide outage (as opposed to "people accessing from New York are currently having issues, as we lost one PoP") is nigh-unto unheard of... can you imagine Akamai (or CDNetworks or EdgeCast or even Amazon) saying "doh, all of our infrastructure everywhere just disappeared"? The core problem with CloudFlare is that they seem to have a highly-centralized take on what is normally a massively-decentralized solution-space, with large numbers of value-adds they encourage customers to use without making it clear that they treat in a haphazard manner, doing very little testing before deploying pushing-the-envelope features while simultaneously having very little in-house debugging expertise to handle serious issues. (As a concrete example of that last complaint, Cydia was crippled for an entire day due to ModMyi turning on CloudFlare's "preloader" transformation, which apparently caused many WebKit-based browsers--including both MobileSafari and Cydia--to entirely lock up; CloudFlare seemed to go the entire day without noticing, which I continue to be utterly shocked by, and it was only after I told them how to fix it that they were able to acknowledge the issue.) http://www.saurik.com/id/14 http://www.saurik.com/id/14 <- When "Dumb Pipes" Get Too Smart, an extensive analysis of this bug

TranceMan 14y ago

Just wondering if the source of the large packets were from a [large] range of hosts or maybe a single host? Ouch if a single host activity took down ~750k websites - whether deliberate and direct or not.

dubcanada 14y ago
It's probably even more then that. CloudFlare hosts cdnjs which a ton of people use. It could have "taken down" millions of sites. And by taken down I mean rendered unusable.

ralph 14y ago

Presumably Jupiter's Junos is closed-source, making investigation more difficult? Do they provide it to some of their bigger clients under an agreement?

noselasd 14y ago

> attack packets were between 99,971 and 99,985 bytes long. This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.

naww 14y ago
http://en.wikipedia.org/wiki/Jumbogram http://en.wikipedia.org/wiki/Jumbogram > An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB
- mprovost 14y ago
  Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.
  - bdonlan 14y ago
    They could've been fragmented IPv6 packets. Or it could've been a bug in their profiler.
    - pyvpx 14y ago
      which is precisely why it seems like lunacy to roll out such an asinine firewall rule to every router. if there was ever a time to "spot check" a change, this was it. they didn't. and they paid the price. good on 'em for the quick and honest post-mortem. regardless, it was a dumb move.
  - cbsmith 14y ago
    You are joking right? The packet size at the higher layer is what they were matching against. The size of the layer 2 packets is irrelevant.

deleted 14y ago

[deleted]

contingencies 14y ago

Developing good software comes down to consistently carrying out fundamental practices (regardless of the technology) - Paul M. Duvall In this case: Development. Versioned change. Test or staging environment. Tests pass. Production.

rurounijones 14y ago
Meanwhile your customers are getting DDOS'ed while you are faffing about. Yes, I fully agree that for things like software and standard network maintenance the above is good. But as someone else mentioned in this thread. DDoSes that require quick resolution put you between a rock and a hard place in terms of doing things "right"
- contingencies 14y ago
  That's true. However, look at what happens to all of your customers when you fail to test. If you haven't limited, or at least tested the extreme ranges of allowable input to a system automatically pushing out live configuration to all of your routers, there's nobody else to blame but yourself. Sorry. Are most people this diligent? No. Should we be? Yes.
  - rurounijones 14y ago
    I actually wondered about testing with extreme ranges in another comment, but this testing is done "offline" (Not on live routers and not in response to current circumstances). However, at least how I read it, your comment was about testing rules in consistently in dev -> staging -> prod when you create one which I think is not viable in this situation since you are on a very tight deadline with immediate impact on your customers.

random42 14y ago

OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?

pyvpx 14y ago
akamai is about 100x the size and probably 200x the price.
- saurik 14y ago
  If you are going to go for the $3k/mo CloudFlare plan, you are already nearing the ballpark of Akamai, and would do good to look at the many CDNs that sit in the middle of that scale (such as CDNetworks or EdgeCast).
acdha 14y ago
Akamai is much larger – 100+K edge nodes, many collocated in ISP's local facilities – and has much, much better global coverage, particularly if you care about Asia or South America. Downside: priced accordingly.
meh02 14y ago
Cloudflare is not a replacement for Akamai. They've just admitted to having a single network ops guy monitoring their network at night. Cloudflare is what happens when amateurs attempt to run a CDN. What reason do you have to pitch Cloudflare?

BoyWizard 14y ago

Two things: 1. That video of the BGP routes disappearing is awesome, and 2. A 40 minute outage sounds bad, but consider the following timeline (based on the writeup): > T+0: route change made, propagates > T+10: Response team online, attempting local fixes > T+30: Routers across 23 data centres in 14 countries hard reset and networks coming back up.

graycat 14y ago

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash. They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time management of server farms and networks. Of course, in that work, goals included 'lights out data centers', that is, don't need people walking around doing manual work but not the case of 'lights out' as in the CloudFlare outage, and very high reliability. Looking into reliability, that is, putting into a few, broad categories the causes of outages, a category causing a large fraction of the outages was humans doing system management, or as in the words of the HAL 9000, "human error". Yup. And the whole thing went down? Yup: One example we worked with was system management of a 'cluster'. Well, one of the computers in the cluster "went a little funny, a little funny in the head" and was throwing all its incoming work into its 'bit bucket'. So, the CPU busy metric on that computer was not very high, and the load leveling started sending nearly all the work to that one computer and, thus, into its bit bucket and, thus, effectively killed the work of the whole cluster. As one response I decided that real time monitoring of a cluster, or any system that is supposed to be 'well balanced' via some version of 'load leveling', should include looking for 'out of balance' situations. So, let's see: Such monitoring can have false positives (false alarms) and false negetives (missed detections). So, such monitoring is necessarily essentially a case of some statistical hypothesis testing, typically with the 'null hypothesis' that the system is healthy, applied continually in near real-time. So, for monitoring 'balancing', we will likely have to work with multi-dimensional data. Next, our chances of knowing the probability distribution of that data, even in the case of a healthy system, is from slim down to none. So we need a statistical hypothesis test that is both multi-dimensional and distribution-free. So, CloudFlare's problems are not really new! I went ahead and did some work, math, prototype software, etc. and maybe someday it will be useful, but it wouldn't have helped CloudFlare here if only because they needed no help noticing that all their systems around the world were crashing. In our work on AI, at times we visited some high end sites, and in some cases we found some extreme, high up off the tops of the charts, concern and discipline for who, what, or why any humans could take any system management actions. E.g., they had learned the lesson that can't let someone just type in a new rule in a production system. Why? Because it was explained that one outage in a year, and the CIO would lose his bonus. Two outages and he would lose his job. Net, we're talking very high concern. No doubt CloudFlare will install lots of discipline around humans taking system management actions on their production systems. Net, I can't blame CloudFlare. If my business gets big enough to need their services, they will be high on the list of companies I will call first!

AsifRahman 14y ago

DNS need to be more redundant. Still appreciate your explanation.

DigitalSea 14y ago

Rather unfortunate for the credibility of Cloudfare as a network provider, but you've got to admire them for their honesty and it'll work out better for them in the end. It's amazing how a few lines of code managed to bring down Cloudfare, they could have told us anything and nobody would have been able to question it; instead they gave us the truth and I really respect that. They didn't blame the intern, they didn't blame their hardware or make an excuse about a power outage. In terms of honesty Cloudfare seems to be leading the way regardless of their public credibility or image being tainted. Very impressive response time and resolution of the issue as well, good job Cloudfare!

tedchs 14y ago

What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static config, and there is a set of firewalls (e.g. Juniper SRX 5800) behind them that are doing the actual filtering and changing configs in response to threats.

seanp2k2 14y ago
So now you have two pieces of gear to test changes on and another interaction where stuff could break / go weird. I don't see how that would solve anything here.
- tedchs 14y ago
  The thing it would solve is risking all their BGP peerings going down as a result of day-to-day service operations (i.e. every time they add a filter).

newman314 14y ago

Posterous seems to be down.

DoubleMalt 14y ago

Funny that now the post mortem is down ...

tomvo 14y ago
same here, ironic indeed.
- Igalze 14y ago
  Ironic is one way of putting it... :)
jgrahamc 14y ago
Yes, ironic. Unfortunately, the CloudFlare blog is hosted on posterous and they seem to be down.
- rdl 14y ago
  You may want to move off posterous before it goes down for good in a month, too :)

automaton 14y ago

I'm getting a -- Oh noes! Something went wrong. --- screen. Seems like the post regarding the outage is facing an outage :)

rschmitty 14y ago

Now we need a Post Mortem on the Post Mortem, as it is now down "Oh noes! Something went wrong."

Ecio78 14y ago

I got a "Oops there was a problem" page from Postereous trying to open the blog page/site...