4 ms·

Update from CEO https://twitter.com/eastdakota/status/1250479501726760960 https://twitter.com/eastdakota/status/1250479501726760960

zymhan 7y ago
How on earth can rented "remote hands" at a datacenter take down the ENTIRE Cloudflare management layer?
- MichaelApproved 7y ago
  and for over 3 hours? What could they do accidentally that would cause an outage for this long? The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster. Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online. I also wonder how much Covid impacted the process, if at all. I'm looking forward to the details after they get things back online. Good luck CF engineers!
  - foobarbazetc 7y ago
    Probably nuked the main DB.
  - eb0la 7y ago
    Three hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking. From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before. When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.
- mcpherrinm 7y ago
  I don't know how big Cloudflare's management layer is, but I'm assuming it's relatively small (maybe dozens of racks at most), hosted in some sort of datacenter (Equinix, Digital Realty, Coresite, etc) that provides the remote hands. Maybe some piece of core network equipment. A small colo may have a single pair of routers, switches, or firewalls at its edge. If one had failed for some reason, and the remote hands removed the wrong one, it is possible you could knock the entire colo offline. There's a bunch of other possible components: Storage platforms, power, maybe something like an HSM storing secrets, or even just a key database server. Their failover to their backup facility may be impaired by the fact that well, their management plane is down. They probably rely on their own services. Avoiding chicken-and-egg issues can require careful ahead-of-time planning.
- deleted 7y ago
  [deleted]
- hinkley 7y ago
  "If things are painful you should do them more often," isn't supposed to be about callouses, but a lot of people take it that way. People with callouses do not experience the activity the way everybody else does. Sounds like a bit of that happened here. I want my most of my mid-level people (and all of the ones bucking for a promotion) to be comfortable running 80-90% of routine maintenance operations and have pretty decent guesses on what should be done for the next 5-10%. I don't often get my way.
bogomipz 7y ago
From your link: >"During planned maintenance remote hands decommissioned some equipment that they shouldn’t have. We’re failing over to a backup facility and working to get the equipment back online." Some questions: When the gear was first powered off why wasn't it just powered back up again as soon as the on-call person's phone started blowing up? Why does this require a "failover" to another datacenter? Why are remote hands decommissioning Cloudflare's gear in the first place? Isn't this supposed to be a security focused company? For context "remote hands" is the term for people that work for the colocation facility such as Eqiunix, Telehouse, etc. They are not employees of the tenant(Cloudflare.) Remote-hands are great resources for things like checking a cable or running new cross connects etc but certainly not decommissioning gear without some form of tenant supervision. Why is this a single point of failure?
- davidu 7y ago
  Just FYI -- Most colo facilities are prohibiting customer access during the COVID-19 lockdowns and have gone to 'smart hands' only for health and safety reasons.
  - bogomipz 7y ago
    Sure but that would seem reason enough to postpone the scheduled maintenance then no? I would think trusting "remote hands" to decommission production gear would carry some serious weight in the risk/reward analysis of keeping the maintenance window. At any rate I would think that those same remote hands should have been able to immediately powered that gear back up as soon as they were made aware of the error.
    - tedk-42 7y ago
      If the risk of the change was minimal, why would they not proceed? How can you plan for things occurring out of your control? CF engineers are people as well. Things like this happen and there will be learnings to take out of it (like how to fail over faster)
      - 7y ago

zymhan 7y ago

How on earth can rented "remote hands" at a datacenter take down the ENTIRE Cloudflare management layer?

MichaelApproved 7y ago
and for over 3 hours? What could they do accidentally that would cause an outage for this long? The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster. Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online. I also wonder how much Covid impacted the process, if at all. I'm looking forward to the details after they get things back online. Good luck CF engineers!
- foobarbazetc 7y ago
  Probably nuked the main DB.
- eb0la 7y ago
  Three hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking. From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before. When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.
mcpherrinm 7y ago
I don't know how big Cloudflare's management layer is, but I'm assuming it's relatively small (maybe dozens of racks at most), hosted in some sort of datacenter (Equinix, Digital Realty, Coresite, etc) that provides the remote hands. Maybe some piece of core network equipment. A small colo may have a single pair of routers, switches, or firewalls at its edge. If one had failed for some reason, and the remote hands removed the wrong one, it is possible you could knock the entire colo offline. There's a bunch of other possible components: Storage platforms, power, maybe something like an HSM storing secrets, or even just a key database server. Their failover to their backup facility may be impaired by the fact that well, their management plane is down. They probably rely on their own services. Avoiding chicken-and-egg issues can require careful ahead-of-time planning.
deleted 7y ago
[deleted]
hinkley 7y ago
"If things are painful you should do them more often," isn't supposed to be about callouses, but a lot of people take it that way. People with callouses do not experience the activity the way everybody else does. Sounds like a bit of that happened here. I want my most of my mid-level people (and all of the ones bucking for a promotion) to be comfortable running 80-90% of routine maintenance operations and have pretty decent guesses on what should be done for the next 5-10%. I don't often get my way.

bogomipz 7y ago

From your link: >"During planned maintenance remote hands decommissioned some equipment that they shouldn’t have. We’re failing over to a backup facility and working to get the equipment back online." Some questions: When the gear was first powered off why wasn't it just powered back up again as soon as the on-call person's phone started blowing up? Why does this require a "failover" to another datacenter? Why are remote hands decommissioning Cloudflare's gear in the first place? Isn't this supposed to be a security focused company? For context "remote hands" is the term for people that work for the colocation facility such as Eqiunix, Telehouse, etc. They are not employees of the tenant(Cloudflare.) Remote-hands are great resources for things like checking a cable or running new cross connects etc but certainly not decommissioning gear without some form of tenant supervision. Why is this a single point of failure?

davidu 7y ago
Just FYI -- Most colo facilities are prohibiting customer access during the COVID-19 lockdowns and have gone to 'smart hands' only for health and safety reasons.
- bogomipz 7y ago
  Sure but that would seem reason enough to postpone the scheduled maintenance then no? I would think trusting "remote hands" to decommission production gear would carry some serious weight in the risk/reward analysis of keeping the maintenance window. At any rate I would think that those same remote hands should have been able to immediately powered that gear back up as soon as they were made aware of the error.
  - tedk-42 7y ago
    If the risk of the change was minimal, why would they not proceed? How can you plan for things occurring out of your control? CF engineers are people as well. Things like this happen and there will be learnings to take out of it (like how to fail over faster)
    - 7y ago