3 ms·

From your link: >"During planned maintenance remote hands decommissioned some equipment that they shouldn’t have. We’re failing over to a backup facility and w

From your link: >"During planned maintenance remote hands decommissioned some equipment that they shouldn’t have. We’re failing over to a backup facility and working to get the equipment back online." Some questions: When the gear was first powered off why wasn't it just powered back up again as soon as the on-call person's phone started blowing up? Why does this require a "failover" to another datacenter? Why are remote hands decommissioning Cloudflare's gear in the first place? Isn't this supposed to be a security focused company? For context "remote hands" is the term for people that work for the colocation facility such as Eqiunix, Telehouse, etc. They are not employees of the tenant(Cloudflare.) Remote-hands are great resources for things like checking a cable or running new cross connects etc but certainly not decommissioning gear without some form of tenant supervision. Why is this a single point of failure?

davidu 7y ago
Just FYI -- Most colo facilities are prohibiting customer access during the COVID-19 lockdowns and have gone to 'smart hands' only for health and safety reasons.
- bogomipz 7y ago
  Sure but that would seem reason enough to postpone the scheduled maintenance then no? I would think trusting "remote hands" to decommission production gear would carry some serious weight in the risk/reward analysis of keeping the maintenance window. At any rate I would think that those same remote hands should have been able to immediately powered that gear back up as soon as they were made aware of the error.
  - tedk-42 7y ago
    If the risk of the change was minimal, why would they not proceed? How can you plan for things occurring out of your control? CF engineers are people as well. Things like this happen and there will be learnings to take out of it (like how to fail over faster)
    - bogomipz 7y ago
      What you mean "what if"? Clearly the risk was not minimal if it resulted in a 3 hour outage and affected customers. Someone at Cloudflare should have been able to identify that the services that were colocated at this facility had no redundancy prior to asking a third party to power down gear. Foregoing the maintenance until proper redundancy was in place was not out of their control. Circulating a MOP(method of procedure) for data center maintenance among all stakeholder is pretty standard. The purpose of the MOPS is so that everyone can vet the plan(roll forward and roll back)and identify the risks.
      - davidu 7y ago
        It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen. As Matthew said on Twitter, this isn't the kind of mistake that happens twice. But for those of us who do operational work, it's easy to see how this happens once. Bit rot in a cab, work orders from someone who didn't know about the legacy patch panel or assumed too much in their instructions and a catastrophe. From now on, photos for every smart hands will probably be a part of the prep-procedure.
        bogomipz 7y ago
        >"It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen." That's a pretty odd and flimsy argument that I can't know what I'm talking about if I hold an opinion that differs from your own. I have years of experience with this particular space. Had a MOP been circulated and the maintenance plan properly vetted then the need for visual confirmation because of "legacy patch panel" and "bit rot in cab" would have been identified. Further you never unplug a cable unless you first verify what the ends of the cable are connected to. As I mentioned this is "datacenter operations 101" stuff. This is not some new startup, this a publicly traded company who has been in the game for a decade now.

davidu 7y ago

Just FYI -- Most colo facilities are prohibiting customer access during the COVID-19 lockdowns and have gone to 'smart hands' only for health and safety reasons.

bogomipz 7y ago
Sure but that would seem reason enough to postpone the scheduled maintenance then no? I would think trusting "remote hands" to decommission production gear would carry some serious weight in the risk/reward analysis of keeping the maintenance window. At any rate I would think that those same remote hands should have been able to immediately powered that gear back up as soon as they were made aware of the error.
- tedk-42 7y ago
  If the risk of the change was minimal, why would they not proceed? How can you plan for things occurring out of your control? CF engineers are people as well. Things like this happen and there will be learnings to take out of it (like how to fail over faster)
  - bogomipz 7y ago
    What you mean "what if"? Clearly the risk was not minimal if it resulted in a 3 hour outage and affected customers. Someone at Cloudflare should have been able to identify that the services that were colocated at this facility had no redundancy prior to asking a third party to power down gear. Foregoing the maintenance until proper redundancy was in place was not out of their control. Circulating a MOP(method of procedure) for data center maintenance among all stakeholder is pretty standard. The purpose of the MOPS is so that everyone can vet the plan(roll forward and roll back)and identify the risks.
    - davidu 7y ago
      It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen. As Matthew said on Twitter, this isn't the kind of mistake that happens twice. But for those of us who do operational work, it's easy to see how this happens once. Bit rot in a cab, work orders from someone who didn't know about the legacy patch panel or assumed too much in their instructions and a catastrophe. From now on, photos for every smart hands will probably be a part of the prep-procedure.
      - bogomipz 7y ago
        >"It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen." That's a pretty odd and flimsy argument that I can't know what I'm talking about if I hold an opinion that differs from your own. I have years of experience with this particular space. Had a MOP been circulated and the maintenance plan properly vetted then the need for visual confirmation because of "legacy patch panel" and "bit rot in cab" would have been identified. Further you never unplug a cable unless you first verify what the ends of the cable are connected to. As I mentioned this is "datacenter operations 101" stuff. This is not some new startup, this a publicly traded company who has been in the game for a decade now.