3 ms·

Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficul

Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.

yRetsyM 14y ago
Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling. My 2cents: Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff) It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live. The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.
nathannecro 14y ago
Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?
- toast0 14y ago
  Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact. If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.

yRetsyM 14y ago

Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling. My 2cents: Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff) It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live. The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.

nathannecro 14y ago

Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?

toast0 14y ago
Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact. If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.