4 ms·

The network problems seem to have started earlier. I received a penny for my NYC2 droplet just before this major incident. Luckily, I decided to shutdown and mo

by akg_67 12y ago

The network problems seem to have started earlier. I received a penny for my NYC2 droplet just before this major incident. Luckily, I decided to shutdown and move the droplet to SFO before this incident. Over the last couple of years dealing with cloud providers, I have learned to dust off the contingency plan as soon there is whiff of issues at a location. Though the recap is good, there are red flags that show gaps in DO processes and incident management. Upgrade of software before the original problem was diagnosed and resolved. This is a big no-no, never introduce a new variable in an existing problem even if the service provider or lab testing shows the chances of failure are minimal. I have worked long enough with technology infrastructure to experience situations where service provider insisted on upgrading software during unrelated incident and made situation worse. A better approach would have been to fail the network to good switch first and when the bad switch was fixed, upgrade the software on bad switch first then failover to the upgraded switch and upgrade software on good switch. This time you guys got lucky but sooner or later your luck will run out.

imbriaco 12y ago
Generally speaking I agree with you. However, in this specific case there were a couple of reasons we chose the path we did: 1. We had experienced bugs with the currently running release which we were fairly sure would manifest when we removed the damaged core from the network. These were primarily around MAC learning. 2. We had performed testing ourselves in a non-production environment and were already planning to take a network maintenance to update these switches in the next few days. Given those factors, we judged that introducing another variable in the new version was less risky than proceeding with the defects that we knew about in the existing version.

by akg_67 12y ago

imbriaco 12y ago

Generally speaking I agree with you. However, in this specific case there were a couple of reasons we chose the path we did: 1. We had experienced bugs with the currently running release which we were fairly sure would manifest when we removed the damaged core from the network. These were primarily around MAC learning. 2. We had performed testing ourselves in a non-production environment and were already planning to take a network maintenance to update these switches in the next few days. Given those factors, we judged that introducing another variable in the new version was less risky than proceeding with the defects that we knew about in the existing version.