2 ms·

We had a similar problem. Medium sized real-time trading firm HQd in the midwest. Had a bunch of dell stuff at our federally required DR datacenter. Dell tech (

We had a similar problem. Medium sized real-time trading firm HQd in the midwest. Had a bunch of dell stuff at our federally required DR datacenter. Dell tech (or whoever, I forget exactly, big company though) enters secure DR datacenter to put in a new blade in our blade server enclosure. I guess plugs a test router into our network via cat-5, then plugs in the power. Test router has the same static IP as some piece of Critical Infrastructure. Suddenly all of our real time trades are being routed to the Dell tech's test router, just into a black hole. All of our network config also gets routed to his test router in our DR datacenter. Everything stops. Even my personal workstation in our office, 10 miles away, my internet traffic gets routed to this guy's test router. Even our VOIP phones get routed into this black hole. 100s of thousands of day trades getting routed into the black hole that is his test router. Tech guy has no idea any of this is happening. Finally after a few minutes,our head of IT/networking/everything finds the tech's personal cell phone, and calls him from his own personal cell phone, because VOIP is dead in the water now. Tech unplugs network device from network, within 5 min everything self-heals. Untold amount of fines and fees for not being able to handle trades during the outage.

kevin_nisbet 4y ago
Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where: 1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat. I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.

kevin_nisbet 4y ago

Interesting. Yea we had a few of these due to I think it was vlan trunking protocol but I don't really remember. We had a cycle over 3 or 4 years where: 1. Configuring VLANs is hard, let's turn on the Cisco feature to sync VLANs 2. Network switch from lab or another network would be moved to production, with a higher revision index. All switches sync from the highest revision. 3. Outage happens because VLANs in lab are not same as production. 4. Investigation says vlan trunking protocol is dangerous, and should be disabled. 5. A year later, forget about the outage and repeat. I think eventually there was a feature that sort of solved this problem, but I don't remember after all this time.