3 ms·
First week as a network engineer, still on probation. I was given the simple task of provisioning a new VLAN onto a few switches, one of which was a fairly larg
by synicalx 10y ago
First week as a network engineer, still on probation. I was given the simple task of provisioning a new VLAN onto a few switches, one of which was a fairly large and important aggregation switch.
Everything was going ok, then I get to the big switch and move over to one of it's port channels and start adding the vlan there;
switchport trunk allowed vlan 123
Shortly after my telnet session dropped. How weird, I thought. I tried re-connecting, no dice. Tried pinging it, nothing.
Then I hear a loud "What the fuck?" from the other side of room, and I look up to see about 30 bright red alerts on our board and a huge flood of red from our GLTail monitoring board showing a very large number of PPPoE session ending suddenly.
I'd missed the word "add" in my command and had wiped the other 200+ vlan's from that interface which had in turn killed the Internet, Phone, and IPTV of about 30,000 customers.
After restarting the switch to get it back to it's startup config, I returned to my desk to find the golden pineapple already displayed prominently on my desk. I also had to wear a cowboy hat for the next 10 changes I made in production.
I'd say lesson learned, but then my co-worker did the exact same thing 2 months later while drunk and on call so I don't think we really learned anything there.
- amingilani 10y agoWhat's the "golden pineapple"?
- synicalx 10y agoLiterally a golden pineapple, it's a gold trophy in the shape of a pineapple awarded to those who stuff something up. Good for a laugh, and also a good reminder to pay attention and try to screw up less.