4 ms·
I recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage
by catminou 4y ago
I recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage. They had switches (Extreme switches - known for their purple color aka Barney the dinosaur switches) in the basement or many buildings and backhauled to a colo to the rest of their network.
They were doing some mass network upgrade during the early hours maintenance window and devices weren't coming back. I don't think they had OOB access to their boxes and it requires someone going out and recovering them after hours of downtime and getting techs there.
The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades.
- jdhendrickson 4y agoI had a boss purchase those switches without consulting me, he liked the color. They were pure garbage. Thanks for reminding me of that, I'll have to ping a former co-worker and have a laugh.
- bpye 4y ago> The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades. I've always been paranoid about this sort of thing when applying BIOS or firmware updates, despite those almost always having checksum validation, but I guess my caution is not unwarranted.