4 ms·
That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events. One thing that occurred to m
by rainsford 14y ago
That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events.
One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn't have out-of-band remote power cycle capabilities.
There may be some factor I'm not considering that would make that an unattractive option, but it does seem like it could cut down an already quick response time even further for any similar events in the future.
- rdl 14y agoI've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though. I think it's mostly that the routers usually have both good OOB management and good watchdog (reboot on freeze) behavior, and that the PSUs in the bigger routers tend to exceed the per-port power limits of most of the external power cyclers. It may be a good idea, though.
- justincormack 14y agoYou would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.
- mprovost 14y agoWe just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.
- macros 14y agoWe've been doing this with mikrotik boxes with either wifi or usb gsm modems depending on the what is available in the location.
- rdl 14y agoYeah, I've seen a lot of great options for OOB access: 1) At carrier hotels, wifi (heh) 2) Cellular modems (ideal for branch offices; a lot of datacenters have bad cell coverage inside the racks/cabinets/floor though) 3) Cross-connect (in places with free/cheap cross connects) to someone you don't use for transit. Can be mutual 4) Some facilities give you an OOB network, although this often has issues (if you buy transit from them, it's possible your outage is due to something going wrong with them, and it might take out your OOB access) I'm looking at the Verizon Private-IP thing (an outsourced private network over Verizon's cell infrastructure) for OOB management of lots of CPE; the cost per device per month is low, and then you pay for bandwidth across all of them. Makes initial provisioning easier, plus ongoing monitoring/maintenance.
- packetslave 14y agoA lot of big routers run (or have the option to run) on DC power, too, which makes it harder. Much of the remote power control gear out there is AC