3 ms·
The default way that most networking devices are managed is crazy in this day and age. Like the post-mortem says, they will put mitigations in place, but this
by jiggawatts 4y ago
The default way that most networking devices are managed is crazy in this day and age.
Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous.
I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here.
What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.
Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one.
This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds.
Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts!
(I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)
- inferiorhuman 4y agoThe power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network. Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).
- jiggawatts 4y agoAgain, you are talking about safety systems and mitigations that device customers “should” implement bespoke themselves. I’m saying this is the problem — the device configuration approach should be safe by default.
- inferiorhuman 4y agoYes/no – something like a watchdog timer should be part of whatever OS the router is running. Remote power management would be closer to bespoke but ought not to be too exotic.
- llama052 4y agoLuckily a lot of modern networking equipment has an automated rollback feature you can key off of. For instance junos based devices have a commit confirmed <time> where it will auto rollback if not confirmed after X amount of time. Still pretty dated but they are designed to be reliable first.
- jrockway 4y ago> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection. Always good. The system built into display settings ("click yes if you can read this, or the change will be reverted in 15 seconds") has saved me a number of times. No reason not to apply that to other settings where the data channel is the same as the control channel.
- jhugo 4y agoEven 20 years ago at a nationwide ISP we used to have a completely separate emergency management network over GSM. I'm amazed that at CF's scale management is still done the way described.
- badcoder72684 4y ago> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection. OpenWRT router distribution has had this for years, it's amazing! (As is OpenWRT) OpenWRT also has SQM CAKE which saved my sanity on parents DSL connection for years. As far as congestion control and bandwidth sharing goes, nothing else compares
- icehawk 4y agoIf what's presented in the blog is actual configuration that's very much a junos device and definitely has the ability to have commit with confirm, auto rollback, and commit history. Not using it is more of an issue with the automation, as they said. (Which is an issue with the /automation/ defaults. I've learned enough to do commit confirm, but by default ansible does a hard commit.)