3 ms·
The router config is not the top of the stack though. There should be orchestration systems that push out changes to these devices and can roll them back when r
by zeroimpl 4y ago
The router config is not the top of the stack though. There should be orchestration systems that push out changes to these devices and can roll them back when required.
- ajross 4y agoIn general, you can't do any kind of automated rollback when the network goes down (same deal for DNS, which is why I mentioned it -- similar datacenter-wide failures have been due to stuff like stampeding initialization herds, etc...). I get what you're saying, I'm just pointing out that it's a little naive. No matter how high you make your stack of abstractions, there's still a human being at the top of the stack with fat fingers.
- zeroimpl 4y agoSure but that’s why I think the take-away here is to design the system such that you can do such automated rollbacks. For example, the devices should also be connected to some sort of isolated management network. Granted, pushing out a config to the routers could also break their connections to said management network, but it is less likely anyways.