3 ms·
> I'd love to see how other people do it I worked in this space for over a decade, doing some pretty complex things along the way, like mpls cutovers involving
by mikena 4y ago
> I'd love to see how other people do it
I worked in this space for over a decade, doing some pretty complex things along the way, like mpls cutovers involving multiple devices and qos deployments, across tens of thousands of devices. Time savings and eliminating the errors of manual deployments were the big drivers. I ended up refining a set of tools that I used and releasing them as a set of perl cpan modules, Mnet [0].
The goal was to be able to make automation scripts that can ssh into devices, parse command outputs, generate reports and debug logs, prepare and push new configs, and record and replay tests. The sample script in the readme serves as a quick walkthrough and a good starting project template. You can run your scripts on one device, do offline replays during development, or concurrently batch process a list of devices.
Reporting and testing are key to successful deployments. First you'll want to report on the current configs, flagging anomalies or anything unexpected. Early in a project you are looking for one-offs in reports, to make sure your script exits if it encounters something it can't handle. You'll also want to identify different scenarios you'll encounter. You can have the script generate an output config for these different scenarios, for review. You can manually run the script on a device and have it record the ssh session and config output to a file, which the script can replay as a regression test. You can add a --deploy option to your script along with code to push the new config to a device, reporting on success or failure. Rollback procedures may vary by project, often we manually handled errors using script output of the prior and new configs, along with the details log outputs. We'd deploy to a couple devices first, then a few devices for every scenario we coded for, then larger numbers as we gained confidence, deploying only to devices that had no errors in the reports. Following the above best practices we didn't end up with many rollbacks. One of our most complex scripts, for qos deployments, averaged maybe a dozen errors per batch of a thousand devices, mostly transient offline devices or authentication errors.
I had some experience with hp openview and ansible, but much preferred my tools. The test record/replay functions are so important to doing anything non-trivial, and were missing in these other systems. The perl code I have is also a lot faster, and perl is well suited to dealing with text network devices configs.
The vlan creation project you mentioned is do-able, but definitely has some meat on it, worth the effort if it would be used often enough. You'd need to identify the minimum inputs needed, and what extra info needs to be harvested from the network. A single script can log into multiple devices, as long as it has credentials available. Perhaps you have a hardcoded list of core switches, or follow routing tables from the targeted access switch to the connected core devices. I'm not sure how much variability you might encounter, besides the vm trunks and vms stuff you mentioned, like ios vs nexus devices using slightly different commands. You don't need to handle everything, but you might want the script to identify those scenarios it can't handle. You'd probably want to check that the new vlan doesn't exist on the access or core switches, that the access port is not already assigned a vlan or l3 address, the l3 address is not already in use elsewhere, the access port is not already in a vlan, that the uplinks can be identified and are already trunked, etc. Perhaps after the deployment you run show commands to check spanning tree, routes, etc. Ideally the deployment engineer can run the script read-only, verify everything looks good, then --deploy and get an 'ok' back. If done well the script serves to document the process.
Feel free to email me, my contact info is in cpan. I'm game to help.
[0] https://metacpan.org/pod/Mnet https://metacpan.org/pod/Mnet