5 ms·

Network Automation Expert Beginners

62 points by zdw 4y ago

lasereyes136 4y ago
> The problem is that the more automation we push, the fewer people know how to use the “old school” way to administer stuff. Yes, that is the irony of automation, when something goes wrong you need a higher level of understanding to fix the problem. No matter how good the automation is something will go wrong because there are situations that automation doesn't know how to handle and you can't prepare for.
simula67 4y ago
> Instead of an in-depth discussion of architectures, data structures, software development methodologies, and challenges of modifying the state of a distributed system, those motivational talks often resulted in a cargo cult of expert beginners focused on low-level tools. We may still be in the early innings of the network automation game. One factor that makes network automation hard is that networking is still lagging behind other resources (such as storage, compute, etc.) in openness and modularity. Network devices, in many places, use proprietary gear from Cisco, etc. that has a custom user interface (modern Cisco devices do have bash). Many networks are still managed using older protocols such as SNMP although newer protocols such as OpenConfig seem to be gaining ground. Many companies are trying to disaggregate networks (commodity hardware + Linux) but it still has not taken off that much. Many interesting projects (such as switchdev) are languishing. Big companies such as Facebook, Microsoft, Amazon etc are building their own Network OSes (FBoss, SONiC, DentOS, etc) rather than consolidating under a common standard. Lack of resources does not help. Even Wikipedia did not have articles on basic networking concepts until recently. The textbooks that teach this subject talk about older abandoned protocols. We will see improved automation techniques in networking as time progresses.
superbaconman 4y ago
I've been building network controllers my entire career and the "hard part" has never been interacting with devices. Like who cares if you're using Ansible? The hard part is configuring these things; Every vendor has a different way of configuring the same 5 solutions to the same problem, and even with a single vendor network you may have to tailor your configs based on the device type that you're configuring. IMO there needs to be some higher level abstractions emerge. Half the time I just need a layer 2 network (please don't ask me to configure it one way for a point-to-point vs greater than two endpoints. if there's an optimization to be made do it for me). The other half the time I need a layer 3 network that can peer with BGP (don't ask me what underlay or overlay protocols to use, I only care about the VLANs I'm terminating on). Slightly related: Cloud providers expose APIs for networks to setup peerings between the cloud provider's network and the network's end customers. These APIs aren't standardized at all, and are a complete PITA for networks that try to integrate their SDN solution with multiple cloud providers. This is one area of complexity that the big guys could cleanup without a lot of work.
- tryauuum 4y ago
  I don't know about network automation much. Is NAPALM still a thing, is it viable?
- dfex 4y ago
  Agreed - this is more or less the problem statement that OpenConfig was designed to solve (minus the making optimisation part), however the chicken-and-egg problem for vendors is always: few people are asking for it, because until every vendor supports it, only a few people will use it.
  - ilyt 4y ago
    Well, industry tried before with NETCONF, good luck...
    - beams_of_light 4y ago
      NETCONF doesn’t abstract anything though. All it did was provide a programmatic method for configuring devices.
pm2222 4y ago
I wish I were able to work in a large network environment and see for myself why network automation is needed. Sure it’s useful but to what degree? Information highway and highway share one trait: not much flexibility is needed at least that’s the case for small/mid network.
- donavanm 4y ago
  At certain points its not “useful”, but mandatory to simply keep functioning. Even small failure and change rates are untenable as device and port counts increase to Very Large numbers. How do you maintain millions of devices? Think on the order of hundreds to thousands if automated remediation workflows for every exception that requires a person in network operations to intervene. Once you move beyond simple deployment and break/fix theres a huge step function in the change count as well. Networking is far behind on concepts like “continuous deployment” or software updates its not funny. And unlike everyones favorite stateless container of the day networks take coordination to perform intrusive maintenance.
- dfex 4y ago
  I mostly agree with this sentiment - working for systems integrators over the years, it is very hard to convince me that most corporate customers need network automation solutions, when on average they deploy a new network maybe once every 5 years, and when the deployment is done, it structurally changes very little until the next hardware replacement. However there are definitely business in the managed service provider, ISP and obviously anyone with a large growing data centre environment where consistency and accuracy in large-scale deployemnts are important.
  - Thlom 4y ago
    Most managed service providers and ISP's have developed their own simple "automation" tools for managing customers network. It's easy when the config is mostly the same across all customers.
    - dfex 4y ago
      Exactly, and the change count (add new customer with 50 sites, remove old customer with 100 sites) are big enough to warrant investment of time/money into these tools
iso1631 4y ago
Everything I see in this area tends to be "here's how to configure ansible to log into a couple of switches and do a show run". Whoopy do. This is a refreshing blog, at least confiming my suspicions that nobody doing network automation actually says anything about it. My team (of 2) run an internal mixed-vendor network of about 150 switches and routers. New sites are deployed from templates, that's easy enough, although it's not "automation". Testing and rollback is pretty tricky when you're talking about physical bits of hardware. I’ve written our own internal stuff to manage part of the network - managing firewall/nat rules on fortigates for example, and obviously theres a great deal of useful open tools out there to report the current state of the network (rancid, librenms, portmappers), but to actually make atomic attributed changes is tricky and relies on the person say changing an interface from vlan 301 to vlan351 to record the reason in the change log (description says what should be on the end and a jira reference for more about it) Currently I’m trying to automate vlan creation on our larger sites. We tend to have a standard dual pair distribution (not stacked - that’s a single point of failure I’ve seen take us out time and time again) vrrp based site with say 30 access switches hanging off. Creating a new vlan involves 1) adding to ipam (vlan and the ip range, then adding the individual hosts) 2) creating the vlans on the core switch 3) adding the vlans to the interface between the two cores 4) adding the l3 interface (including setting the right priority for which core I want to be the normal master), setting spanning tree and pim priorities to match 5) configuring dhcp 6) adding vlans to specific trunk ports to the access switches 7) adding vlans to the access switches and their uplinks And then often 8) adding vlans to vm trunk interfaces 9) creating networks on vms It’s a cesspit of manual errors, and takes an age, almost as bad as punching holes in the firewalls. Automation for automating bgp adjacencies? Meh. There’s very little time saving. For setting vlans on access ports on agile deployments? Again we have web based tools for that to let end users do it, but that's not a pr/test/commit/apply/test/rollback workflow. Currently my CI process pulls the show run from the Ciscos and works out what needs to happen to perform the steps, if any, then put out the instructions as an artifact. I guess next step would be a manual push of the artefact from the CI, and then testing that the IPs actually are resolvable, maybe create a test VM machine on the network (in the case the vlan is pushed to a VM host) Trouble is the testing and rollback process is extremely tricky, especially with Cisco IOS based kit. Network kit can’t be easily or cheaply virtualised (I tried using terraform to deploy a small copy of our network onto an AWS bare-metal system, but the $5/hour you need for bare metal to run gns3). We can't afford to have an identical network setup, so any test system would not scale to the size where automation would tease out problems. I'd love to see how other people do it, but when I talk to the department which looks after our external networking they just ssh into switches and make changes and hope for the best. Online as I say it's all people with ansible running "show version" and then saying "the rest is left as an exercise to the reader". Vendors want to sell proprietary management systems which don't seem to be the kinds of things you can drive from a gitops workflow. Maybe some SDN overlay network can be automated in some fashion, but you're basically learning a new CLI rather than implementing a test-based automation system.

62 points by zdw 4y ago

> The problem is that the more automation we push, the fewer people know how to use the “old school” way to administer stuff. Yes, that is the irony of automation, when something goes wrong you need a higher level of understanding to fix the problem. No matter how good the automation is something will go wrong because there are situations that automation doesn't know how to handle and you can't prepare for.

simula67 4y ago

> Instead of an in-depth discussion of architectures, data structures, software development methodologies, and challenges of modifying the state of a distributed system, those motivational talks often resulted in a cargo cult of expert beginners focused on low-level tools. We may still be in the early innings of the network automation game. One factor that makes network automation hard is that networking is still lagging behind other resources (such as storage, compute, etc.) in openness and modularity. Network devices, in many places, use proprietary gear from Cisco, etc. that has a custom user interface (modern Cisco devices do have bash). Many networks are still managed using older protocols such as SNMP although newer protocols such as OpenConfig seem to be gaining ground. Many companies are trying to disaggregate networks (commodity hardware + Linux) but it still has not taken off that much. Many interesting projects (such as switchdev) are languishing. Big companies such as Facebook, Microsoft, Amazon etc are building their own Network OSes (FBoss, SONiC, DentOS, etc) rather than consolidating under a common standard. Lack of resources does not help. Even Wikipedia did not have articles on basic networking concepts until recently. The textbooks that teach this subject talk about older abandoned protocols. We will see improved automation techniques in networking as time progresses.

superbaconman 4y ago

I've been building network controllers my entire career and the "hard part" has never been interacting with devices. Like who cares if you're using Ansible? The hard part is configuring these things; Every vendor has a different way of configuring the same 5 solutions to the same problem, and even with a single vendor network you may have to tailor your configs based on the device type that you're configuring. IMO there needs to be some higher level abstractions emerge. Half the time I just need a layer 2 network (please don't ask me to configure it one way for a point-to-point vs greater than two endpoints. if there's an optimization to be made do it for me). The other half the time I need a layer 3 network that can peer with BGP (don't ask me what underlay or overlay protocols to use, I only care about the VLANs I'm terminating on). Slightly related: Cloud providers expose APIs for networks to setup peerings between the cloud provider's network and the network's end customers. These APIs aren't standardized at all, and are a complete PITA for networks that try to integrate their SDN solution with multiple cloud providers. This is one area of complexity that the big guys could cleanup without a lot of work.

tryauuum 4y ago
I don't know about network automation much. Is NAPALM still a thing, is it viable?
dfex 4y ago
Agreed - this is more or less the problem statement that OpenConfig was designed to solve (minus the making optimisation part), however the chicken-and-egg problem for vendors is always: few people are asking for it, because until every vendor supports it, only a few people will use it.
- ilyt 4y ago
  Well, industry tried before with NETCONF, good luck...
  - beams_of_light 4y ago
    NETCONF doesn’t abstract anything though. All it did was provide a programmatic method for configuring devices.

pm2222 4y ago

I wish I were able to work in a large network environment and see for myself why network automation is needed. Sure it’s useful but to what degree? Information highway and highway share one trait: not much flexibility is needed at least that’s the case for small/mid network.

donavanm 4y ago
At certain points its not “useful”, but mandatory to simply keep functioning. Even small failure and change rates are untenable as device and port counts increase to Very Large numbers. How do you maintain millions of devices? Think on the order of hundreds to thousands if automated remediation workflows for every exception that requires a person in network operations to intervene. Once you move beyond simple deployment and break/fix theres a huge step function in the change count as well. Networking is far behind on concepts like “continuous deployment” or software updates its not funny. And unlike everyones favorite stateless container of the day networks take coordination to perform intrusive maintenance.
dfex 4y ago
I mostly agree with this sentiment - working for systems integrators over the years, it is very hard to convince me that most corporate customers need network automation solutions, when on average they deploy a new network maybe once every 5 years, and when the deployment is done, it structurally changes very little until the next hardware replacement. However there are definitely business in the managed service provider, ISP and obviously anyone with a large growing data centre environment where consistency and accuracy in large-scale deployemnts are important.
- Thlom 4y ago
  Most managed service providers and ISP's have developed their own simple "automation" tools for managing customers network. It's easy when the config is mostly the same across all customers.
  - dfex 4y ago
    Exactly, and the change count (add new customer with 50 sites, remove old customer with 100 sites) are big enough to warrant investment of time/money into these tools

iso1631 4y ago

Everything I see in this area tends to be "here's how to configure ansible to log into a couple of switches and do a show run". Whoopy do. This is a refreshing blog, at least confiming my suspicions that nobody doing network automation actually says anything about it. My team (of 2) run an internal mixed-vendor network of about 150 switches and routers. New sites are deployed from templates, that's easy enough, although it's not "automation". Testing and rollback is pretty tricky when you're talking about physical bits of hardware. I’ve written our own internal stuff to manage part of the network - managing firewall/nat rules on fortigates for example, and obviously theres a great deal of useful open tools out there to report the current state of the network (rancid, librenms, portmappers), but to actually make atomic attributed changes is tricky and relies on the person say changing an interface from vlan 301 to vlan351 to record the reason in the change log (description says what should be on the end and a jira reference for more about it) Currently I’m trying to automate vlan creation on our larger sites. We tend to have a standard dual pair distribution (not stacked - that’s a single point of failure I’ve seen take us out time and time again) vrrp based site with say 30 access switches hanging off. Creating a new vlan involves 1) adding to ipam (vlan and the ip range, then adding the individual hosts) 2) creating the vlans on the core switch 3) adding the vlans to the interface between the two cores 4) adding the l3 interface (including setting the right priority for which core I want to be the normal master), setting spanning tree and pim priorities to match 5) configuring dhcp 6) adding vlans to specific trunk ports to the access switches 7) adding vlans to the access switches and their uplinks And then often 8) adding vlans to vm trunk interfaces 9) creating networks on vms It’s a cesspit of manual errors, and takes an age, almost as bad as punching holes in the firewalls. Automation for automating bgp adjacencies? Meh. There’s very little time saving. For setting vlans on access ports on agile deployments? Again we have web based tools for that to let end users do it, but that's not a pr/test/commit/apply/test/rollback workflow. Currently my CI process pulls the show run from the Ciscos and works out what needs to happen to perform the steps, if any, then put out the instructions as an artifact. I guess next step would be a manual push of the artefact from the CI, and then testing that the IPs actually are resolvable, maybe create a test VM machine on the network (in the case the vlan is pushed to a VM host) Trouble is the testing and rollback process is extremely tricky, especially with Cisco IOS based kit. Network kit can’t be easily or cheaply virtualised (I tried using terraform to deploy a small copy of our network onto an AWS bare-metal system, but the $5/hour you need for bare metal to run gns3). We can't afford to have an identical network setup, so any test system would not scale to the size where automation would tease out problems. I'd love to see how other people do it, but when I talk to the department which looks after our external networking they just ssh into switches and make changes and hope for the best. Online as I say it's all people with ansible running "show version" and then saying "the rest is left as an exercise to the reader". Vendors want to sell proprietary management systems which don't seem to be the kinds of things you can drive from a gitops workflow. Maybe some SDN overlay network can be automated in some fashion, but you're basically learning a new CLI rather than implementing a test-based automation system.