4 ms·

And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to

And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to push the keys to perform the operation.

throw0101a 5y ago
> And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...
- spaceprison 5y ago
  At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.
  - throw0101a 5y ago
    You're not wrong, but it would be interesting to see some 'hyperscale' operator publish a paper on how many simulated routers you'd need to get a somewhat accurate test: 0.1, 1, 10% of a network?
nysv 5y ago
Automated testing and validation? Something like what the linked article says they have in place? "To minimize impact on production traffic while achieving high release velocity for the BGP agent, we built our own testing and incremental deployment framework, consisting of unit testing, emulation, and canary testing. We use a multi-phase deployment pipeline to push changes to agents."

throw0101a 5y ago

> And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...

spaceprison 5y ago
At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.
- throw0101a 5y ago
  You're not wrong, but it would be interesting to see some 'hyperscale' operator publish a paper on how many simulated routers you'd need to get a somewhat accurate test: 0.1, 1, 10% of a network?

nysv 5y ago

Automated testing and validation? Something like what the linked article says they have in place? "To minimize impact on production traffic while achieving high release velocity for the BGP agent, we built our own testing and incremental deployment framework, consisting of unit testing, emulation, and canary testing. We use a multi-phase deployment pipeline to push changes to agents."