5 ms·

“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him.

“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” — Thomas J. Watson, allegedly

can16358p 5y ago
And if it was in hands of a single person who made the technical mistake (and we're humans, we make mistakes), the real question is: how can a single person, mistakenly or deliberately, have the power to update BGP config to take the entire infrastructure offline? This is plain wrong, and it shouldn't be possible in the first place.
- aeonflux 5y ago
  And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to push the keys to perform the operation.
  - throw0101a 5y ago
    > And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...
    - spaceprison 5y ago
      At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.
      - throw0101a 5y ago
        You're not wrong, but it would be interesting to see some 'hyperscale' operator publish a paper on how many simulated routers you'd need to get a somewhat accurate test: 0.1, 1, 10% of a network?
  - nysv 5y ago
    Automated testing and validation? Something like what the linked article says they have in place? "To minimize impact on production traffic while achieving high release velocity for the BGP agent, we built our own testing and incremental deployment framework, consisting of unit testing, emulation, and canary testing. We use a multi-phase deployment pipeline to push changes to agents."
thih9 5y ago
Then again, training efficiency isn't the same for every employee. If this particular employee has been trained for this situation in the past and still required training, then this employee might need more training in future. Perhaps it is more cost effective to work with a different employee.

can16358p 5y ago

And if it was in hands of a single person who made the technical mistake (and we're humans, we make mistakes), the real question is: how can a single person, mistakenly or deliberately, have the power to update BGP config to take the entire infrastructure offline? This is plain wrong, and it shouldn't be possible in the first place.

aeonflux 5y ago
And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to push the keys to perform the operation.
- throw0101a 5y ago
  > And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...
  - spaceprison 5y ago
    At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.
    - throw0101a 5y ago
      You're not wrong, but it would be interesting to see some 'hyperscale' operator publish a paper on how many simulated routers you'd need to get a somewhat accurate test: 0.1, 1, 10% of a network?
- nysv 5y ago
  Automated testing and validation? Something like what the linked article says they have in place? "To minimize impact on production traffic while achieving high release velocity for the BGP agent, we built our own testing and incremental deployment framework, consisting of unit testing, emulation, and canary testing. We use a multi-phase deployment pipeline to push changes to agents."

thih9 5y ago

Then again, training efficiency isn't the same for every employee. If this particular employee has been trained for this situation in the past and still required training, then this employee might need more training in future. Perhaps it is more cost effective to work with a different employee.