4 ms·

So, who pushed out the configuration changes and is this person still employed at FB?

by martin1975 5y ago

high_5 5y ago
Also, don't forget the reviewers who approved the change. And the designers of the whole system/process that enabled this thing to fall through this process.
- can16358p 5y ago
  I think it's the design. It should have been fault-tolerant to possible issues exactly like this, and should not allow the configuration to go live in the first place, in case the person or team pushing the configuration makes a mistake (or a sabotage attempt).
sdevonoes 5y ago
Of course! That person learnt a valuable lesson yesterday, so they cannot be fired.
- coretx 5y ago
  Not per se. Depending on the type of error made, that person might have become a pariah no one wants to peer with.
  - rocgf 5y ago
    If it was something truly idiotic or malevolent, sure. Otherwise, that would be a really crap culture to work in.
    - coretx 5y ago
      Many people love to work at let's say, a nucleair powerplant or certain aspects of the military and experience similair pressure to not fail.One could say such work is not for everyone but to state it's "crap culture" is somewhat disrespectful towards the people who do it and the masses who suffer when things go wrong.
      - jhugo 5y ago
        Thankfully, here in the real world, nuclear power plants and similar critical infrastructure in most countries operate with a safety culture, where people are encouraged to report their mistakes, learn from them, and help others avoid making the same mistakes, rather than a punitive culture, where people are incentivised to hide mistakes in order to save their job.
        coretx 5y ago
        BGP is spoken by all cultures, including those who do not know what "safety culture" is, and thats a good thing because it allows for everyone to connect to the internet, bridging cultures. When the critical infrastructure in Texas went down, it had everything to do with corporate culture, trying to save jobs, and not with security culture. The engineers where not to blame, what we discus today is likely a similair case. At nuclear power plants, a single individual can't cause a meltdown due to security culture. In case of AS owners/engineers this is not always the case yet all of them know how not to fail with the exception of some who are ruled by another working culture than those who talk bgp.One that is not punitive but open albeit with social hygiene keeping the incompetent & corrupt out. The recent events are not a mistake to be forgiven but a failure to be studied while not taking risks by eliminating the human source.
  - celticninja 5y ago
    *Pariah Also if someone making a mistake means no one wants to work with them, that seems like a terrible team. Perhaps you should reassess how you deal with failures if that is your go to thought on this.
  - baq 5y ago
    i'd go for a beer or four with that dude
happosai 5y ago
Do you want a workplace culture where people hide their mistakes? Because this is how you create such a culture.
- coretx 5y ago
  People who talk bgp should be above mundane & cruel human dynamics and culture because their function is too important. All bgp speaking people i know are and perhaps thats both why this discussion is held and why the internet is still up and running. "If you don't know what you are doing, don't do it"
  - can16358p 5y ago
    But humans, even the smartest ones with great knowledge and experience in their field, make mistakes. Considering such a big outage last occured 13 years ago, I think it's perfectly normal for an experienced entity (whether a human or more likely a team) to make a big mistake in this time window.
- danielmg 5y ago
  It really depends. I've put someone through a disciplinary before for a really expensive failed change. Not because they made the mistake but because they failed to do any sanity checks before hand or post-imp checks - they just bashed it in and walked away. I'd never go this route with a geniune mistake as they are life's best lessons. But there is a difference between an honest mistake and a mistake that could have been easily avoided had process been followed.
liamwire 5y ago
“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” — Thomas J. Watson, allegedly
- can16358p 5y ago
  And if it was in hands of a single person who made the technical mistake (and we're humans, we make mistakes), the real question is: how can a single person, mistakenly or deliberately, have the power to update BGP config to take the entire infrastructure offline? This is plain wrong, and it shouldn't be possible in the first place.
  - aeonflux 5y ago
    And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to push the keys to perform the operation.
    - throw0101a 5y ago
      > And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...
      - spaceprison 5y ago
        At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.
samuel 5y ago
I remember a blog entry about a guy who took Amazon offline for some minutes and was fired, even while the change had beend aproved by several people. I can't find it right now but that's the kind attitude you would expect from Amazon. I guess what FB is going to do.

So, who pushed out the configuration changes and is this person still employed at FB?

Also, don't forget the reviewers who approved the change. And the designers of the whole system/process that enabled this thing to fall through this process.

can16358p 5y ago
I think it's the design. It should have been fault-tolerant to possible issues exactly like this, and should not allow the configuration to go live in the first place, in case the person or team pushing the configuration makes a mistake (or a sabotage attempt).

sdevonoes 5y ago

Of course! That person learnt a valuable lesson yesterday, so they cannot be fired.

coretx 5y ago
Not per se. Depending on the type of error made, that person might have become a pariah no one wants to peer with.
- rocgf 5y ago
  If it was something truly idiotic or malevolent, sure. Otherwise, that would be a really crap culture to work in.
  - coretx 5y ago
    Many people love to work at let's say, a nucleair powerplant or certain aspects of the military and experience similair pressure to not fail.One could say such work is not for everyone but to state it's "crap culture" is somewhat disrespectful towards the people who do it and the masses who suffer when things go wrong.
    - jhugo 5y ago
      Thankfully, here in the real world, nuclear power plants and similar critical infrastructure in most countries operate with a safety culture, where people are encouraged to report their mistakes, learn from them, and help others avoid making the same mistakes, rather than a punitive culture, where people are incentivised to hide mistakes in order to save their job.
      - coretx 5y ago
        BGP is spoken by all cultures, including those who do not know what "safety culture" is, and thats a good thing because it allows for everyone to connect to the internet, bridging cultures. When the critical infrastructure in Texas went down, it had everything to do with corporate culture, trying to save jobs, and not with security culture. The engineers where not to blame, what we discus today is likely a similair case. At nuclear power plants, a single individual can't cause a meltdown due to security culture. In case of AS owners/engineers this is not always the case yet all of them know how not to fail with the exception of some who are ruled by another working culture than those who talk bgp.One that is not punitive but open albeit with social hygiene keeping the incompetent & corrupt out. The recent events are not a mistake to be forgiven but a failure to be studied while not taking risks by eliminating the human source.
- celticninja 5y ago
  *Pariah Also if someone making a mistake means no one wants to work with them, that seems like a terrible team. Perhaps you should reassess how you deal with failures if that is your go to thought on this.
- baq 5y ago
  i'd go for a beer or four with that dude

happosai 5y ago

Do you want a workplace culture where people hide their mistakes? Because this is how you create such a culture.

coretx 5y ago
People who talk bgp should be above mundane & cruel human dynamics and culture because their function is too important. All bgp speaking people i know are and perhaps thats both why this discussion is held and why the internet is still up and running. "If you don't know what you are doing, don't do it"
- can16358p 5y ago
  But humans, even the smartest ones with great knowledge and experience in their field, make mistakes. Considering such a big outage last occured 13 years ago, I think it's perfectly normal for an experienced entity (whether a human or more likely a team) to make a big mistake in this time window.
danielmg 5y ago
It really depends. I've put someone through a disciplinary before for a really expensive failed change. Not because they made the mistake but because they failed to do any sanity checks before hand or post-imp checks - they just bashed it in and walked away. I'd never go this route with a geniune mistake as they are life's best lessons. But there is a difference between an honest mistake and a mistake that could have been easily avoided had process been followed.

liamwire 5y ago

“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” — Thomas J. Watson, allegedly

can16358p 5y ago
And if it was in hands of a single person who made the technical mistake (and we're humans, we make mistakes), the real question is: how can a single person, mistakenly or deliberately, have the power to update BGP config to take the entire infrastructure offline? This is plain wrong, and it shouldn't be possible in the first place.
- aeonflux 5y ago
  And what would you expect? A secret gathering of company elders to commit the change? I guess this change was approved by someone, but in the end someone has to push the keys to perform the operation.
  - throw0101a 5y ago
    > And what would you expect? A secret gathering of company elders to commit the change? Push your changes to a simulated network to see what it does before pushing to real hardware: > VIRL is a network virtualization simulator by Cisco. Similar to the likes of GNS3 and EVE-NG. VIRL will be used to build a test network (shown below), so that we can validate our changes prior to deploying them to production. * https://www.packetcoders.io/netdevops-ci-cd-with-ansible-github-jenkins-and-cisco-virl-part-2-the-components-tools/ https://www.packetcoders.io/netdevops-ci-cd-with-ansible-git...
    - spaceprison 5y ago
      At FB/AMZN scale its really tough to accurately emulate/virtualize the network, these are deep clos fabrics with LOTS OF STATE. At best you can get a sense of what protocols might/should do, but at the end of the day the state of the real environment isn't present. So that micro-loop you created with your new policy that you observed in the lab turns in to a wave of micro-loops in real life because some random router was slow to update due to load.

samuel 5y ago

I remember a blog entry about a guy who took Amazon offline for some minutes and was fired, even while the change had beend aproved by several people. I can't find it right now but that's the kind attitude you would expect from Amazon. I guess what FB is going to do.