3 ms·
> Last week the new aggregation switches finally arrived and were installed in our datacenter. It sounds like you rushed these switches into production, maybe
by jauer 14y ago
> Last week the new aggregation switches finally arrived and were installed in our datacenter.
It sounds like you rushed these switches into production, maybe with insufficient testing.
There are all kinds of bugs and weird interactions in network hardware and software that cause problems that you can't anticipate.
You have got to lab it up and do sanity checks before deploying (referring specifically to the lacp/port chan problems).
- Diederich 14y agoWe always, always, always evacuate the datacenter before we do any non-trivial network changes. And then trickle traffic back in. Lab testing is fine, and is a must, but it may or may not have caught this. Datacenter agility is the only way to scale and have high availability.
- nixgeek 14y agoBut you also work for Facebook, and consequently have slightly larger scale problems than most get to worry about. True multi-datacenter availability in an active-active manner is actually fairly difficult to achieve.
- masklinn 14y agoYou didn't bother reading to the end did you? "Where do we go from here" #2: > With this in mind, we are planning to invest in a duplicate of our network stack from our routers all the way through our access layer switches to be used in a staging environment. This will allow us to more fully test these kinds of changes in the future, and hopefully detect bugs like the one that caused the problems on Friday.