nixgeek

63 ms·

661.
by nixgeek 14y ago
Recently.
662.
by nixgeek 14y ago
Agreed. I completely agree, but the world isn't as kind. Providers often have funny rules about how you can cable things up in their datacenters, and as noted in #4 of where GitHub goes from here that needs to be addressed.
663.
by nixgeek 14y ago
Holidays are actually one of the best times to be making changes as traffic is significantly lower, and IMO, one should be aiming for an infrastructure where you can always ship changes without being afraid of the ramifications. Architectur
664.
by nixgeek 14y ago
Better to take a few days to gather evidence and really understand the problem, than rush out a post-mortem which is inaccurate or incorrect leaving the impression that downtime is not taken seriously and investigated thoroughly. IMO.
665.
by nixgeek 14y ago
Since staging environments are often scaled down but functionally equivalent, it does not track that having one involves spending twice as much. It is also nothing to do with the switch learning 'fast enough' and everything to do with the s
666.
by nixgeek 14y ago
I believe that's a very rational conclusion and is likely correct.
667.
by nixgeek 14y ago
What odds are you giving on that? https://twitter.com/markimbriaco/status/276438853257162752
668.
by nixgeek 14y ago
But you also work for Facebook, and consequently have slightly larger scale problems than most get to worry about. True multi-datacenter availability in an active-active manner is actually fairly difficult to achieve.
669.
by nixgeek 14y ago
Would you be able to describe how a TopN report helps here?
670.
by nixgeek 14y ago
Very little of this outage can be attributed to STP issues, and most of the outage seems to be down to a software fault with the switch itself not learning MAC addresses correctly. I'm not sure how having one big chassis switch helps here,
671.
by nixgeek 14y ago
I'm not sure I'd call GitHub a #NoOps shop, nor am I sure that I'd call the team inexperienced. What was your point again?