3 ms·
Adapting a legacy network to deal with growing pains is hard. Anyone who says different is probably lying, or hasn't administered a network outside of 'bedroom
by nixgeek 14y ago
Adapting a legacy network to deal with growing pains is hard. Anyone who says different is probably lying, or hasn't administered a network outside of 'bedroom scale' previously.
Whilst everyone prefers to work with a blank canvas that is a rare opportunity, and one which still has constraints like budget limitations or "We need this by Wednesday!".
All vendors have their foibles. No vendor is perfect. Datasheets are rarely 100% accurate, behaviour fluctuates based on what code train/version you're running on devices. Network vendors are like everyone else - humans - so just like everyone else they break things occasionally too.
One of the best ways to mitigate this as demonstrated by some of the largest players is to have N+1 physical sites and to be able to control traffic distribution to each of them, including taking one entirely offline without your customers noticing that it's happened. It's also extremely hard to do when you start looking at how CAP theorem applies to your given application, and not just the infrastructure changes required to network it all together, but the application changes required to not have it just explode on you.
- 23david 14y agoAgreed that technical debt is a pain to deal with. But it's definitely possible to migrate and upgrade live systems without significant technical risk. The company just needs to make the decision to invest the proper resources and decide to have executive-level focus on Operations. From your comments it sounds like this was a failure of management to properly plan out these changes. Lack of resources shouldn't be an issue here, since clearly github's $100 Million+ in funding is enough to build a reliable HA system with adequate redundancies and hire the people who are qualified to run it. There just isn't anyone to point fingers at, including vendor screw-ups. All of those items would be addressed in a proper risk assessment plan.