3 ms·
When you start dealing with hundreds of servers or more (perhaps it starts earlier at the high tens), you start looking at all things as trade-offs, and doing s
by kbenson 5y ago
When you start dealing with hundreds of servers or more (perhaps it starts earlier at the high tens), you start looking at all things as trade-offs, and doing so yields interesting insights that aren't necessarily obvious when you're working at smaller levels.
What is the cost (in time and effort and manpower and complexity) to implement? What is the cost to maintain? What is the cost to manage, when you are adding and removing people often? What are the failure scenarios, when any one server that needs to manage things starts to become a liability for disaster recovery and redundancy purposes?
Sometimes the destination is clearly better than your current place, but the road to get there has a cost all its own that makes traveling it the non-optimal choice.
It's very easy for 1-3 admins to decide to implement something over 10-30 servers and keep themselves up to date and with the right access and knowledge to manage and maintain it. It's quite another thing when you're talking about hundreds of servers and you've implemented clear delineations about access and you have 10-20 admins ranging from junior to expert with associated levels of access to servers and tools, and the fleet of servers has evolved over years (or multiple decades, in some cases). Applying changes over that type of system can be complex and error prone and when it affects your ability to actually access and maintain the systems in question, it can be very hard to reason about the problems until you start encountering them. Change comes with risk, and risk assessment of technology becomes a large part of the planning requirements.