Perhaps I can answer some of these questions. We've dealt with scenarios where we've had to fail-over entire data-centers to other regions, or other data-center

Perhaps I can answer some of these questions. We've dealt with scenarios where we've had to fail-over entire data-centers to other regions, or other data-centers within that region. We have a few mitigations in-place for stuff like this. a) A voice server can sit in multiple different load categories. So it's not "best server by score", but rather "best server out of a pool of servers with a given load factor". The load factor is one of ":verylow | :low | :medium | :high | :veryhigh | :extremelyhigh | :full" When looking for a server, we have an index of "best servers by region" that's stored in memory on each node and kept synchronized by service discovery. Additionally, if we don't have enough candidate nodes in a given load category, we will grab a few from the next-best load category. The thought being, that for a given region, we'll have a large set of servers to allocate to. This prevents a server failing from thundering-herding another server. b) A voice server fast-fail (reject) allocation requests, and does so under some circumstances, e.g.: the rate of allocation requests for the server exceeds a threshold, the server is at capacity, or approaching capacity. We do a lot of this fast-failing logic using semaphores around a shared resource (server alloactor): https://github.com/discordapp/semaphore https://github.com/discordapp/semaphore c) We also run things a bit over-provisioned. We try to have enough excess capacity during peak such that we can handle the failure of an entire datacenter within a region, or an entire region to nearby geographical regions. >I imagine that estimating capacity is even more difficult since people can join and leave at any time, and the client doesn't send any packets when there is silence. So the load changes based on how talkative people are being (which means your server always crashes during the best parts of whatever you're discussing). We use a lot of factors to measure load on a server to group it into a load category - in addition to just traffic: we look at concurrent clients connected, concurrent voice servers allocated, packets/sec, bytes/sec.