4 ms·
FTA: > Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and network
by mbesto 5y ago
FTA:
> Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services. We have efforts underway to move to multiple availability zones within these data centers; we have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts.
If they were in AWS they could have used Consul across multi-AZs and done changes in a roll out fashion.