4 ms·
There have been multiple discussions on HN about cloud vs not cloud and there are endless amount of opinions of "cloud is a waste blah blah". This is exactly o
by mbesto 5y ago
There have been multiple discussions on HN about cloud vs not cloud and there are endless amount of opinions of "cloud is a waste blah blah".
This is exactly one of the reasons people go cloud. Introducing an additional AZ is a click of a button and some relatively trivial infrastructure as code scripting, even at this scale.
Running your own data center and AZ on the other hand requires a very tight relationship with your data center provider at global scale.
For a platform like Roblox where downtime equals money loss (i.e. every hour of the day people make purchases), then there is a real tangible benefit to using something like AWS. 72 hours downtime is A LOT, and we're talking potentially millions of dollars of real value lost and millions of potential in brand value lost. I'm not saying definitively they would save money (in this case profit impact) by going to AWS, but there is definitely a story to be had here.
- treis 5y agoBut it wasn't a hardware issue. It was a software one and that would have crossed AZ boundaries.
- mbesto 5y agoSo then why does the post mortem suggest setting up multi-az to address the problems they encountered?
- treis 5y agoI took that to mean sharding Roblox instead of spanning it across data center AZs.
- mbesto 5y agoFTA: > Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services. We have efforts underway to move to multiple availability zones within these data centers; we have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts. If they were in AWS they could have used Consul across multi-AZs and done changes in a roll out fashion.
- smarx007 5y agoSo that next time they can spend 96 hours on recovery, this time adding a split brain issue to the list of problems to deal with. Jokes aside, the write-up is quite good after after thinking about all the problems they had to deal with, I was quite humbled.
- aahortwwy 5y agoIt doesn't really explain how they reached the conclusion that that would help. Like, yes, it's a problem that they had a giant Consul cluster that was a SPOF, but you can run multiple smaller Consul clusters within a single AZ if you want. Honestly it reads to me like an internal movement for a multi-AZ deployment successfully used this crisis as an opportunity.