5 ms·

> Surprised it was a single availability zone, without redundancy. Having multiple fully independent zones seems more reliable and failsafe. It's also a lot mo

> Surprised it was a single availability zone, without redundancy. Having multiple fully independent zones seems more reliable and failsafe. It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage

outworlder 5y ago
> It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage Not sure I agree. Yes, network costs are higher, but your overall costs may not be depending on how you architect. Independent services across AZs? Sure. You'll have multiples of your current costs. Deploying your clusters spanning AZs? Not that much - you'll pay for AZ traffic though.
- adrr 5y ago
  It is when you run your own date centers and have to shell out a large capital outlays to spin up a new datacenter.
  - Symbiote 5y ago
    The usual way this works (and I assume this is the case for Roblox) is not by constructing buildings, but by renting space in someone else's datacentre. Pretty much every city worldwide has at least one place providing power, cooling, racks and (optionally) network. You rent space for one or more servers, or you rent racks, or parts of a floor, or whole floors. You buy your own servers, and either install them yourself, or pay the datacentre staff to install them.
Hamuko 5y ago
How expensive? Remember that the Roblox Corporation does about a billion dollars in revenue per year and takes about 50% of all revenue developers generate on their platform.
- dev_by_day 5y ago
  Right, outages get more expensive the larger you grow. What else needs to be thought of is not just the loss of revenue for the time your service is down but also it's affect on user trust and usability. Customers will gladly leave you for a more reliable competitor once they get fed up.
johnmarcus 5y ago
Multi-AZ is free at Amazon. Having things split amongst 3 AZ's cost no more than having in a single AZ. Multi-Region is a different story.
- suifbwish 5y ago
  Having bare metal may not be less stress but AWS is by no means cheap.
  - johnmarcus 5y ago
    AWS is not cheap, but splitting amongst AZ's is of no additional cost.
    - orangepurple 5y ago
      False Data Transfer within the same AWS Region Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction. https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_within_the_same_AWS_Region https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_...
    - StreamBright 5y ago
      > AWS is not cheap Wrong. Depends on the use case AWS can be very cheap. > splitting amongst AZ's is of no additional cost. Wrong. " across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction. Effectively, cross-AZ data transfer in AWS costs 2¢ per gigabyte and each gigabyte transferred counts as 2GB on the bill: once for sending and once for receiving."
- otterley 5y ago
  There are definitely cost and other considerations you have to think about when going multi-AZ. Cross-AZ network traffic has charges associated with it. Inter-AZ network latency is higher than intra-AZ latency. And there are other limitations as well, such as EBS volumes being attachable only to an instance in the same AZ as the volume. That said, AWS does recommend using multiple Availability Zones to improve overall availability and reduce Mean Time to Recovery (MTTR). (I work for AWS. Opinions are my own and not necessarily those of my employer.)
sam0x17 5y ago
Most startups I've worked at literally have a script to deploy their whole setup to a new region when desired. Then you just need latency-based routing running on top of it to ensure people are processed in the closest region to them. Really not expensive. You can do this with under $200/month in terms of complexity and the bandwidth + database costs are going to be roughly the same as they normally are because you're splitting your load between regions. Now if you stupidly just duplicate your current infrastructure entirely, yes it would be expensive because you'd be massively overpaying on DB. In theory the only additional cost should be the latency-based routing itself, which is $50/month. Other than that, you'll probably save money if you choose the right regions.
- e4e78a06 5y ago
  Correctly handling failure edge cases in a active-active multi-region distributed database requires work. SaaS DBs do a lot of the heavy lifting but they are still highly configurable and you need to understand the impact of the config you use. Not to mention your scale-up runbooks need to be established so a stampede from a failure in one region doesn't cause the other region to go down. You also need to avoid cross-region traffic even though you might have stateful services that aren't replicated across regions. That might mean changes in config or business logic across all your services. It is absolutely not as simple as spinning up a cluster on AWS at Roblox's scale.
- bradly 5y ago
  Are the same services available in all regions? Are the same instance sizes available in all regions? Are there enough instances of the sizes you need? Do you have reserved instances in the other region? Are your increased quotas applied to all regions? What region are your S3 assets in? Are you going to migrate those as well? Is it acceptable for all user sessions to be terminated? Have you load tested the other region? How often are you going to test the region fail over? Yearly? Quarterly? With every code change? What is the acceptable RTO and RPO with executives and board-members? And all of that is without thinking about cache warming, database migration/mirror/replication, solr indexing (are you going to migrate the index or rebuild? Do you know how long it takes to rebuild your solr index?). The startups you worked at probably had different needs the Roblox. I was the tech leach on a Rails app that was embedded in TurboTax and QuickBooks and was rendered on each TT screen transition and reading your comment in that context shows a lot of inexperience in large, production systems.
bradly 5y ago
Yes. If you are running in two zones in the hopes that you will be up if one goes down, you need to be handling less than 50% load in each zone. If you can scale up fast enough for your use case, great. But when a zone goes down and everyone is trying to launch in the zone still up, there may not be instances for you available at that time. Our site had a billion in revenue or something based on a single day, so for us it was worth the cost, but it not easy (or at least it wasn't at the time).

outworlder 5y ago

> It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage Not sure I agree. Yes, network costs are higher, but your overall costs may not be depending on how you architect. Independent services across AZs? Sure. You'll have multiples of your current costs. Deploying your clusters spanning AZs? Not that much - you'll pay for AZ traffic though.

adrr 5y ago
It is when you run your own date centers and have to shell out a large capital outlays to spin up a new datacenter.
- Symbiote 5y ago
  The usual way this works (and I assume this is the case for Roblox) is not by constructing buildings, but by renting space in someone else's datacentre. Pretty much every city worldwide has at least one place providing power, cooling, racks and (optionally) network. You rent space for one or more servers, or you rent racks, or parts of a floor, or whole floors. You buy your own servers, and either install them yourself, or pay the datacentre staff to install them.

Hamuko 5y ago

How expensive? Remember that the Roblox Corporation does about a billion dollars in revenue per year and takes about 50% of all revenue developers generate on their platform.

dev_by_day 5y ago
Right, outages get more expensive the larger you grow. What else needs to be thought of is not just the loss of revenue for the time your service is down but also it's affect on user trust and usability. Customers will gladly leave you for a more reliable competitor once they get fed up.

johnmarcus 5y ago

Multi-AZ is free at Amazon. Having things split amongst 3 AZ's cost no more than having in a single AZ. Multi-Region is a different story.

suifbwish 5y ago
Having bare metal may not be less stress but AWS is by no means cheap.
- johnmarcus 5y ago
  AWS is not cheap, but splitting amongst AZ's is of no additional cost.
  - orangepurple 5y ago
    False Data Transfer within the same AWS Region Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction. https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_within_the_same_AWS_Region https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_...
  - StreamBright 5y ago
    > AWS is not cheap Wrong. Depends on the use case AWS can be very cheap. > splitting amongst AZ's is of no additional cost. Wrong. " across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction. Effectively, cross-AZ data transfer in AWS costs 2¢ per gigabyte and each gigabyte transferred counts as 2GB on the bill: once for sending and once for receiving."
otterley 5y ago
There are definitely cost and other considerations you have to think about when going multi-AZ. Cross-AZ network traffic has charges associated with it. Inter-AZ network latency is higher than intra-AZ latency. And there are other limitations as well, such as EBS volumes being attachable only to an instance in the same AZ as the volume. That said, AWS does recommend using multiple Availability Zones to improve overall availability and reduce Mean Time to Recovery (MTTR). (I work for AWS. Opinions are my own and not necessarily those of my employer.)

sam0x17 5y ago

Most startups I've worked at literally have a script to deploy their whole setup to a new region when desired. Then you just need latency-based routing running on top of it to ensure people are processed in the closest region to them. Really not expensive. You can do this with under $200/month in terms of complexity and the bandwidth + database costs are going to be roughly the same as they normally are because you're splitting your load between regions. Now if you stupidly just duplicate your current infrastructure entirely, yes it would be expensive because you'd be massively overpaying on DB. In theory the only additional cost should be the latency-based routing itself, which is $50/month. Other than that, you'll probably save money if you choose the right regions.

e4e78a06 5y ago
Correctly handling failure edge cases in a active-active multi-region distributed database requires work. SaaS DBs do a lot of the heavy lifting but they are still highly configurable and you need to understand the impact of the config you use. Not to mention your scale-up runbooks need to be established so a stampede from a failure in one region doesn't cause the other region to go down. You also need to avoid cross-region traffic even though you might have stateful services that aren't replicated across regions. That might mean changes in config or business logic across all your services. It is absolutely not as simple as spinning up a cluster on AWS at Roblox's scale.
bradly 5y ago
Are the same services available in all regions? Are the same instance sizes available in all regions? Are there enough instances of the sizes you need? Do you have reserved instances in the other region? Are your increased quotas applied to all regions? What region are your S3 assets in? Are you going to migrate those as well? Is it acceptable for all user sessions to be terminated? Have you load tested the other region? How often are you going to test the region fail over? Yearly? Quarterly? With every code change? What is the acceptable RTO and RPO with executives and board-members? And all of that is without thinking about cache warming, database migration/mirror/replication, solr indexing (are you going to migrate the index or rebuild? Do you know how long it takes to rebuild your solr index?). The startups you worked at probably had different needs the Roblox. I was the tech leach on a Rails app that was embedded in TurboTax and QuickBooks and was rendered on each TT screen transition and reading your comment in that context shows a lot of inexperience in large, production systems.

bradly 5y ago

Yes. If you are running in two zones in the hopes that you will be up if one goes down, you need to be handling less than 50% load in each zone. If you can scale up fast enough for your use case, great. But when a zone goes down and everyone is trying to launch in the zone still up, there may not be instances for you available at that time. Our site had a billion in revenue or something based on a single day, so for us it was worth the cost, but it not easy (or at least it wasn't at the time).