9 ms·

>> We are working to move to multiple availability zones and data centers. Surprised it was a single availability zone, without redundancy. Having multiple ful

>> We are working to move to multiple availability zones and data centers. Surprised it was a single availability zone, without redundancy. Having multiple fully independent zones seems more reliable and failsafe.

kreeben 5y ago
>> Having multiple fully independent zones seems more reliable I don't think these independent zones exist. See AWS's recent outages, where east cripples west and vice versa.
- count 5y ago
  That's not how they work. They exist, and work extremely well within their defined engineering / design goals. It's much more nuanced than 'everything works independently'.
  - kreeben 5y ago
    If the design goal of these zones is that they should be independent of each other then, no, they do not work extremely well.
- Karrot_Kream 5y ago
  Availability Zones aren't the same thing as regions. AWS regions have multiple Availability Zones. Independent availability zones publishes lower reliability SLAs so you need to load balance across multiple independent availability zones in a region to reach higher reliability. Per AZ SLAs are discussed in more detail here [1] (N.B. I find HN commentary on AWS outages pretty depressing because it becomes pretty obvious that folks don't understand cloud networking concepts at all.) [1]: https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
  - kreeben 5y ago
    >> you need to load balance across multiple independent availability zones The only problem with that is, there are no independent availability zones. What we do have, though, is an architecture where errors propagate cross-zone until they can't propagate any further, because services can't take any more requests, because they froze, because they weren't designed for a split brain scenario, and then, half the internet goes down.
    - outworlder 5y ago
      > The only problem with that is, there are no independent availability zones. There are - they can be as independent as you need them to be. Errors won't necessarily propagate cross-zone. If they do, someone either screwed up, or they made a trade-off. Screwing up is easy, so you need to do chaos testing to make sure your system will survive as intended.
      - kreeben 5y ago
        I'm not talking about my global app. I'm talking about the system I deploy to, the actual plumbing, and how a huge turd in a western toilet causes east's sewerage system to over-flow.
  - mlyle 5y ago
    > (N.B. I find HN commentary on AWS outages pretty depressing because it becomes pretty obvious that folks don't understand cloud networking concepts at all.) What he said was perfectly cogent. Outages in us-east-1 AZ us-east-1a have caused outages in us-west-1a, which is a different region and a different AZ. Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that can cause outages to propagate. So, if you span multiple availability zones, you are not spared from events that will impact all of them.
    - Karrot_Kream 5y ago
      > Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that can cause outages to propagate. It's up to the _user_ of AWS to design around this level of reliability. This isn't any different than not using AWS. I can run my web business on the super cheap by running it out of my house. Of course, then my site's availability is based around the uptime of my residential internet connection, my residential power, my own ability to keep my server plugged into power, and general reliability of my server's components. I can try to make things more reliable by putting it into a DC, but if a backhoe takes out the fiber to that DC, then the DC will become unavailable. It's up to the _user_ to architect their services to be reliable. AWS isn't magic reliability sauce you sprinkle on your web apps to make them stay up for longer. AWS clearly states in their SLA pages what their EC2 instance SLAs are in a given AZ; it's 99.5% availability for a given EC2 instance in a given region and AZ. This is roughly ~1.82 days, or ~ 43.8 hours, of downtime in a year. If you add a SPOF around a single EC2 instance in a given AZ then your system has a 99.5% availability SLA. Remember the cloud is all about leveraging large amounts commodity hardware instead of leveraging large, high-reliability mainframe style design. This isn't a secret. It's openly called out, like in Nishtala et al's "Scaling Memcache at Facebook" [1] from 2013! The background of all of this is that it costs money, in terms of knowledgable engineers (not like the kinds in this comment thread who are conflating availability zones and regions) who understand these issues. Most companies don't care; they're okay with being down for a couple days a year. But if you want to design high reliability architectures, there are plenty of senior engineers willing to help, _if_ you're willing to pay their salaries. If you want to come up with a lower cognitive overhead cloud solution for high reliability services that's economical for companies, be my guest. I think we'd all welcome innovation in this space. [1]: https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf https://www.usenix.org/system/files/conference/nsdi13/nsdi13...
  - johnmarcus 5y ago
    Yup, so true. People think redundant == 100% uptime, or that when they advertise 99.9% uptime, it's the same thing as 100% minus a tiny bit for "glitches". It's not. .1% of 36524 = 87.6 hours of downtime - that's over 3 days of complete downtime every year! For a more complete list of their SLA's for every service: https://aws.amazon.com/legal/service-level-agreements/?aws-sla-cards.sort-by=item.additionalFields.serviceNameLower&aws-sla-cards.sort-order=asc&awsf.tech-category-filter=*all https://aws.amazon.com/legal/service-level-agreements/?aws-s... They only refund 100% when they fall below 95% of availability! 95-99= 30%. I believe the real target is above 99.9% though, as that results in 0 refund to the customer. What that means is, 3 days of downtime is acceptable! Alternatively, you can return to your own datacenter and find out first hand that it's not particularly as easy to deliver that as you may think. You too will have power outages, network provider disruptions, and the occasional "oh shit, did someone just kick that power cord out?" or complete disk array meltdowns. Anywho, they have a lot more room in their published SLA's than you think. Edit: as someone correctly pointed out i did a typo in my math. it is only ~9 hours of aloted downtime. Keeping in mind that this is per service though - meaning each service can have a different 9 hours of downtime before they need to pay out 10% of that one service. I still stand by my statement thier SLA's have a lot of wiggle room that people should take more seriously.
    - mqnfred 5y ago
      Your computation is incorrect, 3 days out of 365 is 1% of downtime, not 0.1%. I believe your error stems from reporting .1% as 0.1. Indeed: 0.001 (.1%) * 8760 (365d*24h) = 8.76h Alternatively, the common industry standard in infrastructure (the place I work at at least,) is 4 nines, so 99.99% availability, which is around 52 mins a year or 4 mins a month iirc. There's not as much room as you'd think! :)
    - sciurus 5y ago
      As someone else said, your math is off. Your point is still reasonable, though. The uptime.is website is a handy resource for these calculations. For example, http://uptime.is/99.9 http://uptime.is/99.9 says "SLA level of 99.9 % uptime/availability results in the following periods of allowed downtime/unavailability: Daily: 1m 26s Weekly: 10m 4s Monthly: 43m 49s Quarterly: 2h 11m 29s Yearly: 8h 45m 56s"
- Bluecobra 5y ago
  > I don't think these independent zones exist. Wouldn't it be possible to create fully independent zones with multiple cloud providers, like AWS, GCP, Azure? This is assuming that your workloads don't rely on proprietary services from a given provider.
  - dpifke 5y ago
    Yes, and would also protect you from administrative outages like, "AWS shut off our account because we missed the email about our credit card expiring." (But wouldn't protect you from software/configuration issues if you're running the same stack in every zone.)
vorpalhex 5y ago
I'm more impressed that it hasn't been an issue until now.
bob1029 5y ago
> Having multiple fully independent zones seems more reliable and failsafe. This also introduces new modes of failure which did not exist before. There are no silver bullets for this problem.
- rhizome 5y ago
  There are no silver bullets to any problem, but there are other ways of implementing services and architecture that can sidestep these things.
hedwall 5y ago
A guess would be that game servers are distributed across the globe but backend services l are in one place. A common pattern in game companies.
foobarian 5y ago
> Surprised it was a single availability zone, without redundancy. Having multiple fully independent zones seems more reliable and failsafe. It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage
- outworlder 5y ago
  > It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage Not sure I agree. Yes, network costs are higher, but your overall costs may not be depending on how you architect. Independent services across AZs? Sure. You'll have multiples of your current costs. Deploying your clusters spanning AZs? Not that much - you'll pay for AZ traffic though.
  - adrr 5y ago
    It is when you run your own date centers and have to shell out a large capital outlays to spin up a new datacenter.
    - Symbiote 5y ago
      The usual way this works (and I assume this is the case for Roblox) is not by constructing buildings, but by renting space in someone else's datacentre. Pretty much every city worldwide has at least one place providing power, cooling, racks and (optionally) network. You rent space for one or more servers, or you rent racks, or parts of a floor, or whole floors. You buy your own servers, and either install them yourself, or pay the datacentre staff to install them.
- Hamuko 5y ago
  How expensive? Remember that the Roblox Corporation does about a billion dollars in revenue per year and takes about 50% of all revenue developers generate on their platform.
  - dev_by_day 5y ago
abarringer 5y ago
Was on a call with a bank VP that had moved to AWS. Asked how it was going. Said it was going great after six months but just learning about availability zones so they were going to have to rework a bunch of things. Astonishing how our important infrastructure is moved to AWS with zero knowledge of how AWS works.
maxclark 5y ago
No surprised at all. Multi AZ is a PITA. You'd be surprised how many 7fig+/month infra is single region/az
- mhitza 5y ago
  For example parts of AWS itself. us-east-1 having issues? Looks like aws console all over the world have issues. You constantly hear about multi zone, region, cloud. But in practice when things break you hear all these stories of them running in a single region+zone
mbesto 5y ago
There have been multiple discussions on HN about cloud vs not cloud and there are endless amount of opinions of "cloud is a waste blah blah". This is exactly one of the reasons people go cloud. Introducing an additional AZ is a click of a button and some relatively trivial infrastructure as code scripting, even at this scale. Running your own data center and AZ on the other hand requires a very tight relationship with your data center provider at global scale. For a platform like Roblox where downtime equals money loss (i.e. every hour of the day people make purchases), then there is a real tangible benefit to using something like AWS. 72 hours downtime is A LOT, and we're talking potentially millions of dollars of real value lost and millions of potential in brand value lost. I'm not saying definitively they would save money (in this case profit impact) by going to AWS, but there is definitely a story to be had here.
- treis 5y ago
  But it wasn't a hardware issue. It was a software one and that would have crossed AZ boundaries.
  - mbesto 5y ago
    So then why does the post mortem suggest setting up multi-az to address the problems they encountered?
    - treis 5y ago
      I took that to mean sharding Roblox instead of spanning it across data center AZs.
      - mbesto 5y ago
        FTA: > Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services. We have efforts underway to move to multiple availability zones within these data centers; we have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts. If they were in AWS they could have used Consul across multi-AZs and done changes in a roll out fashion.

kreeben 5y ago

>> Having multiple fully independent zones seems more reliable I don't think these independent zones exist. See AWS's recent outages, where east cripples west and vice versa.

count 5y ago
That's not how they work. They exist, and work extremely well within their defined engineering / design goals. It's much more nuanced than 'everything works independently'.
- kreeben 5y ago
  If the design goal of these zones is that they should be independent of each other then, no, they do not work extremely well.
Karrot_Kream 5y ago
Availability Zones aren't the same thing as regions. AWS regions have multiple Availability Zones. Independent availability zones publishes lower reliability SLAs so you need to load balance across multiple independent availability zones in a region to reach higher reliability. Per AZ SLAs are discussed in more detail here [1] (N.B. I find HN commentary on AWS outages pretty depressing because it becomes pretty obvious that folks don't understand cloud networking concepts at all.) [1]: https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
- kreeben 5y ago
  >> you need to load balance across multiple independent availability zones The only problem with that is, there are no independent availability zones. What we do have, though, is an architecture where errors propagate cross-zone until they can't propagate any further, because services can't take any more requests, because they froze, because they weren't designed for a split brain scenario, and then, half the internet goes down.
  - outworlder 5y ago
    > The only problem with that is, there are no independent availability zones. There are - they can be as independent as you need them to be. Errors won't necessarily propagate cross-zone. If they do, someone either screwed up, or they made a trade-off. Screwing up is easy, so you need to do chaos testing to make sure your system will survive as intended.
    - kreeben 5y ago
      I'm not talking about my global app. I'm talking about the system I deploy to, the actual plumbing, and how a huge turd in a western toilet causes east's sewerage system to over-flow.
- mlyle 5y ago
  > (N.B. I find HN commentary on AWS outages pretty depressing because it becomes pretty obvious that folks don't understand cloud networking concepts at all.) What he said was perfectly cogent. Outages in us-east-1 AZ us-east-1a have caused outages in us-west-1a, which is a different region and a different AZ. Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that can cause outages to propagate. So, if you span multiple availability zones, you are not spared from events that will impact all of them.
  - Karrot_Kream 5y ago
    > Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that can cause outages to propagate. It's up to the _user_ of AWS to design around this level of reliability. This isn't any different than not using AWS. I can run my web business on the super cheap by running it out of my house. Of course, then my site's availability is based around the uptime of my residential internet connection, my residential power, my own ability to keep my server plugged into power, and general reliability of my server's components. I can try to make things more reliable by putting it into a DC, but if a backhoe takes out the fiber to that DC, then the DC will become unavailable. It's up to the _user_ to architect their services to be reliable. AWS isn't magic reliability sauce you sprinkle on your web apps to make them stay up for longer. AWS clearly states in their SLA pages what their EC2 instance SLAs are in a given AZ; it's 99.5% availability for a given EC2 instance in a given region and AZ. This is roughly ~1.82 days, or ~ 43.8 hours, of downtime in a year. If you add a SPOF around a single EC2 instance in a given AZ then your system has a 99.5% availability SLA. Remember the cloud is all about leveraging large amounts commodity hardware instead of leveraging large, high-reliability mainframe style design. This isn't a secret. It's openly called out, like in Nishtala et al's "Scaling Memcache at Facebook" [1] from 2013! The background of all of this is that it costs money, in terms of knowledgable engineers (not like the kinds in this comment thread who are conflating availability zones and regions) who understand these issues. Most companies don't care; they're okay with being down for a couple days a year. But if you want to design high reliability architectures, there are plenty of senior engineers willing to help, _if_ you're willing to pay their salaries. If you want to come up with a lower cognitive overhead cloud solution for high reliability services that's economical for companies, be my guest. I think we'd all welcome innovation in this space. [1]: https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf https://www.usenix.org/system/files/conference/nsdi13/nsdi13...
- johnmarcus 5y ago
  Yup, so true. People think redundant == 100% uptime, or that when they advertise 99.9% uptime, it's the same thing as 100% minus a tiny bit for "glitches". It's not. .1% of 36524 = 87.6 hours of downtime - that's over 3 days of complete downtime every year! For a more complete list of their SLA's for every service: https://aws.amazon.com/legal/service-level-agreements/?aws-sla-cards.sort-by=item.additionalFields.serviceNameLower&aws-sla-cards.sort-order=asc&awsf.tech-category-filter=*all https://aws.amazon.com/legal/service-level-agreements/?aws-s... They only refund 100% when they fall below 95% of availability! 95-99= 30%. I believe the real target is above 99.9% though, as that results in 0 refund to the customer. What that means is, 3 days of downtime is acceptable! Alternatively, you can return to your own datacenter and find out first hand that it's not particularly as easy to deliver that as you may think. You too will have power outages, network provider disruptions, and the occasional "oh shit, did someone just kick that power cord out?" or complete disk array meltdowns. Anywho, they have a lot more room in their published SLA's than you think. Edit: as someone correctly pointed out i did a typo in my math. it is only ~9 hours of aloted downtime. Keeping in mind that this is per service though - meaning each service can have a different 9 hours of downtime before they need to pay out 10% of that one service. I still stand by my statement thier SLA's have a lot of wiggle room that people should take more seriously.
  - mqnfred 5y ago
    Your computation is incorrect, 3 days out of 365 is 1% of downtime, not 0.1%. I believe your error stems from reporting .1% as 0.1. Indeed: 0.001 (.1%) * 8760 (365d*24h) = 8.76h Alternatively, the common industry standard in infrastructure (the place I work at at least,) is 4 nines, so 99.99% availability, which is around 52 mins a year or 4 mins a month iirc. There's not as much room as you'd think! :)
  - sciurus 5y ago
    As someone else said, your math is off. Your point is still reasonable, though. The uptime.is website is a handy resource for these calculations. For example, http://uptime.is/99.9 http://uptime.is/99.9 says "SLA level of 99.9 % uptime/availability results in the following periods of allowed downtime/unavailability: Daily: 1m 26s Weekly: 10m 4s Monthly: 43m 49s Quarterly: 2h 11m 29s Yearly: 8h 45m 56s"
Bluecobra 5y ago
> I don't think these independent zones exist. Wouldn't it be possible to create fully independent zones with multiple cloud providers, like AWS, GCP, Azure? This is assuming that your workloads don't rely on proprietary services from a given provider.
- dpifke 5y ago
  Yes, and would also protect you from administrative outages like, "AWS shut off our account because we missed the email about our credit card expiring." (But wouldn't protect you from software/configuration issues if you're running the same stack in every zone.)

vorpalhex 5y ago

I'm more impressed that it hasn't been an issue until now.

bob1029 5y ago

> Having multiple fully independent zones seems more reliable and failsafe. This also introduces new modes of failure which did not exist before. There are no silver bullets for this problem.

rhizome 5y ago
There are no silver bullets to any problem, but there are other ways of implementing services and architecture that can sidestep these things.

hedwall 5y ago

A guess would be that game servers are distributed across the globe but backend services l are in one place. A common pattern in game companies.

foobarian 5y ago

> Surprised it was a single availability zone, without redundancy. Having multiple fully independent zones seems more reliable and failsafe. It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage

outworlder 5y ago
> It's also a lot more expensive. Probably order of magnitude more expensive than the cost of a 1 day outage Not sure I agree. Yes, network costs are higher, but your overall costs may not be depending on how you architect. Independent services across AZs? Sure. You'll have multiples of your current costs. Deploying your clusters spanning AZs? Not that much - you'll pay for AZ traffic though.
- adrr 5y ago
  It is when you run your own date centers and have to shell out a large capital outlays to spin up a new datacenter.
  - Symbiote 5y ago
    The usual way this works (and I assume this is the case for Roblox) is not by constructing buildings, but by renting space in someone else's datacentre. Pretty much every city worldwide has at least one place providing power, cooling, racks and (optionally) network. You rent space for one or more servers, or you rent racks, or parts of a floor, or whole floors. You buy your own servers, and either install them yourself, or pay the datacentre staff to install them.
Hamuko 5y ago
How expensive? Remember that the Roblox Corporation does about a billion dollars in revenue per year and takes about 50% of all revenue developers generate on their platform.
- dev_by_day 5y ago

abarringer 5y ago

Was on a call with a bank VP that had moved to AWS. Asked how it was going. Said it was going great after six months but just learning about availability zones so they were going to have to rework a bunch of things. Astonishing how our important infrastructure is moved to AWS with zero knowledge of how AWS works.

maxclark 5y ago

No surprised at all. Multi AZ is a PITA. You'd be surprised how many 7fig+/month infra is single region/az

mhitza 5y ago
For example parts of AWS itself. us-east-1 having issues? Looks like aws console all over the world have issues. You constantly hear about multi zone, region, cloud. But in practice when things break you hear all these stories of them running in a single region+zone

mbesto 5y ago

There have been multiple discussions on HN about cloud vs not cloud and there are endless amount of opinions of "cloud is a waste blah blah". This is exactly one of the reasons people go cloud. Introducing an additional AZ is a click of a button and some relatively trivial infrastructure as code scripting, even at this scale. Running your own data center and AZ on the other hand requires a very tight relationship with your data center provider at global scale. For a platform like Roblox where downtime equals money loss (i.e. every hour of the day people make purchases), then there is a real tangible benefit to using something like AWS. 72 hours downtime is A LOT, and we're talking potentially millions of dollars of real value lost and millions of potential in brand value lost. I'm not saying definitively they would save money (in this case profit impact) by going to AWS, but there is definitely a story to be had here.

treis 5y ago
But it wasn't a hardware issue. It was a software one and that would have crossed AZ boundaries.
- mbesto 5y ago
  So then why does the post mortem suggest setting up multi-az to address the problems they encountered?
  - treis 5y ago
    I took that to mean sharding Roblox instead of spanning it across data center AZs.
    - mbesto 5y ago
      FTA: > Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services. We have efforts underway to move to multiple availability zones within these data centers; we have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts. If they were in AWS they could have used Consul across multi-AZs and done changes in a roll out fashion.