3 ms·

> Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that

> Or, to put it in the terms of reliability engineering: even though these are abstracted as independent systems, in reality there are common-mode failures that can cause outages to propagate. It's up to the _user_ of AWS to design around this level of reliability. This isn't any different than not using AWS. I can run my web business on the super cheap by running it out of my house. Of course, then my site's availability is based around the uptime of my residential internet connection, my residential power, my own ability to keep my server plugged into power, and general reliability of my server's components. I can try to make things more reliable by putting it into a DC, but if a backhoe takes out the fiber to that DC, then the DC will become unavailable. It's up to the _user_ to architect their services to be reliable. AWS isn't magic reliability sauce you sprinkle on your web apps to make them stay up for longer. AWS clearly states in their SLA pages what their EC2 instance SLAs are in a given AZ; it's 99.5% availability for a given EC2 instance in a given region and AZ. This is roughly ~1.82 days, or ~ 43.8 hours, of downtime in a year. If you add a SPOF around a single EC2 instance in a given AZ then your system has a 99.5% availability SLA. Remember the cloud is all about leveraging large amounts commodity hardware instead of leveraging large, high-reliability mainframe style design. This isn't a secret. It's openly called out, like in Nishtala et al's "Scaling Memcache at Facebook" [1] from 2013! The background of all of this is that it costs money, in terms of knowledgable engineers (not like the kinds in this comment thread who are conflating availability zones and regions) who understand these issues. Most companies don't care; they're okay with being down for a couple days a year. But if you want to design high reliability architectures, there are plenty of senior engineers willing to help, _if_ you're willing to pay their salaries. If you want to come up with a lower cognitive overhead cloud solution for high reliability services that's economical for companies, be my guest. I think we'd all welcome innovation in this space. [1]: https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf https://www.usenix.org/system/files/conference/nsdi13/nsdi13...

mlyle 5y ago
Yes, but the underlying point you're willfully missing is: You can't engineer around AWS AZ common-mode failures using AWS. The moment that you have failures that are not independent and common mode, you can't just multiply together failure probabilities to know your outage times.
roughly 5y ago
During a recent AWS outage, the STS service running in us-east-1 was unavailable. Unfortunately, all of the other _regions_ - not AZs, but _regions_, rely on the STS service in us-east-1, which meant that customers which had built around Amazon’s published reliability model had services in every region impacted by an outage in one specific availability zone. This is what kreeben was referring to - not some abstract misconception about the difference between AZs and Regions, but an actual real world incident in which a failure in one AZ had an impact in other Regions.
- otterley 5y ago
  It's more subtle than that. For high availability, STS offers regional endpoints -- and AWS recommends using them[1] -- but the SDKs don't use them by default. The author of the client code, or the person configuring the software, has to enable them. [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... (I work for AWS. Opinions are my own and not necessarily those of my employer.)
  - roughly 5y ago
    The client code which defaults to STS in us-east-1 includes the AWS console website, as far as I can tell. Real question, though - are those genuinely separate endpoints that remained up and operational during the outage? I don’t think I saw or knew a single person unaffected by this outage, so either there’s still some bleed over on the backend or knowledge of the regional STS endpoints is basically zero (which I Can believe, y’all run a big shop)
    - Karrot_Kream 5y ago
      My team didn't use STS but I know other teams at the company did. Those that did rely on non-us-east-1 endpoints did stay up IIRC. Our company barely use the AWS console at all and base most of our stuff around their APIs to hook into our deployment/CI processes. But I don't work at AWS so I don't know if it's true or if there was some other backend replication lag or anything else going on that was impacted by us-east-1 being down. We had some failures for some of our older services that were not properly sharded out, but most of our stuff failed over and continued to work as expected.
mlyle 5y ago
> knowledgable engineers (not like the kinds in this comment thread who are conflating availability zones and regions) I think this breaks the site guidelines. Worse, I don't think the other people are wrong: being in a different region implies being in a different availability zone. That is, I've read the comments to say "they're not only in different AZ's, they're in different regions". It seems you seem determined to pick a reading that lets you feel smugly superior about your level of knowledge, and then cast out digs at other people based on that presumed superiority.
- Karrot_Kream 5y ago
  > Worse, I don't think the other people are wrong: being in a different region implies being in a different availability zone. Availability zones do not map across regions. AZs are specific to a region. Different regions have differing numbers of AZs. us-east-1 has 3. IIRC ap-southeast-1 has 2. > That is, I've read the comments to say "they're not only in different AZ's, they're in different regions" So I've read. The earlier example about STS that someone brought up was incorrect; both I and another commenter linked to the doc with the correct information. > It seems you seem determined to pick a reading that lets you feel smugly superior about your level of knowledge, and then cast out digs at other people based on that presumed superiority. You obviously feel very strongly about this. You've replied to my parent twice now. You're right that the parenthetical was harsh but I wouldn't say it's uncalled for. Every one of these outage threads descends into a slew of easily defensible complaints about cloud providers. The quality of these discussions is terrible. I spend a lot of time at my dayjob (and as a hobby) working on networking related things. Understanding the subtle guarantees offered by AWS is a large part of my day-to-day. When I see people here make easily falsifiable comments full of hearsay ("I had a friend of a friend who works at Amazon and they did X, Y, Z bad things") and use that to drum up a frenzy, it flies in the face of what I do everyday. There's lots of issues with cloud providers as a whole and AWS in particular but to get to that level of conversation you need to understand what the system is actually doing, not just get angry and guess why it's failing.
  - mlyle 5y ago
    > > being in a different region implies being in a different availability zone. > Availability zones do not map across regions. AZs are specific to a region. Different regions have differing numbers of AZs. us-east-1 has 3. IIRC ap-southeast-1 has 2. Right.. So if you are in a different region, you are by definition in a different availability zone. > You obviously feel very strongly about this. You've replied to my parent twice now. You're right that the parenthetical was harsh but I wouldn't say it's uncalled for. Yah, I really thought about it and you're just reeking unkindness. And the people above that you're replying to and mocking are not wrong. > Every one of these outage threads descends into a slew of easily defensible complaints about cloud providers. The quality of these discussions is terrible. I spend a lot of time at my dayjob (and as a hobby) working on networking related things. Understanding the subtle guarantees offered by AWS is a large part of my day-to-day. If you're unable to be civil about this, maybe you should avoid the threads. Amazon seeks to avoid common-mode failures between AZs (and thus regions). This doesn't mean that Amazon attains this goal. And the larger point: as I'm sure you're aware, building a distributed system that attains higher uptimes by crossing multiple AZs is hard and costly and can only be justified in some cases. I've got >20 years of experience in building geographically distributed, sharded, and consensus-based systems. I think you are being unfair to the people you're discussing with. Be nice.

mlyle 5y ago

Yes, but the underlying point you're willfully missing is: You can't engineer around AWS AZ common-mode failures using AWS. The moment that you have failures that are not independent and common mode, you can't just multiply together failure probabilities to know your outage times.

roughly 5y ago

During a recent AWS outage, the STS service running in us-east-1 was unavailable. Unfortunately, all of the other _regions_ - not AZs, but _regions_, rely on the STS service in us-east-1, which meant that customers which had built around Amazon’s published reliability model had services in every region impacted by an outage in one specific availability zone. This is what kreeben was referring to - not some abstract misconception about the difference between AZs and Regions, but an actual real world incident in which a failure in one AZ had an impact in other Regions.

otterley 5y ago
It's more subtle than that. For high availability, STS offers regional endpoints -- and AWS recommends using them[1] -- but the SDKs don't use them by default. The author of the client code, or the person configuring the software, has to enable them. [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... (I work for AWS. Opinions are my own and not necessarily those of my employer.)
- roughly 5y ago
  The client code which defaults to STS in us-east-1 includes the AWS console website, as far as I can tell. Real question, though - are those genuinely separate endpoints that remained up and operational during the outage? I don’t think I saw or knew a single person unaffected by this outage, so either there’s still some bleed over on the backend or knowledge of the regional STS endpoints is basically zero (which I Can believe, y’all run a big shop)
  - Karrot_Kream 5y ago
    My team didn't use STS but I know other teams at the company did. Those that did rely on non-us-east-1 endpoints did stay up IIRC. Our company barely use the AWS console at all and base most of our stuff around their APIs to hook into our deployment/CI processes. But I don't work at AWS so I don't know if it's true or if there was some other backend replication lag or anything else going on that was impacted by us-east-1 being down. We had some failures for some of our older services that were not properly sharded out, but most of our stuff failed over and continued to work as expected.

mlyle 5y ago

> knowledgable engineers (not like the kinds in this comment thread who are conflating availability zones and regions) I think this breaks the site guidelines. Worse, I don't think the other people are wrong: being in a different region implies being in a different availability zone. That is, I've read the comments to say "they're not only in different AZ's, they're in different regions". It seems you seem determined to pick a reading that lets you feel smugly superior about your level of knowledge, and then cast out digs at other people based on that presumed superiority.

Karrot_Kream 5y ago
> Worse, I don't think the other people are wrong: being in a different region implies being in a different availability zone. Availability zones do not map across regions. AZs are specific to a region. Different regions have differing numbers of AZs. us-east-1 has 3. IIRC ap-southeast-1 has 2. > That is, I've read the comments to say "they're not only in different AZ's, they're in different regions" So I've read. The earlier example about STS that someone brought up was incorrect; both I and another commenter linked to the doc with the correct information. > It seems you seem determined to pick a reading that lets you feel smugly superior about your level of knowledge, and then cast out digs at other people based on that presumed superiority. You obviously feel very strongly about this. You've replied to my parent twice now. You're right that the parenthetical was harsh but I wouldn't say it's uncalled for. Every one of these outage threads descends into a slew of easily defensible complaints about cloud providers. The quality of these discussions is terrible. I spend a lot of time at my dayjob (and as a hobby) working on networking related things. Understanding the subtle guarantees offered by AWS is a large part of my day-to-day. When I see people here make easily falsifiable comments full of hearsay ("I had a friend of a friend who works at Amazon and they did X, Y, Z bad things") and use that to drum up a frenzy, it flies in the face of what I do everyday. There's lots of issues with cloud providers as a whole and AWS in particular but to get to that level of conversation you need to understand what the system is actually doing, not just get angry and guess why it's failing.
- mlyle 5y ago
  > > being in a different region implies being in a different availability zone. > Availability zones do not map across regions. AZs are specific to a region. Different regions have differing numbers of AZs. us-east-1 has 3. IIRC ap-southeast-1 has 2. Right.. So if you are in a different region, you are by definition in a different availability zone. > You obviously feel very strongly about this. You've replied to my parent twice now. You're right that the parenthetical was harsh but I wouldn't say it's uncalled for. Yah, I really thought about it and you're just reeking unkindness. And the people above that you're replying to and mocking are not wrong. > Every one of these outage threads descends into a slew of easily defensible complaints about cloud providers. The quality of these discussions is terrible. I spend a lot of time at my dayjob (and as a hobby) working on networking related things. Understanding the subtle guarantees offered by AWS is a large part of my day-to-day. If you're unable to be civil about this, maybe you should avoid the threads. Amazon seeks to avoid common-mode failures between AZs (and thus regions). This doesn't mean that Amazon attains this goal. And the larger point: as I'm sure you're aware, building a distributed system that attains higher uptimes by crossing multiple AZs is hard and costly and can only be justified in some cases. I've got >20 years of experience in building geographically distributed, sharded, and consensus-based systems. I think you are being unfair to the people you're discussing with. Be nice.