4 ms·

This is a non-issue. "We are experiencing elevated API error rates and network connectivity errors in a single Availability Zone." Key fact "A SINGLE AZ". Av

This is a non-issue. "We are experiencing elevated API error rates and network connectivity errors in a single Availability Zone." Key fact "A SINGLE AZ". Availability Zone's are isolated from each other with redundant power supplies and internet connectivity and most often physically different datacenter locations. Well architected applications are designed to allow for a single AZ to become unavailable. This is precisely why the cloud is useful: you can bring up new capacity in the other Availability Zones behind the same load balancer with zero effort - the autoscaling does that for you automatically.

gjtempleton 7y ago
Except it's not currently handling this nicely, we've got an ASG behind an ELB, spread across multiple AZs including the affected one, and the ASG doesn't see the new instances as it scales up as ever coming into service - they just stick in "Not yet in service"
- carlsborg 7y ago
  I spun up a new t2.small instance in eu-central-1 and it took less than 90 seconds, which is more than it usually takes, but not that bad. https://pastebin.com/WC1hkh0c https://pastebin.com/WC1hkh0c (see ts on uptime command, ignore timezone differences)
- mvanbaak 7y ago
  We have exactly this for 2 autoscaling groups. Very anoying :(
- the_70x 7y ago
  ASG is in dead, not terminating or spinning new instances at this moment
lxgr 7y ago
Are you talking about stateless applications here? The story is a bit different for databases on AWS in my experience. Yes, you can do multi-AZ synchronous replication, but failover times are between 1-2 minutes. Also, I don't know how AWS implements failovers internally, but I can see several potential issues that could arise from connectivity loss between availability zones, especially if both EC2 and RDS are involved – few of which are trivial to handle if your application cannot tolerate loss of consistency.
- carlsborg 7y ago
  We use DynamoDB for storage and state and its unaffected. Are you referring to this kind of scenario? https://github.blog/2018-10-30-oct21-post-incident-analysis/ https://github.blog/2018-10-30-oct21-post-incident-analysis/
  - lxgr 7y ago
    Then you probably don't have the strict consistency and durability requirements that I mentioned. That is probably true for many use cases, but certainly not for all. Yes, after skimming your referenced article, this is exactly the type of issue that prevents a "simple" failover when databases are involved. Depending on your application, even returning stale data on one AZ in a split brain scenario might not be acceptable (this is a stronger requirement than not allowing non-reconcilable conflicting writes).
    - carlsborg 7y ago
      DynamoDB allows strongly consistent reads and your data is replicated across AZs. Id argue that its even better than an RDS setup in that respect. RDS lets you choose a DB Engine (MySQL, Postgres, etc). the replication depends on the engine you choose, e.g. Postgres native streaming. Heres a good writup on how it works and best practices (https://aws.amazon.com/blogs/database/best-practices-for-amazon-rds-postgresql-replication/ https://aws.amazon.com/blogs/database/best-practices-for-ama...)
aspyct 7y ago
What do you mean, "non issue"? We had timeouts all over the place. It doesn't matter how well you app is designed, if the network is congested, you're gonna have calls from your clients...
- carlsborg 7y ago
  Its a non-issue because you are not immune to the laws of probability. Things will fail, and you should expect them to. The beautiful thing here is that the failovers work to spec, and in case a single AZ goes out, mean time to recovery for you app can be in the order of single digit minutes (if built right).
  - runarb 7y ago
    My managed ElasticSearch cluster did go down despite being multi az.
  - thecopy 7y ago
    This is a very naive perspective to take. >The beautiful thing here is that the failovers work to spec Who's failover? AWS' certainly did not for us.
DasIch 7y ago
It doesn't affect a single AZ though. We see numerous problems with AWS services including but not limited to being unable to create new EC2 instances in any AZ. This is creating a huge number of issues as you lose capacity with the AZ going down and can't recover by creating new instances.
- ti_ranger 7y ago
  > We see numerous problems with AWS services including but not limited to being unable to create new EC2 instances in any AZ. How were you launching? Using EC2 directly (e.g. `aws ec2` cli or RunInstances API call - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R... ), or using e.g. ASGs? If you were launching directly with EC2, were you doing targeted launches (e.g. specifying Placement.AvailabilityZone for classic or SubnetId for VPC)? In the case of an AZ failure, launches which don't specify an AvailabilityZone or SubnetId may be impacted for a few minutes, whereas launches which do specify them should still succeed.
mrmattyboy 7y ago
I have a couple of arguments, even if you are spread evenly: * If you lose an AZ,, you are in a state of reduced redundency. * If you have intermittent drops, you can have lots of issues as it will be coming up/dropping out and cause issues. This will also have an impact on replication of services and cause additional load on instances in working AZs (RDS saw this).

gjtempleton 7y ago

Except it's not currently handling this nicely, we've got an ASG behind an ELB, spread across multiple AZs including the affected one, and the ASG doesn't see the new instances as it scales up as ever coming into service - they just stick in "Not yet in service"

carlsborg 7y ago
I spun up a new t2.small instance in eu-central-1 and it took less than 90 seconds, which is more than it usually takes, but not that bad. https://pastebin.com/WC1hkh0c https://pastebin.com/WC1hkh0c (see ts on uptime command, ignore timezone differences)
mvanbaak 7y ago
We have exactly this for 2 autoscaling groups. Very anoying :(
the_70x 7y ago
ASG is in dead, not terminating or spinning new instances at this moment

lxgr 7y ago

Are you talking about stateless applications here? The story is a bit different for databases on AWS in my experience. Yes, you can do multi-AZ synchronous replication, but failover times are between 1-2 minutes. Also, I don't know how AWS implements failovers internally, but I can see several potential issues that could arise from connectivity loss between availability zones, especially if both EC2 and RDS are involved – few of which are trivial to handle if your application cannot tolerate loss of consistency.

carlsborg 7y ago
We use DynamoDB for storage and state and its unaffected. Are you referring to this kind of scenario? https://github.blog/2018-10-30-oct21-post-incident-analysis/ https://github.blog/2018-10-30-oct21-post-incident-analysis/
- lxgr 7y ago
  Then you probably don't have the strict consistency and durability requirements that I mentioned. That is probably true for many use cases, but certainly not for all. Yes, after skimming your referenced article, this is exactly the type of issue that prevents a "simple" failover when databases are involved. Depending on your application, even returning stale data on one AZ in a split brain scenario might not be acceptable (this is a stronger requirement than not allowing non-reconcilable conflicting writes).
  - carlsborg 7y ago
    DynamoDB allows strongly consistent reads and your data is replicated across AZs. Id argue that its even better than an RDS setup in that respect. RDS lets you choose a DB Engine (MySQL, Postgres, etc). the replication depends on the engine you choose, e.g. Postgres native streaming. Heres a good writup on how it works and best practices (https://aws.amazon.com/blogs/database/best-practices-for-amazon-rds-postgresql-replication/ https://aws.amazon.com/blogs/database/best-practices-for-ama...)

aspyct 7y ago

What do you mean, "non issue"? We had timeouts all over the place. It doesn't matter how well you app is designed, if the network is congested, you're gonna have calls from your clients...

carlsborg 7y ago
Its a non-issue because you are not immune to the laws of probability. Things will fail, and you should expect them to. The beautiful thing here is that the failovers work to spec, and in case a single AZ goes out, mean time to recovery for you app can be in the order of single digit minutes (if built right).
- runarb 7y ago
  My managed ElasticSearch cluster did go down despite being multi az.
- thecopy 7y ago
  This is a very naive perspective to take. >The beautiful thing here is that the failovers work to spec Who's failover? AWS' certainly did not for us.

DasIch 7y ago

It doesn't affect a single AZ though. We see numerous problems with AWS services including but not limited to being unable to create new EC2 instances in any AZ. This is creating a huge number of issues as you lose capacity with the AZ going down and can't recover by creating new instances.

ti_ranger 7y ago
> We see numerous problems with AWS services including but not limited to being unable to create new EC2 instances in any AZ. How were you launching? Using EC2 directly (e.g. `aws ec2` cli or RunInstances API call - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R... ), or using e.g. ASGs? If you were launching directly with EC2, were you doing targeted launches (e.g. specifying Placement.AvailabilityZone for classic or SubnetId for VPC)? In the case of an AZ failure, launches which don't specify an AvailabilityZone or SubnetId may be impacted for a few minutes, whereas launches which do specify them should still succeed.

mrmattyboy 7y ago

I have a couple of arguments, even if you are spread evenly: * If you lose an AZ,, you are in a state of reduced redundency. * If you have intermittent drops, you can have lots of issues as it will be coming up/dropping out and cause issues. This will also have an impact on replication of services and cause additional load on instances in working AZs (RDS saw this).