5 ms·

Amazon EC2 Issues in Frankfurt AZ

36 points by mrmattyboy 7y ago

intsunny 7y ago
It is baffling that the Europe tab for AWS' Status page posts timestamps in PST. What the fuck is PST? We have the ability to do everything and anything in Javascript EXCEPT for fucking UTC or localized time zones. Is PST the new UTC? Is there an RFC for this? Did I miss the memo? Un-fucking-real.
- sm4rk0 7y ago
  That's due to the general ignorance about the world outside the US.
  - lxgr 7y ago
    Outside the Bay Area, really.
    - sargun 7y ago
      AWS is based in Seattle.
      - lxgr 7y ago
        Sorry, Bay Area plus Seattle ;)
  - vkizl 7y ago
    hurr durr murica bad
  - ti_ranger 7y ago
    Actually, the post was written by someone located outside the U.S. (a place not using any US-specific time-zone), and posted to the status page by someone else located outside the U.S.
- aspyct 7y ago
  PST - Potato Standard Time, everybody knows that.
- polack 7y ago
  And how about they inform us what availability zone is affected instead of just writing "single availability zone" 10 times.
  - AnssiH 7y ago
    That is not that easy as the availability zone numbering is account-specific. Doable, of course, but it is not just a matter of putting it into the issue description.
    - RKearney 7y ago
      Amazon publishes all the underlying zone names for your account so it would be trivial for them to specify the exact availability zone.
mangatmodi 7y ago
This year has been really bad for the cloud vendors in terms of stability.
- karma_fountain 7y ago
  Probably unrelated but there is also an issue with JIRA cloud based solution. https://status.atlassian.com/ https://status.atlassian.com/
  - mbo 7y ago
    It's related. Atlassian runs on AWS.
- StreamBright 7y ago
  This is a single AZ. Your application should tolerate single AZ outages. That is the first rule of building reliable, highly available services on AWS. 12:08 AM PST We are investigating increased network connectivity errors to instances in a single Availability Zone in the EU-CENTRAL-1 Region.
  - mvanbaak 7y ago
    True, but autoscaling at the moment is unable to scale in/out because of this issue. If your autoscaling group includes the affected AZ, you are out of luck because those instances are being terminated since their health checks fail. But because of the failures, autoscaling is unable to complete this, and unable to launch new instances in the other AZ's as it is stuck on the terminating part.
    - StreamBright 7y ago
      That is an interesting detail. Would you consider having 3 separate autoscaling groups (one per AZ) or this is not feasible for some reason? One of the interesting aspects of running services on AWS was to remove the AZ from "rotation" while the outage lasts. Meaning, having a DNS change and exclude the public endpoint from taking any traffic. If you have 3 separate groups doing 1/3 of the load and having independent DNS entries, autoscaling groups, etc. then moving traffic from one AZ to another is probably easier. Not sure about the details of your setup though, you might have reasons not to do this.
carlsborg 7y ago
This is a non-issue. "We are experiencing elevated API error rates and network connectivity errors in a single Availability Zone." Key fact "A SINGLE AZ". Availability Zone's are isolated from each other with redundant power supplies and internet connectivity and most often physically different datacenter locations. Well architected applications are designed to allow for a single AZ to become unavailable. This is precisely why the cloud is useful: you can bring up new capacity in the other Availability Zones behind the same load balancer with zero effort - the autoscaling does that for you automatically.
- gjtempleton 7y ago
  Except it's not currently handling this nicely, we've got an ASG behind an ELB, spread across multiple AZs including the affected one, and the ASG doesn't see the new instances as it scales up as ever coming into service - they just stick in "Not yet in service"
  - carlsborg 7y ago
    I spun up a new t2.small instance in eu-central-1 and it took less than 90 seconds, which is more than it usually takes, but not that bad. https://pastebin.com/WC1hkh0c https://pastebin.com/WC1hkh0c (see ts on uptime command, ignore timezone differences)
  - mvanbaak 7y ago
    We have exactly this for 2 autoscaling groups. Very anoying :(
  - the_70x 7y ago
    ASG is in dead, not terminating or spinning new instances at this moment
- lxgr 7y ago
  Are you talking about stateless applications here? The story is a bit different for databases on AWS in my experience. Yes, you can do multi-AZ synchronous replication, but failover times are between 1-2 minutes. Also, I don't know how AWS implements failovers internally, but I can see several potential issues that could arise from connectivity loss between availability zones, especially if both EC2 and RDS are involved – few of which are trivial to handle if your application cannot tolerate loss of consistency.
Roritharr 7y ago
We were affected with one of our marketing pages that we migrated to ECS in a non-HA configuration, our main applications are setup in Multi-AZ HA Configs and weren't affected by this. Not a big deal to remediate as all the other AZs are working.
fyfy18 7y ago
I wonder if this is related to TransferWise issues, their app and even card payments have been down all morning. https://twitter.com/TransferWise/status/1194168200210124800?s=20 https://twitter.com/TransferWise/status/1194168200210124800?...
nrki 7y ago
Looks like this took down TransferWise (debit card/forex/payments processor): https://status.transferwise.com https://status.transferwise.com

36 points by mrmattyboy 7y ago

It is baffling that the Europe tab for AWS' Status page posts timestamps in PST. What the fuck is PST? We have the ability to do everything and anything in Javascript EXCEPT for fucking UTC or localized time zones. Is PST the new UTC? Is there an RFC for this? Did I miss the memo? Un-fucking-real.

sm4rk0 7y ago
That's due to the general ignorance about the world outside the US.
- lxgr 7y ago
  Outside the Bay Area, really.
  - sargun 7y ago
    AWS is based in Seattle.
    - lxgr 7y ago
      Sorry, Bay Area plus Seattle ;)
- vkizl 7y ago
  hurr durr murica bad
- ti_ranger 7y ago
  Actually, the post was written by someone located outside the U.S. (a place not using any US-specific time-zone), and posted to the status page by someone else located outside the U.S.
aspyct 7y ago
PST - Potato Standard Time, everybody knows that.
polack 7y ago
And how about they inform us what availability zone is affected instead of just writing "single availability zone" 10 times.
- AnssiH 7y ago
  That is not that easy as the availability zone numbering is account-specific. Doable, of course, but it is not just a matter of putting it into the issue description.
  - RKearney 7y ago
    Amazon publishes all the underlying zone names for your account so it would be trivial for them to specify the exact availability zone.

mangatmodi 7y ago

This year has been really bad for the cloud vendors in terms of stability.

karma_fountain 7y ago
Probably unrelated but there is also an issue with JIRA cloud based solution. https://status.atlassian.com/ https://status.atlassian.com/
- mbo 7y ago
  It's related. Atlassian runs on AWS.
StreamBright 7y ago
This is a single AZ. Your application should tolerate single AZ outages. That is the first rule of building reliable, highly available services on AWS. 12:08 AM PST We are investigating increased network connectivity errors to instances in a single Availability Zone in the EU-CENTRAL-1 Region.
- mvanbaak 7y ago
  True, but autoscaling at the moment is unable to scale in/out because of this issue. If your autoscaling group includes the affected AZ, you are out of luck because those instances are being terminated since their health checks fail. But because of the failures, autoscaling is unable to complete this, and unable to launch new instances in the other AZ's as it is stuck on the terminating part.
  - StreamBright 7y ago
    That is an interesting detail. Would you consider having 3 separate autoscaling groups (one per AZ) or this is not feasible for some reason? One of the interesting aspects of running services on AWS was to remove the AZ from "rotation" while the outage lasts. Meaning, having a DNS change and exclude the public endpoint from taking any traffic. If you have 3 separate groups doing 1/3 of the load and having independent DNS entries, autoscaling groups, etc. then moving traffic from one AZ to another is probably easier. Not sure about the details of your setup though, you might have reasons not to do this.

carlsborg 7y ago

This is a non-issue. "We are experiencing elevated API error rates and network connectivity errors in a single Availability Zone." Key fact "A SINGLE AZ". Availability Zone's are isolated from each other with redundant power supplies and internet connectivity and most often physically different datacenter locations. Well architected applications are designed to allow for a single AZ to become unavailable. This is precisely why the cloud is useful: you can bring up new capacity in the other Availability Zones behind the same load balancer with zero effort - the autoscaling does that for you automatically.

gjtempleton 7y ago
Except it's not currently handling this nicely, we've got an ASG behind an ELB, spread across multiple AZs including the affected one, and the ASG doesn't see the new instances as it scales up as ever coming into service - they just stick in "Not yet in service"
- carlsborg 7y ago
  I spun up a new t2.small instance in eu-central-1 and it took less than 90 seconds, which is more than it usually takes, but not that bad. https://pastebin.com/WC1hkh0c https://pastebin.com/WC1hkh0c (see ts on uptime command, ignore timezone differences)
- mvanbaak 7y ago
  We have exactly this for 2 autoscaling groups. Very anoying :(
- the_70x 7y ago
  ASG is in dead, not terminating or spinning new instances at this moment
lxgr 7y ago
Are you talking about stateless applications here? The story is a bit different for databases on AWS in my experience. Yes, you can do multi-AZ synchronous replication, but failover times are between 1-2 minutes. Also, I don't know how AWS implements failovers internally, but I can see several potential issues that could arise from connectivity loss between availability zones, especially if both EC2 and RDS are involved – few of which are trivial to handle if your application cannot tolerate loss of consistency.

Roritharr 7y ago

We were affected with one of our marketing pages that we migrated to ECS in a non-HA configuration, our main applications are setup in Multi-AZ HA Configs and weren't affected by this. Not a big deal to remediate as all the other AZs are working.

fyfy18 7y ago

I wonder if this is related to TransferWise issues, their app and even card payments have been down all morning. https://twitter.com/TransferWise/status/1194168200210124800?s=20 https://twitter.com/TransferWise/status/1194168200210124800?...

nrki 7y ago

Looks like this took down TransferWise (debit card/forex/payments processor): https://status.transferwise.com https://status.transferwise.com