4 ms·

We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at hand

We've had alerts for packet loss and had issues in recovering region-spanning services (both AWS and 3rd party). Yes, some of these we should be better at handling ourselves, but... it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands. edit: just to short circuit any "well, why aren't you running redundant regions" - we run redundant regions at all times. But for reasons of latency, many customers will bind to their closest region, and the nature of our technology is highly location-bound It is not possible for us to move active sessions to an alternate region. So something like this is... unpleasant.

bobby_the_whale 5y ago
mentat 5y ago
You don't have health checks?
- codeduck 5y ago
  How are health checks supposed to help when you can't do anything?
  - buchanmilne 5y ago
    You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances. If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way Does that help, or have I misunderstood your problem?
    - codeduck 5y ago
      You have misunderstood our problem. Ec2 behind elb/alb/nlb is the least of our issues.

bobby_the_whale 5y ago

mentat 5y ago

You don't have health checks?

codeduck 5y ago
How are health checks supposed to help when you can't do anything?
- buchanmilne 5y ago
  You said: > it's all very well to say "expect to lose an AZ" but during this outage it's not been physically possible to remove the broken AZ instances from multi-AZ services because we cannot physically get them to respond to or acknowledge commands "Expect to lose an AZ" includes not being able to make any changes to existing instances in the affected AZ. If you had instances across multiple AZs behind an ELB with health checks, then the ELB should automatically remove the affected instances. If you have a different architecture, you would want to: * Have another mechanism that automatically stops sending traffic to impaired instances (ideal), or * Have a means to manually remove the instances from service without being able to interact with or modify those instances in any way Does that help, or have I misunderstood your problem?
  - codeduck 5y ago
    You have misunderstood our problem. Ec2 behind elb/alb/nlb is the least of our issues.