4 ms·

I think the main discrepancy is that you see networking in a single zone not working as a catastrophic event, but AWS doesn't. In all the AWS documentation you

I think the main discrepancy is that you see networking in a single zone not working as a catastrophic event, but AWS doesn't. In all the AWS documentation you are reminded time and time again that if you want any guarantee of availability: choose two zones. If you want more availability: choose two regions. If you're operating globally, you probably should anyway.

ajmurmann 10y ago
It is of course true that the author should have deployed to two zones. That said you'd still noticed something is going wrong and now need to identify what's happening and if you need to take any corrective action. Being easily able to see that half your infrastructure is down and you didn't break it would save allot of time and nerves in a potentially critical situation. Unless you significantly overprovision you likely will also need to scale up the other zone. For that decision making it's again important to quickly know what's happening. Amazon's reporting helps nobody but their reporting and probably internal metrics and likely bonuses.
- jedberg 10y ago
  It's probably not wise to rely on your provider's monitoring for such critical things. As AWS has proven, their alerts are slow, and even if they weren't, they aren't monitoring the very specific thing you need. Your best bet is to monitor your own systems, and have enough monitoring in place to tell you that one zone is unavailable without having to rely on AWS to tell you. Their dashboard has no effect on their bonuses BTW (at least it didn't the last time I asked), but it is slow to update because it is purposely gated by a human so as not to cause false positives, and that human has to manually verify the problem before reporting it, which takes time.
  - deleted 10y ago
    [deleted]
  - hueving 10y ago
    >It's probably not wise to rely on your provider's monitoring for such critical things. As AWS has proven, their alerts are slow, and even if they weren't, they aren't monitoring the very specific thing you need. In this particular case they are not monitoring that they lost an entire zone? This seems like a cop-out to cover up for the fact that they didn't reveal a major outage that they most definitely monitor internally.
- matt_oriordan 10y ago
  We use three availability zones. Unfortunately that was not the point of the article. The point was one AZ was effectively completely down, yet AWS was reporting the entire region as healthy without any warnings.
matt_oriordan 10y ago
Joel, the point of my article was nothing to do with architecture and using multiple AZs. Of course we use a minimum of two AZs and often three in reach region. The problem is that AWS continued to report the entire region as completely healthy in spit of an entire AZ being down, and worse, continuing to route traffic from an ELB in that AZ to that unhealthy AZ data center.
- joelhaasnoot 10y ago
  Do you have health checks implemented corrected? ELB isn't magic: if the ELB is running and your instance isn't the health check needs to return that your application is unhealthy.

ajmurmann 10y ago

It is of course true that the author should have deployed to two zones. That said you'd still noticed something is going wrong and now need to identify what's happening and if you need to take any corrective action. Being easily able to see that half your infrastructure is down and you didn't break it would save allot of time and nerves in a potentially critical situation. Unless you significantly overprovision you likely will also need to scale up the other zone. For that decision making it's again important to quickly know what's happening. Amazon's reporting helps nobody but their reporting and probably internal metrics and likely bonuses.

jedberg 10y ago
It's probably not wise to rely on your provider's monitoring for such critical things. As AWS has proven, their alerts are slow, and even if they weren't, they aren't monitoring the very specific thing you need. Your best bet is to monitor your own systems, and have enough monitoring in place to tell you that one zone is unavailable without having to rely on AWS to tell you. Their dashboard has no effect on their bonuses BTW (at least it didn't the last time I asked), but it is slow to update because it is purposely gated by a human so as not to cause false positives, and that human has to manually verify the problem before reporting it, which takes time.
- deleted 10y ago
  [deleted]
- hueving 10y ago
  >It's probably not wise to rely on your provider's monitoring for such critical things. As AWS has proven, their alerts are slow, and even if they weren't, they aren't monitoring the very specific thing you need. In this particular case they are not monitoring that they lost an entire zone? This seems like a cop-out to cover up for the fact that they didn't reveal a major outage that they most definitely monitor internally.
matt_oriordan 10y ago
We use three availability zones. Unfortunately that was not the point of the article. The point was one AZ was effectively completely down, yet AWS was reporting the entire region as healthy without any warnings.

matt_oriordan 10y ago

Joel, the point of my article was nothing to do with architecture and using multiple AZs. Of course we use a minimum of two AZs and often three in reach region. The problem is that AWS continued to report the entire region as completely healthy in spit of an entire AZ being down, and worse, continuing to route traffic from an ELB in that AZ to that unhealthy AZ data center.

joelhaasnoot 10y ago
Do you have health checks implemented corrected? ELB isn't magic: if the ELB is running and your instance isn't the health check needs to return that your application is unhealthy.