5 ms·

How many times do we have to hear about EBS issues and companies like Reddit reporting EBS being a disaster before we stop seeing outages from it? Either, Amazo

How many times do we have to hear about EBS issues and companies like Reddit reporting EBS being a disaster before we stop seeing outages from it? Either, Amazon is looking into it, publicly, or people have wised up to the facts and stop using EBS-backed services. Those include Elastic Load Balancer (ELB), Relational Database Service (RDS), Elastic Beanstalk, and others. It's just part of the AWS fabric; you "need" to be in multi-region, multi-AZ, and architect to be on ephemeral disks that can disappear at at any instant.

mattlong 13y ago
I agree with you 100%. For those of us needing to run MySQL, what would be the best ways to minimize the dependency on EBS and maximize HA?
- nknighthb 13y ago
  I haven't implemented this on EC2 myself, but Galera Cluster is a godsend for MariaDB/MySQL HA. Multi-AZ is probably viable, but multi-region may hurt. A multi-AZ cluster with ordinary binary log replication to another region (with preparations to launch a new cluster in that region based on that slave on short notice) might be a good solution.
  - falcolas 13y ago
    You would not want to implement PXC across regions - the round trip time would be too hard in individual transactions, and the ISTs would be murder to your bandwidth. Between AZs should be better, but remember with PXC that your transaction time is limited by your network round trip to the slowest node. Regular asynchronous replication is your best bet, with a regular run of pt-table-checksum scheduled to ensure your data is consistent.
    - nknighthb 13y ago
      For those confused (as I was at first), falcolas is referencing Percona XtraDB Cluster, a particular distribution of MySQL + XtraDB + Galera Cluster. > Regular asynchronous replication is your best bet That doesn't get you HA in any practical form. I've done the heartbeat thing, it created more outages by itself than would have occurred with no HA solution at all, usually failed to kick in during a real failure, and we could never fix all the split-brain scenarios. Galera is absolute magic by comparison. Hence my suggestion that a cluster be deployed within one datacenter, with asynchronous replication to a standby.
      - falcolas 13y ago
        Galera is magic, I'll agree, but there are just too many shortcomings for me to recommend it for most people. Heartbeat is very problematic, I agree, but it's hardly the only solution out there (and far from the best solution). That said, I have learned that it's remarkably configurable, so many of the problems you encountered could probably be addressed, if you're willing to learn about Pacemaker and really dig into its configuration.
        nknighthb 13y ago
        I wish you'd elaborate on those shortcomings, because individual transaction latency is the only real one I've found, and for most workloads, it's not nearly enough to overcome the HA and CPU scaling benefits.
- falcolas 13y ago
  We run a master-master setup between Oregon and Virginia, and its served us very well. We monitor MySQL instances for multiple customers, so uptime is a must; having a multi region setup has filled that role perfectly.
- scott_w 13y ago
  Can't you use RDS? It has Multi-AZ replication and failover. It may seem costly, but it means you don't need to manage all that stuff yourself. I should mention that the system I'm building that uses it has yet to go into production, so others may have more experience on this than I do.
  - mattlong 13y ago
    We're already using Multi-AZ, but still had DB connectivity issues throughout the outage.
spartango 13y ago
EBS has its problems, but I think a blanket rejection of EBS is probably unwarranted. It's true that in 2011, EBS was very shaky. Amazon has since invested quite a bit of energy into EBS; at that time it was a new service that couldn't handle its own growth. EBS has become substantially more reliable, both in terms of availability and performance consistency (provisioned IOPS). That doesn't mean you should rely on it absolutely, at least within a single AZ. Multi-AZ is a critical part of any AWS deployment at scale, if not multi-region (depending on your needs). EBS is not a cure-all, either; if you have higher performance demands than it offers (between provisioned IOPS and IO-optimized instances) then that's fine, just measure first. Finally, I'd like to point out that Reddit's complaints about EBS aren't the best to echo as an example of EBS issues: first, they are a few years old, and at the time both Reddit and EBS were architected with substantial deficiencies.
abalone 13y ago
Are Instagram, Vine and Airbnb NOT multi-AZ??
- thezilch 13y ago
  I can't say for sure, but they could be relying on an EBS-backed service (ELB, RDS, etc) or suffered from an AWS issue that suffers even in multi-AZ. A lot of AWS outages could have only been mitigated with multi-region.
  - abalone 13y ago
    Amazon claims the issues only affected 1 AZ. From approximately 12:51 PM PDT to 1:42 PM PDT network packet loss caused elevated EBS-related API error rates in a single AZ, a small number of EBS volumes in that AZ to experience degraded performance, and a small number of EC2 instances to become unreachable due to packet loss in a single AZ in the US-EAST-1 Region. http://status.aws.amazon.com http://status.aws.amazon.com So.. are we to believe that Instagram, Vine, etc. are not properly architected for multi-AZ failover? Or is there more to this than what Amazon is reporting? Bottom line, why did these apps have any significant downtime at all?
    - thezilch 13y ago
      Potentially, they do not have multi-AZ ELBs and only thought to have their EC2 or similar instances in multi-AZ. Or they don't have proper tuning of their DNS to stop routing to failing ELBs. Or... Amazon is unaware or not making us aware of the side-effects that their subnet outages tend to have on the overall network in a region.
eikenberry 13y ago
EBS is now per-zone. So as long as you are properly distributed across zones you should be fine. The only caveat to this would be if you have a single point of failure. Like a PostgreSQL database or something else that just doesn't have a good high availability story.

mattlong 13y ago

I agree with you 100%. For those of us needing to run MySQL, what would be the best ways to minimize the dependency on EBS and maximize HA?

nknighthb 13y ago
I haven't implemented this on EC2 myself, but Galera Cluster is a godsend for MariaDB/MySQL HA. Multi-AZ is probably viable, but multi-region may hurt. A multi-AZ cluster with ordinary binary log replication to another region (with preparations to launch a new cluster in that region based on that slave on short notice) might be a good solution.
- falcolas 13y ago
  You would not want to implement PXC across regions - the round trip time would be too hard in individual transactions, and the ISTs would be murder to your bandwidth. Between AZs should be better, but remember with PXC that your transaction time is limited by your network round trip to the slowest node. Regular asynchronous replication is your best bet, with a regular run of pt-table-checksum scheduled to ensure your data is consistent.
  - nknighthb 13y ago
    For those confused (as I was at first), falcolas is referencing Percona XtraDB Cluster, a particular distribution of MySQL + XtraDB + Galera Cluster. > Regular asynchronous replication is your best bet That doesn't get you HA in any practical form. I've done the heartbeat thing, it created more outages by itself than would have occurred with no HA solution at all, usually failed to kick in during a real failure, and we could never fix all the split-brain scenarios. Galera is absolute magic by comparison. Hence my suggestion that a cluster be deployed within one datacenter, with asynchronous replication to a standby.
    - falcolas 13y ago
      Galera is magic, I'll agree, but there are just too many shortcomings for me to recommend it for most people. Heartbeat is very problematic, I agree, but it's hardly the only solution out there (and far from the best solution). That said, I have learned that it's remarkably configurable, so many of the problems you encountered could probably be addressed, if you're willing to learn about Pacemaker and really dig into its configuration.
      - nknighthb 13y ago
        I wish you'd elaborate on those shortcomings, because individual transaction latency is the only real one I've found, and for most workloads, it's not nearly enough to overcome the HA and CPU scaling benefits.
falcolas 13y ago
We run a master-master setup between Oregon and Virginia, and its served us very well. We monitor MySQL instances for multiple customers, so uptime is a must; having a multi region setup has filled that role perfectly.
scott_w 13y ago
Can't you use RDS? It has Multi-AZ replication and failover. It may seem costly, but it means you don't need to manage all that stuff yourself. I should mention that the system I'm building that uses it has yet to go into production, so others may have more experience on this than I do.
- mattlong 13y ago
  We're already using Multi-AZ, but still had DB connectivity issues throughout the outage.

spartango 13y ago

EBS has its problems, but I think a blanket rejection of EBS is probably unwarranted. It's true that in 2011, EBS was very shaky. Amazon has since invested quite a bit of energy into EBS; at that time it was a new service that couldn't handle its own growth. EBS has become substantially more reliable, both in terms of availability and performance consistency (provisioned IOPS). That doesn't mean you should rely on it absolutely, at least within a single AZ. Multi-AZ is a critical part of any AWS deployment at scale, if not multi-region (depending on your needs). EBS is not a cure-all, either; if you have higher performance demands than it offers (between provisioned IOPS and IO-optimized instances) then that's fine, just measure first. Finally, I'd like to point out that Reddit's complaints about EBS aren't the best to echo as an example of EBS issues: first, they are a few years old, and at the time both Reddit and EBS were architected with substantial deficiencies.

abalone 13y ago

Are Instagram, Vine and Airbnb NOT multi-AZ??

thezilch 13y ago
I can't say for sure, but they could be relying on an EBS-backed service (ELB, RDS, etc) or suffered from an AWS issue that suffers even in multi-AZ. A lot of AWS outages could have only been mitigated with multi-region.
- abalone 13y ago
  Amazon claims the issues only affected 1 AZ. From approximately 12:51 PM PDT to 1:42 PM PDT network packet loss caused elevated EBS-related API error rates in a single AZ, a small number of EBS volumes in that AZ to experience degraded performance, and a small number of EC2 instances to become unreachable due to packet loss in a single AZ in the US-EAST-1 Region. http://status.aws.amazon.com http://status.aws.amazon.com So.. are we to believe that Instagram, Vine, etc. are not properly architected for multi-AZ failover? Or is there more to this than what Amazon is reporting? Bottom line, why did these apps have any significant downtime at all?
  - thezilch 13y ago
    Potentially, they do not have multi-AZ ELBs and only thought to have their EC2 or similar instances in multi-AZ. Or they don't have proper tuning of their DNS to stop routing to failing ELBs. Or... Amazon is unaware or not making us aware of the side-effects that their subnet outages tend to have on the overall network in a region.

eikenberry 13y ago

EBS is now per-zone. So as long as you are properly distributed across zones you should be fine. The only caveat to this would be if you have a single point of failure. Like a PostgreSQL database or something else that just doesn't have a good high availability story.