5 ms·

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage,

by EwanToo 14y ago

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive image will disappear at any moment - as it did here. Beyond that, any time you're running a database, no matter who the provider is, if you're not doing backups every day or hour, then you're not doing things right.

JohnHaugeland 14y ago
They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it is for me, a car industry outsider, to trust the claims that Chevrolet makes about my car. Do you know how to properly quality evaluate everything in your life? Hope you don't need a doctor, lawyer, or plumber soon.
- efsavage 14y ago
  I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ http://aws.amazon.com/ebs/. What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. Which to any engineer, "real" or otherwise, should be a pretty strong signal that something bad will happen.
  - andrewvc 14y ago
    Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.
    - danudey 14y ago
      Frequent snapshots only help you recover after a failure. RAID1 in software is a good way to prevent downtime, as well as potentially improve read speeds.
      - andrewvc 14y ago
        Not really, according to the EBS docs, they can pull missing blocks from EBS from S3. You shouldn't have to perform a restore. The uncertainty should only apply to the delta unless I'm mistaken.
- ceejayoz 14y ago
  > They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. Where do they do this? All the docs I've seen are pretty clear that EC2/EBS stuff could disappear and that you have to plan a fault-tolerant system.
- electrum 14y ago
  Amazon only claims "obscene number of nines" for S3 durability (99.999999999%). And this claim seems to be accurate: I've never seen a publicly reported case of anyone losing data. Anytime you read their forums about people reporting data loss, a typical response is AWS staff saying "we see delete requests for those objects on date X" with the users responding "oh, oops, we had this background delete process". However, for EBS volumes, Amazon is very clear about the expected data loss rate: "The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives."
  - JohnHaugeland 14y ago
    I withdraw my position as apparently in error. Thank you for the exceptionally clear counterpoint.
kennu 14y ago
What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.
- electrum 14y ago
  You use EBS and back it up using the built-in snapshot functionality, and/or back it up another way. RDS, which is presumably based on EBS, provides backups and snapshots for precisely this reason.
- EwanToo 14y ago
  The Netflix approach (which definitely isn't for everyone), is to use no persistent storage. They cluster their databases across multiple instances, availability zones and regions, and back them up constantly to S3. Their ultimate recovery plan is to restore from backup if necessary, which is pretty much the same as assuming EBS will corrupt your data. While it's far from trivial to do it that way, they seem to be the most successful user of the AWS systems.
  - semanticist 14y ago
    I'm more and more of the opinion that EC2 only makes sense for very small (2-3 node, no significant load) or very large (Netflix) deployments, and almost everyone else would be better off with either Linode-style VM rental or actual dedicated servers, depending on their needs. The only exception I've seen are people with very spiky traffic patterns, who can save a bit of money by only running instances when they need them - but even then I suspect that the money saved might be less than you'd think.
    - bluestix 14y ago
      For any constant load applications dedicated servers are definitely much cheaper then ec2. Amazon wins for APIs and convenience but not for price.
moe 14y ago
you simply should not use it without a very good reason EBS is fine as long as you're aware of the constraints and plan accordingly. you have to assume any given drive image will disappear at any moment Duh! You mean it behaves exactly as documented? That is outrageous.
jahewson 14y ago
You get what you pay for. Single-AZ databases were lost due to a failure in a single AZ, which Amazon tells you will happen. If you want durability you need Multi-AZ, which is the only place I'd put a production database.
snorkel 14y ago
You can snapshot EBS volumes, however your EBS disk I/O will be blocked while the snapshot is being made, so you can't do it often on a large busy volume. You can also raid EBS volumes together for realtime redundancy, but adds cost and complexity.
- EwanToo 14y ago
  True, you're probably better off running the replication at the database layer, which can normally by either synchronous or asynchronous depending on what people need. For people who don't want to spend the money on a hot standby, you can still produce the database log files from MySQL (or whatever you're using), and copy them off the machine onto S3 every few seconds, minutes, etc. There's no end of solutions to the problem but, for whatever reason, a lot of people don't see EBS as a potential problem area.

by EwanToo 14y ago

JohnHaugeland 14y ago

They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it is for me, a car industry outsider, to trust the claims that Chevrolet makes about my car. Do you know how to properly quality evaluate everything in your life? Hope you don't need a doctor, lawyer, or plumber soon.

efsavage 14y ago
I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ http://aws.amazon.com/ebs/. What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. Which to any engineer, "real" or otherwise, should be a pretty strong signal that something bad will happen.
- andrewvc 14y ago
  Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.
  - danudey 14y ago
    Frequent snapshots only help you recover after a failure. RAID1 in software is a good way to prevent downtime, as well as potentially improve read speeds.
    - andrewvc 14y ago
      Not really, according to the EBS docs, they can pull missing blocks from EBS from S3. You shouldn't have to perform a restore. The uncertainty should only apply to the delta unless I'm mistaken.
ceejayoz 14y ago
> They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. Where do they do this? All the docs I've seen are pretty clear that EC2/EBS stuff could disappear and that you have to plan a fault-tolerant system.
electrum 14y ago
Amazon only claims "obscene number of nines" for S3 durability (99.999999999%). And this claim seems to be accurate: I've never seen a publicly reported case of anyone losing data. Anytime you read their forums about people reporting data loss, a typical response is AWS staff saying "we see delete requests for those objects on date X" with the users responding "oh, oops, we had this background delete process". However, for EBS volumes, Amazon is very clear about the expected data loss rate: "The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives."
- JohnHaugeland 14y ago
  I withdraw my position as apparently in error. Thank you for the exceptionally clear counterpoint.

kennu 14y ago

What would you recommend for persistent disk storage on AWS instead of EBS then? Assuming you need to put database files somewhere where they don't disappear when instances terminate, and your database can access them.

electrum 14y ago
You use EBS and back it up using the built-in snapshot functionality, and/or back it up another way. RDS, which is presumably based on EBS, provides backups and snapshots for precisely this reason.
EwanToo 14y ago
The Netflix approach (which definitely isn't for everyone), is to use no persistent storage. They cluster their databases across multiple instances, availability zones and regions, and back them up constantly to S3. Their ultimate recovery plan is to restore from backup if necessary, which is pretty much the same as assuming EBS will corrupt your data. While it's far from trivial to do it that way, they seem to be the most successful user of the AWS systems.
- semanticist 14y ago
  I'm more and more of the opinion that EC2 only makes sense for very small (2-3 node, no significant load) or very large (Netflix) deployments, and almost everyone else would be better off with either Linode-style VM rental or actual dedicated servers, depending on their needs. The only exception I've seen are people with very spiky traffic patterns, who can save a bit of money by only running instances when they need them - but even then I suspect that the money saved might be less than you'd think.
  - bluestix 14y ago
    For any constant load applications dedicated servers are definitely much cheaper then ec2. Amazon wins for APIs and convenience but not for price.

moe 14y ago

you simply should not use it without a very good reason EBS is fine as long as you're aware of the constraints and plan accordingly. you have to assume any given drive image will disappear at any moment Duh! You mean it behaves exactly as documented? That is outrageous.

jahewson 14y ago

You get what you pay for. Single-AZ databases were lost due to a failure in a single AZ, which Amazon tells you will happen. If you want durability you need Multi-AZ, which is the only place I'd put a production database.

snorkel 14y ago

You can snapshot EBS volumes, however your EBS disk I/O will be blocked while the snapshot is being made, so you can't do it often on a large busy volume. You can also raid EBS volumes together for realtime redundancy, but adds cost and complexity.

EwanToo 14y ago
True, you're probably better off running the replication at the database layer, which can normally by either synchronous or asynchronous depending on what people need. For people who don't want to spend the money on a hot standby, you can still produce the database log files from MySQL (or whatever you're using), and copy them off the machine onto S3 every few seconds, minutes, etc. There's no end of solutions to the problem but, for whatever reason, a lot of people don't see EBS as a potential problem area.