6 ms·

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced

by jwr 17y ago

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

mechanical_fish 17y ago
There are people in the world who do not understand probability. Like the OP: EBS is either reliable or not. You cannot be a little pregnant. Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages. They happen all the time, often before a woman realizes she is pregnant. Then there are the false pregnancies: http://en.wikipedia.org/wiki/False_pregnancy http://en.wikipedia.org/wiki/False_pregnancy And these are just the common edge cases.
- astrodust 17y ago
  You can be "a little bit pregnant" when you're talking about large groups of people. Of the women in your country, how many are pregnant at any given time? It's probably "a little bit". Obviously someone is confusing a statistical sampling of 1 with the kind of volume Amazon must deal with. I find it informative that a rare event such as this is being given so much press. If this was more routine, this wouldn't be news. Amazon seems to be doing better than 0.1% based on that alone.
mynameishere 17y ago
I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?
- kiujhygfghjk 17y ago
  Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!
  - mynameishere 17y ago
    If amazon is advertising that single failures can't cause data losses, then a single exploding power supply (for instance) breaks that agreement.
    - azim 17y ago
      If by "exploding" power supply you mean the potential for a power supply to explode with eruptive force and destroy other equipment in the rack, I believe that the other equipment being destroyed would be considered an multiple failure. It's very unlikely that a power supply could do that (although I guess anything is possible). A proper datacenter built with Datacenter-grade equipment will have multiple redundant power supplies in a blade enclosure, fed by different rails, which come off of different main lines in to the building. So a single failed source of power won't cause a failure.
  - Tamerlin 17y ago
    What's more relevant is that the odds of either drive failing are identical. The likelihood that the first drive will fail doesn't change because the second one failed -- which is the classic statistical blunder. ("I got 42 heads in a row, the next one's BOUND to be tails!") Every single drive in the datacenter typically has the same likelihood of failure (since they're usually the same make and model, and similar production runs), so the odds that a drive somewhere in a large data center is failing RIGHT NOW are in reality rather high. The Big Mac ended up being a good example of this. The likelihood of having a memory error on a normal PC is low, because the likelihood of a single DIMM having a memory error is low. Punch that up to the 22,000 or thereabouts DIMMs that populated the Big Mac cluster, and now you're looking at a very HIGH likelihood that a large-scale computation using the cluster will experience memory errors, and therefore produce invalid results -- which is why UVa ended up replacing the entire cluster with ECC-equipped machines in short order. Until they did that, researchers had to run simulations multiple times and compare results to make sure that their simulations weren't contaminated by memory errors.
    - kiujhygfghjk 17y ago
- lallysingh 17y ago
  Higher than expected: http://labs.google.com/papers/disk_failures.pdf http://labs.google.com/papers/disk_failures.pdf
- lamby 17y ago
  Very low. But that assumes they are independent events, which has never been the case in my experience.
- jonknee 17y ago
  The percentages they give aren't for a single drive failure, they're for data loss (multiple drive failure). So the odds are .1-.5%.
- btilly 17y ago
  Your assumptions are unreasonable, insufficiently well specified, and are asking the wrong question. If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse. Next, you're asking about the odds of 2 failing on the same machine. How many disks are on a machine? 1? 10? 100? Are failures independent events? It makes a huge difference. In fact they are not independent because when the motherboard craps out you lose access to all disks on that machine at once. At their scale it is too much work to figure out whether some of that data is recoverable - you just assume there is another copy somewhere and throw away the stale data. If you're wrong, then oops. You are also throwing out the 15 minute disk replacement time. It may take 15 minutes to replace a disk, but that figure is irrelevant. To replace a disk you have to locate the machine, and it has to matter enough to you to send a person out. I guarantee you that the time before a person gets involved is going to average more than 15 minutes. Generally a lot more than 15 minutes. (Google famously takes the attitude that it is generally more work than it is worth to find the broken machine, and lets most dead machines sit there indefinitely. I wouldn't be surprised if other cloud providers imitate this.) Next you have to consider that the end user shouldn't care about machines. For the purpose of redundancy Amazon is not going to keep multiple copies of the same data on the same machine. They are going to put them in different machines, and hopefully in different places. That will reduce the odds of a single failure losing your data. All of that said I am somewhat shocked that Amazon would advertise a 0.5-0.1% rate of data loss as acceptable. I don't know Google's actual failure rate, but I'd be willing to bet large amounts of money that it is much lower than that. For instance search for "gmail lost data". The only significant gmail data loss that turns up was in 2006. (See http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-mass-email-deletions/ http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-m... for more.) A grand total of 60 accounts got wiped out. Subsequently most of the lost data was restored from backup. (I doubt that the error was at the data storage layer.) That's not just better than what Amazon delivers. That is ridiculously better.
  - skorgu 17y ago
    To be fair you'd have to include failures of EBS snapshots and of failures to multiple datacenters to gain parity with Google in your comparison. I'm sure the gmail app doesn't use its storage subsystem naively no matter what the numbers are. You're absolutely right in general though.
jacquesm 17y ago
No, he experienced a 'user error'. And those happen far more frequently.
GrandMasterBirt 17y ago
The OP's point: Either you are reliable or not. If you claim that the data is backed up and so on, then you are giving a garantee. Now yes, if all of amazon's datacenters burn down, or the one where ur data is, yes it will be lost, but that should be the corner case that you prepare for. Thats when you do an offsite backup in your house daily/weekly to ensure that at least there is no one place that burns down = company down the tube. No matter the solution there is always the probability that shit will happen to all of it. However if you are selling a reliable service, don't sell an unreliable reliable service. Reliable should mean that at least you back it up so that if one hard drive rack blows up, the data is not gone.
- dedward 17y ago
  They didn't say it was bulletproof. They stated that data was replicated to more than one device to ensure that a single component failure would not result in data loss. They didn't say it was bulletproof or impossible to loose data. They explain fairly well how things work, what the failure rates are, and give you the tools to do the same risk analysis and cost/benefit calculations you would do anyway, whether using a cloud service or rolling your own. Multiple equipment failure can happen. Even across data centers and availability zones. The larger the entire AWS system gets, the higher the chances of eventually seeing edge cases where the wrong equipment at just the wrong time screws up to lose data. If you want bulletproof data storage system that has such a rediculously low failure rate that you are guaranteed not to loose data for a hundred years, you can get it - but it's going to cost a heck of a lot more than anything Amazon is selling you.

by jwr 17y ago

mechanical_fish 17y ago

There are people in the world who do not understand probability. Like the OP: EBS is either reliable or not. You cannot be a little pregnant. Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages. They happen all the time, often before a woman realizes she is pregnant. Then there are the false pregnancies: http://en.wikipedia.org/wiki/False_pregnancy http://en.wikipedia.org/wiki/False_pregnancy And these are just the common edge cases.

astrodust 17y ago
You can be "a little bit pregnant" when you're talking about large groups of people. Of the women in your country, how many are pregnant at any given time? It's probably "a little bit". Obviously someone is confusing a statistical sampling of 1 with the kind of volume Amazon must deal with. I find it informative that a rare event such as this is being given so much press. If this was more routine, this wouldn't be news. Amazon seems to be doing better than 0.1% based on that alone.

mynameishere 17y ago

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

kiujhygfghjk 17y ago
Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!
- mynameishere 17y ago
  If amazon is advertising that single failures can't cause data losses, then a single exploding power supply (for instance) breaks that agreement.
  - azim 17y ago
    If by "exploding" power supply you mean the potential for a power supply to explode with eruptive force and destroy other equipment in the rack, I believe that the other equipment being destroyed would be considered an multiple failure. It's very unlikely that a power supply could do that (although I guess anything is possible). A proper datacenter built with Datacenter-grade equipment will have multiple redundant power supplies in a blade enclosure, fed by different rails, which come off of different main lines in to the building. So a single failed source of power won't cause a failure.
- Tamerlin 17y ago
  What's more relevant is that the odds of either drive failing are identical. The likelihood that the first drive will fail doesn't change because the second one failed -- which is the classic statistical blunder. ("I got 42 heads in a row, the next one's BOUND to be tails!") Every single drive in the datacenter typically has the same likelihood of failure (since they're usually the same make and model, and similar production runs), so the odds that a drive somewhere in a large data center is failing RIGHT NOW are in reality rather high. The Big Mac ended up being a good example of this. The likelihood of having a memory error on a normal PC is low, because the likelihood of a single DIMM having a memory error is low. Punch that up to the 22,000 or thereabouts DIMMs that populated the Big Mac cluster, and now you're looking at a very HIGH likelihood that a large-scale computation using the cluster will experience memory errors, and therefore produce invalid results -- which is why UVa ended up replacing the entire cluster with ECC-equipped machines in short order. Until they did that, researchers had to run simulations multiple times and compare results to make sure that their simulations weren't contaminated by memory errors.
  - kiujhygfghjk 17y ago
lallysingh 17y ago
Higher than expected: http://labs.google.com/papers/disk_failures.pdf http://labs.google.com/papers/disk_failures.pdf
lamby 17y ago
Very low. But that assumes they are independent events, which has never been the case in my experience.
jonknee 17y ago
The percentages they give aren't for a single drive failure, they're for data loss (multiple drive failure). So the odds are .1-.5%.
btilly 17y ago
Your assumptions are unreasonable, insufficiently well specified, and are asking the wrong question. If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse. Next, you're asking about the odds of 2 failing on the same machine. How many disks are on a machine? 1? 10? 100? Are failures independent events? It makes a huge difference. In fact they are not independent because when the motherboard craps out you lose access to all disks on that machine at once. At their scale it is too much work to figure out whether some of that data is recoverable - you just assume there is another copy somewhere and throw away the stale data. If you're wrong, then oops. You are also throwing out the 15 minute disk replacement time. It may take 15 minutes to replace a disk, but that figure is irrelevant. To replace a disk you have to locate the machine, and it has to matter enough to you to send a person out. I guarantee you that the time before a person gets involved is going to average more than 15 minutes. Generally a lot more than 15 minutes. (Google famously takes the attitude that it is generally more work than it is worth to find the broken machine, and lets most dead machines sit there indefinitely. I wouldn't be surprised if other cloud providers imitate this.) Next you have to consider that the end user shouldn't care about machines. For the purpose of redundancy Amazon is not going to keep multiple copies of the same data on the same machine. They are going to put them in different machines, and hopefully in different places. That will reduce the odds of a single failure losing your data. All of that said I am somewhat shocked that Amazon would advertise a 0.5-0.1% rate of data loss as acceptable. I don't know Google's actual failure rate, but I'd be willing to bet large amounts of money that it is much lower than that. For instance search for "gmail lost data". The only significant gmail data loss that turns up was in 2006. (See http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-mass-email-deletions/ http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-m... for more.) A grand total of 60 accounts got wiped out. Subsequently most of the lost data was restored from backup. (I doubt that the error was at the data storage layer.) That's not just better than what Amazon delivers. That is ridiculously better.
- skorgu 17y ago
  To be fair you'd have to include failures of EBS snapshots and of failures to multiple datacenters to gain parity with Google in your comparison. I'm sure the gmail app doesn't use its storage subsystem naively no matter what the numbers are. You're absolutely right in general though.

jacquesm 17y ago

No, he experienced a 'user error'. And those happen far more frequently.

GrandMasterBirt 17y ago

The OP's point: Either you are reliable or not. If you claim that the data is backed up and so on, then you are giving a garantee. Now yes, if all of amazon's datacenters burn down, or the one where ur data is, yes it will be lost, but that should be the corner case that you prepare for. Thats when you do an offsite backup in your house daily/weekly to ensure that at least there is no one place that burns down = company down the tube. No matter the solution there is always the probability that shit will happen to all of it. However if you are selling a reliable service, don't sell an unreliable reliable service. Reliable should mean that at least you back it up so that if one hard drive rack blows up, the data is not gone.

dedward 17y ago
They didn't say it was bulletproof. They stated that data was replicated to more than one device to ensure that a single component failure would not result in data loss. They didn't say it was bulletproof or impossible to loose data. They explain fairly well how things work, what the failure rates are, and give you the tools to do the same risk analysis and cost/benefit calculations you would do anyway, whether using a cloud service or rolling your own. Multiple equipment failure can happen. Even across data centers and availability zones. The larger the entire AWS system gets, the higher the chances of eventually seeing edge cases where the wrong equipment at just the wrong time screws up to lose data. If you want bulletproof data storage system that has such a rediculously low failure rate that you are guaranteed not to loose data for a hundred years, you can get it - but it's going to cost a heck of a lot more than anything Amazon is selling you.