8 ms·

"Amazon EBS sucks. I just lost all my data"

111 points by cemerick 16y ago

brk 16y ago
Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a certain minimal amount of scale). But you DO need to have someone who is thinking about backups and availability, and a valid solution is not to assume that the smart folks at Amazon or Rackspace or any other hosting provider are going to be completely and consistently working with your best interests and uptime in mind. EVRYTHING fails at some point. Every server, every generator, every upstream connection, every hosting provider big or small. And in this case I mean fail as in goes dark for some period of time not covered by backups or hot-spares. So, plan accordingly.
- geedee77 16y ago
  I agree with this 100%. It seems that an awful lot of companies like to pass the blame as soon as something unwanted happens. Yes, it may be the underlying fault of a service provider but, to your customers, you are the only company they deal with and it's your reputation that will be impacted no matter how many posts you put on your site and all over the internet. Using service companies does not excuse poor customer service.
  - hga 16y ago
    This could well turn out to be a fatal flaw for enterprise adoption of third party clouds: when corporate IT staff screw up like this or just messing with a control panel (check the AWS forums where too often a "It disappeared!" complaint has a reply of "It was deleted at thus and so time and date using the control panel."), are they going to accept the blame or shift it to the third party?
    - acdha 16y ago
      This already happens every day when the blame is passed through to Oracle, Gartner, etc. without any acknowledgement of the fact that, bad as the software truly is, it was selected and purchased at great cost and [usually] considerable delay by the people disclaiming responsibility. In other words, this will only be a fatal flaw if the business customers stop giving IT departments a free pass for being bad at technology.
- andrewvc 16y ago
  That's true, I agree 100%. I'd add to that though, if you're a small company sometimes you must recognize that a larger 3rd party can use scale to provide a more reliable service than you can.
- shahin 16y ago
  "You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!
  - tkaemming 16y ago
    Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)
    - shahin 16y ago
      The point is that they are not backing up your data they way they advertise it. The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Also if adding physical redundancy improves reliability only by 3.5% it means that they have different definition of the term.
      - jodrellblank 16y ago
        The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Which bit of "multiple failures of the underlying hardware components" leads you to think that a single point of failure still exists?
  - jacquesm 16y ago
    AWS promises you physical redundancy, what they don't offer you is an ironclad guarantee that nobody, customer included will ever screw up. So you need backups. It's really simple.
- patio11 16y ago
  You can't outsource your liability. Of course you can. That is the entire reason the insurance industry exists. More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seems to get discovered that the most inconvenient of times.)
  - brk 16y ago
    What is your "insurance" for hosting? I'm not aware of any SLAs that actually pay out anything near equal value to an outage they caused. If you're paying a hosting provider $500/mo for a sever that you generate an average of $1000/hr in web sales from and they have a 3 hour outage, you'll get an SLA credit for a couple of bucks applied to your next months service. The Internet is filled with news about hosting provider provided backups being unusable for a number of reasons at inopportune times. You should have a backup copy of your code/databases in your control, on a machine that is completely independent from whatever you are doing your production hosting on. You should have done a "warm metal" install and test of that code another server to be sure that you can recover operations in a reasonable amount of time (whatever is appropriate for your case). For your scale (based on your posts here, my assumption: Single developer, or developer with a couple of contractors; production site; 1-5K visits per month; non time-sensitive/mission-critical service.) you probably don't need high-availability auto-failover. But, you SHOULD have your DNS hosted separate from your hosting provider, you SHOULD have low TTL's, you SHOULD have a backup server in a warm state at some other provider, and you SHOULD know how to at least do a basic DNS update to redirect traffic over to a backup site that either runs the service or puts up a basic, friendly "OOPS, BRB" page. I've often thought of a startup that would basically human-automate these things for guys like you. You still wouldn't be 100% self-sufficient, but you would be able to outsource SOME of your entire reliance on 1 provider. You'd then have to have 2 tiers of total failure (your provider, and this service) to encounter complete down time.
  - jemfinch 16y ago
    > More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own EBS lets you create backup snapshots in S3 with one single command. The problem the (I think the epithet is warranted here) idiot who posted the original article had is that he didn't use it.
- jacquesm 16y ago
  Amen. It's quite amazing how many people think they can point the finger at some third party because they 'handed it off'. After all, if it's amazon then it's safe as the bank of England right? But even banks can burn down and even amazon does not owe you a penny if they lose your stuff. They'll say sorry, really nicely and maybe they'll offer you some store credit, comes in great when you have lots of free time, you can get some books. Maybe on how to back-up your data or so. But if you are halfway competent you smile and take that backup and you continue your business, because the value of your business outweighs amazons liability by a huge factor. Backing up is risk mitigation, if the risk to you is small then you can afford to live without backups, it means that the cost of backing up outweighed the cost of re-creation of the data. If the risk is larger then you probably should accept that, and go do something about it. Either way, the responsibility is yours.
mml 16y ago
Odd, my car lost my coffee when I put it on the roof on the way to work. Good thing there was backup coffee at the office.
- kiujhygfghjk 16y ago
  Do you also have RAID (Redundant Array Independant Donuts) ?
imp 16y ago
...because I didn't have my own backup."
garnaat 16y ago
There was a pretty lively exchange on twitter last night regarding this. I strongly disagree with the AWS forum poster. EBS does not suck. In fact, EBS and other services from AWS and Rackspace provide the building blocks to allow you to construct incredibly scalable, available systems. However, you have to accept that when you use IaaS you are taking on some of the operational responsibility and you have to know what you are doing or find someone who does. If this user had been snapshotting regularly to S3, the worst thing they would have experienced is a couple of hours of downtime. All of their data would have been safe and easily recovered. They didn't do that and the worst case scenario that AWS clearly describes in it's docs (failure of MULTIPLE devices) happened. And it will happen again, someday. Accept that and accept that failure is a feature when systems are designed properly.
dogas 16y ago
Why is this on HN? AWS provides a great way to back up EBS volumes called snapshots. Snapshots only store the deltas from the previous snapshot, and all the work to create one is done by AWS, not the server it is attached to. This guy didn't read the docs and did not use AWS snapshots. It was the equivalent of not having a backup strategy for your local hard drive.
- mceachen 16y ago
  I agree--this is just noise. The comment in the parent post saying "I run datacenters, and things don't fail" should be crossposted to thedailywtf. EVERYTHING fails. Plan for it.
- bcl 16y ago
  At first I thought the same thing. But I think this is a useful discussion to have on HN, since not everyone here understands probabilities or the realities of 'cloud' services. Maybe by having this the top rated story for a bit it will make someone think twice about their virtual infrastructure and examine their assumptions for errors. Do it for the data ;)
- cemerick 16y ago
  I submitted it because stories like this are common, but the underlying issues and misunderstandings aren't often aired out as they have been nicely here. Lots of people don't understand what they are and are not getting with "cloud" providers, nevermind grokking specifics like MTBF stats (and similar).
tkaemming 16y ago
The Amazon EBS page states (which the author quotes): > As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives. Nowhere within that does it say 0.00% failure rate, and later in the page they even describe how to mitigate the risk of losing data due to disk failure using snapshots, mirrored across availability zones.
jwr 16y ago
> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.
- mechanical_fish 16y ago
  There are people in the world who do not understand probability. Like the OP: EBS is either reliable or not. You cannot be a little pregnant. Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages. They happen all the time, often before a woman realizes she is pregnant. Then there are the false pregnancies: http://en.wikipedia.org/wiki/False_pregnancy http://en.wikipedia.org/wiki/False_pregnancy And these are just the common edge cases.
  - astrodust 16y ago
    You can be "a little bit pregnant" when you're talking about large groups of people. Of the women in your country, how many are pregnant at any given time? It's probably "a little bit". Obviously someone is confusing a statistical sampling of 1 with the kind of volume Amazon must deal with. I find it informative that a rare event such as this is being given so much press. If this was more routine, this wouldn't be news. Amazon seems to be doing better than 0.1% based on that alone.
- mynameishere 16y ago
  I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?
  - kiujhygfghjk 16y ago
    Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!
hipsterelitist 16y ago
At least you didn't have to pay thousands of dollars to delete your data!
mattew 16y ago
Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?
- blantonl 16y ago
  there is not, however it is really easy to write a script to freeze the volume and execute a snapshot. # xfs_freeze -f /data # ec2-create-snapshot vol-###### # xfs_freeze -u /data
  - mattew 16y ago
    One of my clients has 2 windows instances. I will need to look at the windows tools to see if there is something equivalent there.
  - spudlyo 16y ago
    So that's an xfs feature, but is is this really necessary on a journaling filesystem? I guess for ext3 I'd replace ext_freeze with sync.
- shlomo 16y ago
  This article includes a script that will perform a consistent snapshot of an XFS-formatted EBS drive (in linux). http://alestic.com/2009/09/ec2-consistent-snapshot http://alestic.com/2009/09/ec2-consistent-snapshot Consistent snapshots are needed when the disk contains multiple files that must be kept in sync with each other - such as a database's data files and the journal. If your DB gets its disk snapshotted while these files are out of sync, the DB will need to recover the missing data by replaying the journal. Not all DB's can do this reliably for all types of changes (e.g. MySQL's MyISAM engine is fragile in this regard). So you need to make sure that the files in the snapshot all reflect a consistent state of the disk.
jrockway 16y ago
Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; three drive failures in a week!) My fatal mistake, BTW, was ordering from Newegg. Apparently they do not ship OEM drives correctly, and they are almost guaranteed to fail. I was a little suspicious when I saw a raw drive in a plastic shell with some packing peanuts around it. When I had the drives replaced, they did not come from the factory that way!
- X-Istence 16y ago
  I have bought hundreds of OEM drives from NewEgg, all of them have had the plastic shell around them, and so far I've seen a 2% failure rate. The plastic shell is the way the manufacturers ship them direct to NewEgg, that is not NewEgg's doing.
  - hga 16y ago
    No, the plastic shelled disks are packed in solid foam in boxes holding something like 20 disks. Companies like NewEgg and Amazon break those boxes but do not follow the manufacturer's requirements (e.g. for shipping a failed disk back to them) when they repack them for individual orders. Last time I bought disks ZipZoomFly properly repacked them by putting the shelled disks into individual foam boxes.
- barrkel 16y ago
  Drives generally come packed either in those plastic shells, or in anti-static wrap, and then packed into specially cut foam (foam with slots, like a very parallel toaster) in a box with foam at the bottom and foam at the top. If you're not buying the drives in bulk, you don't get the box with foam.
  - jrockway 16y ago
    Yeah, when I get the replacements from the drive manufacturer, that's how I receive them. But I've ordered pillows that are better packed than what I get from Newegg.
Bjoern 16y ago
There is a easy rule of thumb. Make a backup of your important stuff often and regularly no matter how many redundancies are in place (see Murphy's law). Right now? Yes, like really right now if you didn't. EDIT: Spelling
braindead_in 16y ago
Can this also happen to an S3 bucket? How do I backup an S3 bucket? Any ideas?
- garnaat 16y ago
  I suppose it's possible but S3 is designed for nine 9's wrt to durability. There are > 100 billion objects and I'm not aware of any being lost due to AWS fault.
  - garnaat 16y ago
    Actually, I'm wrong. It's eleven 9's. http://bit.ly/ageV9D http://bit.ly/ageV9D
- holdenc 16y ago
  Also curious to know of any S3 bucket failure stories out there. I would assume this is uncommon, but I also assumed that the EBS failure rate was much lower prior to reading this.
jasonkester 16y ago
This post happens about once a week on the Amazon forums. I've watched it play out dozens of times on the S3 and Cloudfront forums too, and every single time it turns out to be operator error. In this case, the guy didn't realize he needed to take snapshots of his volumes. It's not surprising, really, since the documentation isn't so great for AWS, and it's probably even more painful knowing that it would have been a single button click to back up his volume using Amazon's tools. But in the end, there's nothing to see here. Just like the guy who wakes up in the morning to find all his S3 files mysteriously gone (after he 'renamed' his bucked the previous night by dropping and recreating it), it always turns out to be the user shooting himself in the foot. And in the cases when Amazon actually does something wrong, they're always on top of it immediately and back with a public explanation within hours. (from my experience)
- wmf 16y ago
  OTOH, if operators keep making the same mistake over and over again, maybe the UI should be changed. I think there's a fundamental mismatch in the EC2 control panel because it looks like anyone should be able to use it, but you have to be a competent sysadmin to use it safely.
  - wmf 16y ago
    Reuven Cohen makes a more detailed version of this point: "[Amazon] expect a certain level of knowledge of both system administration as well as how AWS itself has been designed to be used. Newbies need not apply or should use at you're own risk. Which isn't all that clear to a new user, who hears that cloud computing is safe and the answer to all your problems. ... You need the late adopters for the real revenue opportunities, but these same late adopters require a different more gentle kind of cloud service... As IaaS matures it is becoming obvious that the "Über Geek" developers who first adopted the service is not where the long tail revenue opportunities are." http://www.elasticvapor.com/2010/05/failure-as-service.html http://www.elasticvapor.com/2010/05/failure-as-service.html
mark_l_watson 16y ago
Sounds like he did not make S3 snapshots of his EBS volumes. Ouch. I feel very confident about the robustness of data that I store on AWS because I can make an S3 snapshot, and recover from that snapshot on a fresh EC2 to test the backup. BTW, I changed the way I use AWS: now I always make bootable EBS images, increasing the size > the 10 GB limit so I snapshot my OS setup and data and apps all at the same time.
rodh257 16y ago
another reason to http://lookafteryourdata.com http://lookafteryourdata.com
vl 16y ago
Most people that don't have extended experience with large-scale data stores do not understand basic principle: redundancy decreases probability of data loss, but it never eliminates it completely. All massive data stores slowly bleed data, it's just they bleed it so slow that it's acceptable for most scenarios. In case of this specific example, once number of users is large enough, there always be somebody who lost their volume. To illustrate this: think about a-la-GFS randomly triplicating data store on 1000 nodes. Once enough data is put in (lets say 100M blobs), there always be blob unique to any given triplet. In other words simultaneous loss of any 3 nodes out of 1000 will always result in data loss. (Simultaneous is in the sense "faster than time to detect failure and recover"). Of course failures are not limited to node loss, but there is corruption in transit, hard drive loss, bad sectors, rack-level failures. As the volume of the data and number of nodes grows it all adds up, so even if for each particular blob mean time to data loss is astronomically high, probability to loose some blob on any given day is very real.

111 points by cemerick 16y ago

brk 16y ago

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a certain minimal amount of scale). But you DO need to have someone who is thinking about backups and availability, and a valid solution is not to assume that the smart folks at Amazon or Rackspace or any other hosting provider are going to be completely and consistently working with your best interests and uptime in mind. EVRYTHING fails at some point. Every server, every generator, every upstream connection, every hosting provider big or small. And in this case I mean fail as in goes dark for some period of time not covered by backups or hot-spares. So, plan accordingly.

geedee77 16y ago
I agree with this 100%. It seems that an awful lot of companies like to pass the blame as soon as something unwanted happens. Yes, it may be the underlying fault of a service provider but, to your customers, you are the only company they deal with and it's your reputation that will be impacted no matter how many posts you put on your site and all over the internet. Using service companies does not excuse poor customer service.
- hga 16y ago
  This could well turn out to be a fatal flaw for enterprise adoption of third party clouds: when corporate IT staff screw up like this or just messing with a control panel (check the AWS forums where too often a "It disappeared!" complaint has a reply of "It was deleted at thus and so time and date using the control panel."), are they going to accept the blame or shift it to the third party?
  - acdha 16y ago
    This already happens every day when the blame is passed through to Oracle, Gartner, etc. without any acknowledgement of the fact that, bad as the software truly is, it was selected and purchased at great cost and [usually] considerable delay by the people disclaiming responsibility. In other words, this will only be a fatal flaw if the business customers stop giving IT departments a free pass for being bad at technology.
andrewvc 16y ago
That's true, I agree 100%. I'd add to that though, if you're a small company sometimes you must recognize that a larger 3rd party can use scale to provide a more reliable service than you can.
shahin 16y ago
"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!
- tkaemming 16y ago
  Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)
  - shahin 16y ago
    The point is that they are not backing up your data they way they advertise it. The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Also if adding physical redundancy improves reliability only by 3.5% it means that they have different definition of the term.
    - jodrellblank 16y ago
      The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Which bit of "multiple failures of the underlying hardware components" leads you to think that a single point of failure still exists?
- jacquesm 16y ago
  AWS promises you physical redundancy, what they don't offer you is an ironclad guarantee that nobody, customer included will ever screw up. So you need backups. It's really simple.
patio11 16y ago
You can't outsource your liability. Of course you can. That is the entire reason the insurance industry exists. More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seems to get discovered that the most inconvenient of times.)
- brk 16y ago
  What is your "insurance" for hosting? I'm not aware of any SLAs that actually pay out anything near equal value to an outage they caused. If you're paying a hosting provider $500/mo for a sever that you generate an average of $1000/hr in web sales from and they have a 3 hour outage, you'll get an SLA credit for a couple of bucks applied to your next months service. The Internet is filled with news about hosting provider provided backups being unusable for a number of reasons at inopportune times. You should have a backup copy of your code/databases in your control, on a machine that is completely independent from whatever you are doing your production hosting on. You should have done a "warm metal" install and test of that code another server to be sure that you can recover operations in a reasonable amount of time (whatever is appropriate for your case). For your scale (based on your posts here, my assumption: Single developer, or developer with a couple of contractors; production site; 1-5K visits per month; non time-sensitive/mission-critical service.) you probably don't need high-availability auto-failover. But, you SHOULD have your DNS hosted separate from your hosting provider, you SHOULD have low TTL's, you SHOULD have a backup server in a warm state at some other provider, and you SHOULD know how to at least do a basic DNS update to redirect traffic over to a backup site that either runs the service or puts up a basic, friendly "OOPS, BRB" page. I've often thought of a startup that would basically human-automate these things for guys like you. You still wouldn't be 100% self-sufficient, but you would be able to outsource SOME of your entire reliance on 1 provider. You'd then have to have 2 tiers of total failure (your provider, and this service) to encounter complete down time.
- jemfinch 16y ago
  > More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own EBS lets you create backup snapshots in S3 with one single command. The problem the (I think the epithet is warranted here) idiot who posted the original article had is that he didn't use it.
jacquesm 16y ago
Amen. It's quite amazing how many people think they can point the finger at some third party because they 'handed it off'. After all, if it's amazon then it's safe as the bank of England right? But even banks can burn down and even amazon does not owe you a penny if they lose your stuff. They'll say sorry, really nicely and maybe they'll offer you some store credit, comes in great when you have lots of free time, you can get some books. Maybe on how to back-up your data or so. But if you are halfway competent you smile and take that backup and you continue your business, because the value of your business outweighs amazons liability by a huge factor. Backing up is risk mitigation, if the risk to you is small then you can afford to live without backups, it means that the cost of backing up outweighed the cost of re-creation of the data. If the risk is larger then you probably should accept that, and go do something about it. Either way, the responsibility is yours.

mml 16y ago

Odd, my car lost my coffee when I put it on the roof on the way to work. Good thing there was backup coffee at the office.

kiujhygfghjk 16y ago
Do you also have RAID (Redundant Array Independant Donuts) ?

imp 16y ago

...because I didn't have my own backup."

garnaat 16y ago

There was a pretty lively exchange on twitter last night regarding this. I strongly disagree with the AWS forum poster. EBS does not suck. In fact, EBS and other services from AWS and Rackspace provide the building blocks to allow you to construct incredibly scalable, available systems. However, you have to accept that when you use IaaS you are taking on some of the operational responsibility and you have to know what you are doing or find someone who does. If this user had been snapshotting regularly to S3, the worst thing they would have experienced is a couple of hours of downtime. All of their data would have been safe and easily recovered. They didn't do that and the worst case scenario that AWS clearly describes in it's docs (failure of MULTIPLE devices) happened. And it will happen again, someday. Accept that and accept that failure is a feature when systems are designed properly.

dogas 16y ago

Why is this on HN? AWS provides a great way to back up EBS volumes called snapshots. Snapshots only store the deltas from the previous snapshot, and all the work to create one is done by AWS, not the server it is attached to. This guy didn't read the docs and did not use AWS snapshots. It was the equivalent of not having a backup strategy for your local hard drive.

mceachen 16y ago
I agree--this is just noise. The comment in the parent post saying "I run datacenters, and things don't fail" should be crossposted to thedailywtf. EVERYTHING fails. Plan for it.
bcl 16y ago
At first I thought the same thing. But I think this is a useful discussion to have on HN, since not everyone here understands probabilities or the realities of 'cloud' services. Maybe by having this the top rated story for a bit it will make someone think twice about their virtual infrastructure and examine their assumptions for errors. Do it for the data ;)
cemerick 16y ago
I submitted it because stories like this are common, but the underlying issues and misunderstandings aren't often aired out as they have been nicely here. Lots of people don't understand what they are and are not getting with "cloud" providers, nevermind grokking specifics like MTBF stats (and similar).

tkaemming 16y ago

The Amazon EBS page states (which the author quotes): > As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives. Nowhere within that does it say 0.00% failure rate, and later in the page they even describe how to mitigate the risk of losing data due to disk failure using snapshots, mirrored across availability zones.

jwr 16y ago

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

mechanical_fish 16y ago
There are people in the world who do not understand probability. Like the OP: EBS is either reliable or not. You cannot be a little pregnant. Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages. They happen all the time, often before a woman realizes she is pregnant. Then there are the false pregnancies: http://en.wikipedia.org/wiki/False_pregnancy http://en.wikipedia.org/wiki/False_pregnancy And these are just the common edge cases.
- astrodust 16y ago
  You can be "a little bit pregnant" when you're talking about large groups of people. Of the women in your country, how many are pregnant at any given time? It's probably "a little bit". Obviously someone is confusing a statistical sampling of 1 with the kind of volume Amazon must deal with. I find it informative that a rare event such as this is being given so much press. If this was more routine, this wouldn't be news. Amazon seems to be doing better than 0.1% based on that alone.
mynameishere 16y ago
I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?
- kiujhygfghjk 16y ago
  Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!

hipsterelitist 16y ago

At least you didn't have to pay thousands of dollars to delete your data!

mattew 16y ago

Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?

blantonl 16y ago
there is not, however it is really easy to write a script to freeze the volume and execute a snapshot. # xfs_freeze -f /data # ec2-create-snapshot vol-###### # xfs_freeze -u /data
- mattew 16y ago
  One of my clients has 2 windows instances. I will need to look at the windows tools to see if there is something equivalent there.
- spudlyo 16y ago
  So that's an xfs feature, but is is this really necessary on a journaling filesystem? I guess for ext3 I'd replace ext_freeze with sync.
shlomo 16y ago
This article includes a script that will perform a consistent snapshot of an XFS-formatted EBS drive (in linux). http://alestic.com/2009/09/ec2-consistent-snapshot http://alestic.com/2009/09/ec2-consistent-snapshot Consistent snapshots are needed when the disk contains multiple files that must be kept in sync with each other - such as a database's data files and the journal. If your DB gets its disk snapshotted while these files are out of sync, the DB will need to recover the missing data by replaying the journal. Not all DB's can do this reliably for all types of changes (e.g. MySQL's MyISAM engine is fragile in this regard). So you need to make sure that the files in the snapshot all reflect a consistent state of the disk.

jrockway 16y ago

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; three drive failures in a week!) My fatal mistake, BTW, was ordering from Newegg. Apparently they do not ship OEM drives correctly, and they are almost guaranteed to fail. I was a little suspicious when I saw a raw drive in a plastic shell with some packing peanuts around it. When I had the drives replaced, they did not come from the factory that way!

X-Istence 16y ago
I have bought hundreds of OEM drives from NewEgg, all of them have had the plastic shell around them, and so far I've seen a 2% failure rate. The plastic shell is the way the manufacturers ship them direct to NewEgg, that is not NewEgg's doing.
- hga 16y ago
  No, the plastic shelled disks are packed in solid foam in boxes holding something like 20 disks. Companies like NewEgg and Amazon break those boxes but do not follow the manufacturer's requirements (e.g. for shipping a failed disk back to them) when they repack them for individual orders. Last time I bought disks ZipZoomFly properly repacked them by putting the shelled disks into individual foam boxes.
barrkel 16y ago
Drives generally come packed either in those plastic shells, or in anti-static wrap, and then packed into specially cut foam (foam with slots, like a very parallel toaster) in a box with foam at the bottom and foam at the top. If you're not buying the drives in bulk, you don't get the box with foam.
- jrockway 16y ago
  Yeah, when I get the replacements from the drive manufacturer, that's how I receive them. But I've ordered pillows that are better packed than what I get from Newegg.

Bjoern 16y ago

There is a easy rule of thumb. Make a backup of your important stuff often and regularly no matter how many redundancies are in place (see Murphy's law). Right now? Yes, like really right now if you didn't. EDIT: Spelling

braindead_in 16y ago

Can this also happen to an S3 bucket? How do I backup an S3 bucket? Any ideas?

garnaat 16y ago
I suppose it's possible but S3 is designed for nine 9's wrt to durability. There are > 100 billion objects and I'm not aware of any being lost due to AWS fault.
- garnaat 16y ago
  Actually, I'm wrong. It's eleven 9's. http://bit.ly/ageV9D http://bit.ly/ageV9D
holdenc 16y ago
Also curious to know of any S3 bucket failure stories out there. I would assume this is uncommon, but I also assumed that the EBS failure rate was much lower prior to reading this.

jasonkester 16y ago

This post happens about once a week on the Amazon forums. I've watched it play out dozens of times on the S3 and Cloudfront forums too, and every single time it turns out to be operator error. In this case, the guy didn't realize he needed to take snapshots of his volumes. It's not surprising, really, since the documentation isn't so great for AWS, and it's probably even more painful knowing that it would have been a single button click to back up his volume using Amazon's tools. But in the end, there's nothing to see here. Just like the guy who wakes up in the morning to find all his S3 files mysteriously gone (after he 'renamed' his bucked the previous night by dropping and recreating it), it always turns out to be the user shooting himself in the foot. And in the cases when Amazon actually does something wrong, they're always on top of it immediately and back with a public explanation within hours. (from my experience)

wmf 16y ago
OTOH, if operators keep making the same mistake over and over again, maybe the UI should be changed. I think there's a fundamental mismatch in the EC2 control panel because it looks like anyone should be able to use it, but you have to be a competent sysadmin to use it safely.
- wmf 16y ago
  Reuven Cohen makes a more detailed version of this point: "[Amazon] expect a certain level of knowledge of both system administration as well as how AWS itself has been designed to be used. Newbies need not apply or should use at you're own risk. Which isn't all that clear to a new user, who hears that cloud computing is safe and the answer to all your problems. ... You need the late adopters for the real revenue opportunities, but these same late adopters require a different more gentle kind of cloud service... As IaaS matures it is becoming obvious that the "Über Geek" developers who first adopted the service is not where the long tail revenue opportunities are." http://www.elasticvapor.com/2010/05/failure-as-service.html http://www.elasticvapor.com/2010/05/failure-as-service.html

mark_l_watson 16y ago

Sounds like he did not make S3 snapshots of his EBS volumes. Ouch. I feel very confident about the robustness of data that I store on AWS because I can make an S3 snapshot, and recover from that snapshot on a fresh EC2 to test the backup. BTW, I changed the way I use AWS: now I always make bootable EBS images, increasing the size > the 10 GB limit so I snapshot my OS setup and data and apps all at the same time.

rodh257 16y ago

another reason to http://lookafteryourdata.com http://lookafteryourdata.com

vl 16y ago

Most people that don't have extended experience with large-scale data stores do not understand basic principle: redundancy decreases probability of data loss, but it never eliminates it completely. All massive data stores slowly bleed data, it's just they bleed it so slow that it's acceptable for most scenarios. In case of this specific example, once number of users is large enough, there always be somebody who lost their volume. To illustrate this: think about a-la-GFS randomly triplicating data store on 1000 nodes. Once enough data is put in (lets say 100M blobs), there always be blob unique to any given triplet. In other words simultaneous loss of any 3 nodes out of 1000 will always result in data loss. (Simultaneous is in the sense "faster than time to detect failure and recover"). Of course failures are not limited to node loss, but there is corruption in transit, hard drive loss, bad sectors, rack-level failures. As the volume of the data and number of nodes grows it all adds up, so even if for each particular blob mean time to data loss is astronomically high, probability to loose some blob on any given day is very real.