6 ms·

Amazon RDS failure - data has been lost

82 points by akhkharu 14y ago

Our RDS instance is in "failure" state after 8 hours of downtime. Have to restore from point in time backup which does not have actual data. Amazon says: Jun 15, 4:03 AM PDT The RDS service is now operating normally. All affected Multi-AZ RDS instances operated normally throughout the power event after failing over. We were able to recover many Single AZ instances successfully, but storage volumes attached to some Single-AZ instances could not be restored, resulting in those instances being placed in Storage failure mode. Customers with automated backups turned on for an affected database instance have the option of initiating a Point-in-Time Restore operation. This will launch a new database instance using a backup of the affected database instance from before the event. To do this, follow these steps: 1) Log into the AWS Management console 2) Access the RDS tab, and select DB Instances on the left-side navigation 3) Select the affected database instance 4) Click on the "Restore to Point in Time" button 5) Select "Use Latest Restorable Time 6) Select a DB instance class that is at least the same size as the original DB instance 7) Make sure No Preference is selected for Availability Zone 8) Launch DB Instance and connect your application We will be following up here with the root cause of this event.

justincormack 14y ago
Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.
- akhkharu 14y ago
  As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.
  - nevinera 14y ago
    >Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.
    - akhkharu 14y ago
      No, I understand that replication is the double cost. I meant cloud solution should not have storage failures which causes data loss.
      - kiallmacinnes 14y ago
        I disagree. Amazon simply gave you the choice over what level of reliability you require. Would you have preferred they only offered Multi-AZ databases? (and in the process doubled your development, staging and QA environment costs..)
      - nevinera 14y ago
        And google should give us all ponies. Would you mind defending your expectation that other people and companies will give you extra services for free? Amazon has been very clear on the expected failure rate of ebs volumes and the attendant rds failures. If you want data safety, multi-az deployment offers it.
    - dspillett 14y ago
      A lot of people do (incorrectly) assume this sort of thing, even in cases when it would only take cursory understanding/research to spot the limitations a given service has and where it might fail in a disaster recovery situation. No matter how good you think a given "cloud" solution is, no matter how much too big to fail you thing the company responsible is, you should make use of the redundancy options they provide and make sure you have reasonable backups elsewhere too (local to you or on another completely separate remote service). This is what made me dismiss Google's App Engine when it first turned up (I've not looked for some time, they may have addressed this concern long ago): there was no easy way to backup all your data to another location/service and the not-so-easy ways would probably all end up costing a fair bit in bandwidth charges.
    - seanp2k2 14y ago
      This kinda highlights the problem with "cloud"...many people, even engineers, don't really understand what they're getting into. At least with a single server, you know what you're getting, and you have only yourself to blame if you didn't plan for a typical, known, documented failure mode.
  - acdha 14y ago
    > As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any server can go down. The very modest increase for a multi-AZ setup buys you real, meaningful improvements as you just learned. I'm sorry that you had to learn a lesson the hard way but there's a reason why AWS recommends a multi-AZ deployment for failover and it's not revenue. The next step up would require you having multiple widely separated servers, which is where you really start talking about large amounts of money because you're talking about non-trivial engineering and taking on the operational overhead of 24x7 support.
    - akhkharu 14y ago
      Yes, networking issue which brought down even multi-az deployments. I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.
      - ceejayoz 14y ago
        "Brought down" and "Brought down and lost data" are very different severities. Many businesses can handle occasional downtime as long as data's not disappearing into the ether.
      - acdha 14y ago
        > Yes, networking issue which brought down even multi-az deployments. You might want to learn more about this before making business decisions on it. RDS was completely unaffected, as were all of my EC2 servers. They didn't receive any traffic from the internet but the systems were running fine throughout the brief outage interval. A quick Google search will reveal that this is not uncommon for any hosting setup - data centers have lost network connections, routers can fail or be misconfigured, etc. - which is why anyone with serious uptime requirements has multiple widely separated data centers. Using AWS doesn't magically remove the need to avoid single points of failure in your system design. > Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable. I hope you have a good ops team and extra engineering resources; otherwise you'll learn very quickly that dedicated servers have the same failure modes. So far we're at ~18 minutes of AWS downtime this year - that's not going to be easy to beat.
- snorkel 14y ago
  Sure, just duplicate your entire stack in at least two zones and replicate all data in realtime, and then you just need to convince your boss/investors/yourself that spending 2X on hosting is worth it. Once you add up the costs you soon realize that risking several hours of downtime once per year is more acceptable than doubling hosting costs. For anyone who is not hosting air traffic control or banking systems it's really not worth it. Seriously, if your web service is hosting social brain farts or selling cup holders, and not landing a space shuttles, then it can be offline a for few hours per year.
purephase 14y ago
I'm not sure I understand the "which does not have actual data" part of your statement. Could you explain that a bit more?
- akhkharu 14y ago
  Point in Time backup was created before the actual failure and does not contain the latest data (~ 1 hour).
  - philjohn 14y ago
    Surely you realised that is the case with a point-in-time backup? If you absolutely cannot lose data then as others have said, Multi AZ is required, or, at the very least, have a transaction log that you can replay (once again, hosted somewhere else).
- horatiumocian 14y ago
  Probably he means that the data from the backup is not up-to-date (i.e. not actual).
PaulHoule 14y ago
If you had a database running on a dedi you could get trashed by a server failure too. Good backups are the best defense.
- akhkharu 14y ago
  Yeah, but I have more control over it to prevent such failures.
- shiftpgdn 14y ago
  To me this is a fallacious argument. Dedicated servers are wildly cheaper than RDS/AWS. Isn't that the whole point of AWS? To have a team of experts managing your hosting to prevent a failure like this?
  - PaulHoule 14y ago
    I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway. The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data out, but jeese...) One time I made a ticket and somehow my record in the trouble ticket system got screwed up and I couldn't put more tickets in. Some wizard fixed that in the SQL monitor after I talked to 3 other people who had no idea this could happen. As for costs it's not so simple. I've got a processing job I run each week that costs $6 of CPU time because I pay just for what I need. I wouldnt want to run it on a dedi because it's a beast.
    - huggyface 14y ago
      Your anecdote is interesting, but not relevant. The parent rightly observes that the whole point of a service like RDS is that you don't have to babysit it. If you still do then it's all of the disadvantages of your own box, plus more disadvantages.
EwanToo 14y ago
RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive image will disappear at any moment - as it did here. Beyond that, any time you're running a database, no matter who the provider is, if you're not doing backups every day or hour, then you're not doing things right.
- JohnHaugeland 14y ago
  They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it is for me, a car industry outsider, to trust the claims that Chevrolet makes about my car. Do you know how to properly quality evaluate everything in your life? Hope you don't need a doctor, lawyer, or plumber soon.
  - efsavage 14y ago
    I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ http://aws.amazon.com/ebs/. What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. Which to any engineer, "real" or otherwise, should be a pretty strong signal that something bad will happen.
    - andrewvc 14y ago
      Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.
debacle 14y ago
But but...the cloud.
mschalle 14y ago
Always assume Murphy's law will hold, regardless of what service provider you use. If you were running your own database, you surely would have had rigorous backups because the responsibility was on you. Assume that if a service can fail, it will. If data can be lost, it will be. Then, plan accordingly. EDIT: grammar
bananashake 14y ago
Why do you think the "Restore to Point in Time" failed to work? That puzzles me the most in this catastrophe and no has addressed it. In theory with Point-in-Time restoration you should not lose data from a failure on just the storage where the InnoDB is stored.

82 points by akhkharu 14y ago

justincormack 14y ago

Use multi AZ then, which performed as expected. There have been so many warnings about single AZ that you would hope people get it by now.

akhkharu 14y ago
As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data.
- nevinera 14y ago
  >Cloud solutions even with single AZ should not loss data. You mean you think all cloud db solutions should implement replication for you? There aren't very many backup solutions that never lose any data.
  - akhkharu 14y ago
    No, I understand that replication is the double cost. I meant cloud solution should not have storage failures which causes data loss.
    - kiallmacinnes 14y ago
      I disagree. Amazon simply gave you the choice over what level of reliability you require. Would you have preferred they only offered Multi-AZ databases? (and in the process doubled your development, staging and QA environment costs..)
    - nevinera 14y ago
      And google should give us all ponies. Would you mind defending your expectation that other people and companies will give you extra services for free? Amazon has been very clear on the expected failure rate of ebs volumes and the attendant rds failures. If you want data safety, multi-az deployment offers it.
  - dspillett 14y ago
    A lot of people do (incorrectly) assume this sort of thing, even in cases when it would only take cursory understanding/research to spot the limitations a given service has and where it might fail in a disaster recovery situation. No matter how good you think a given "cloud" solution is, no matter how much too big to fail you thing the company responsible is, you should make use of the redundancy options they provide and make sure you have reasonable backups elsewhere too (local to you or on another completely separate remote service). This is what made me dismiss Google's App Engine when it first turned up (I've not looked for some time, they may have addressed this concern long ago): there was no easy way to backup all your data to another location/service and the not-so-easy ways would probably all end up costing a fair bit in bandwidth charges.
  - seanp2k2 14y ago
    This kinda highlights the problem with "cloud"...many people, even engineers, don't really understand what they're getting into. At least with a single server, you know what you're getting, and you have only yourself to blame if you didn't plan for a typical, known, documented failure mode.
- acdha 14y ago
  > As far as I remember, previous failure which happen with Amazon earlier this year have also affected Multi-AZ deployments too. Which failure? The networking issue which had nothing to do with RDS and left your data unaffected? > Anyway, I don't think that we are ready to invest large amount of money on Multi-AZ deployments to the doubtful reliability. Cloud solutions even with single AZ should not loss data. Any server can go down. The very modest increase for a multi-AZ setup buys you real, meaningful improvements as you just learned. I'm sorry that you had to learn a lesson the hard way but there's a reason why AWS recommends a multi-AZ deployment for failover and it's not revenue. The next step up would require you having multiple widely separated servers, which is where you really start talking about large amounts of money because you're talking about non-trivial engineering and taking on the operational overhead of 24x7 support.
  - akhkharu 14y ago
    Yes, networking issue which brought down even multi-az deployments. I don't want to setup highly available fault tolerant systems, I just want a good level of reliability of a service provider. Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable.
    - ceejayoz 14y ago
      "Brought down" and "Brought down and lost data" are very different severities. Many businesses can handle occasional downtime as long as data's not disappearing into the ether.
    - acdha 14y ago
      > Yes, networking issue which brought down even multi-az deployments. You might want to learn more about this before making business decisions on it. RDS was completely unaffected, as were all of my EC2 servers. They didn't receive any traffic from the internet but the systems were running fine throughout the brief outage interval. A quick Google search will reveal that this is not uncommon for any hosting setup - data centers have lost network connections, routers can fail or be misconfigured, etc. - which is why anyone with serious uptime requirements has multiple widely separated data centers. Using AWS doesn't magically remove the need to avoid single points of failure in your system design. > Probably, we will migrate to the dedicated servers out of Amazon soon. It will be harder to maintain, but cheaper and, as practice shows, more reliable. I hope you have a good ops team and extra engineering resources; otherwise you'll learn very quickly that dedicated servers have the same failure modes. So far we're at ~18 minutes of AWS downtime this year - that's not going to be easy to beat.
snorkel 14y ago
Sure, just duplicate your entire stack in at least two zones and replicate all data in realtime, and then you just need to convince your boss/investors/yourself that spending 2X on hosting is worth it. Once you add up the costs you soon realize that risking several hours of downtime once per year is more acceptable than doubling hosting costs. For anyone who is not hosting air traffic control or banking systems it's really not worth it. Seriously, if your web service is hosting social brain farts or selling cup holders, and not landing a space shuttles, then it can be offline a for few hours per year.

purephase 14y ago

I'm not sure I understand the "which does not have actual data" part of your statement. Could you explain that a bit more?

akhkharu 14y ago
Point in Time backup was created before the actual failure and does not contain the latest data (~ 1 hour).
- philjohn 14y ago
  Surely you realised that is the case with a point-in-time backup? If you absolutely cannot lose data then as others have said, Multi AZ is required, or, at the very least, have a transaction log that you can replay (once again, hosted somewhere else).
horatiumocian 14y ago
Probably he means that the data from the backup is not up-to-date (i.e. not actual).

PaulHoule 14y ago

If you had a database running on a dedi you could get trashed by a server failure too. Good backups are the best defense.

akhkharu 14y ago
Yeah, but I have more control over it to prevent such failures.
shiftpgdn 14y ago
To me this is a fallacious argument. Dedicated servers are wildly cheaper than RDS/AWS. Isn't that the whole point of AWS? To have a team of experts managing your hosting to prevent a failure like this?
- PaulHoule 14y ago
  I find it's very expensive to talk with salespeople to get my dedis configured properly. Particularly when they screw it up anyway. The reason I moved to AWS was because when I added a new (cheap) hard drive to my dedi, they didn't put a partition table on the disk. When the machine rebooted, the superblock got overwritten and I lost access to the file system. (I did manage to recreate the superblock and get the data out, but jeese...) One time I made a ticket and somehow my record in the trouble ticket system got screwed up and I couldn't put more tickets in. Some wizard fixed that in the SQL monitor after I talked to 3 other people who had no idea this could happen. As for costs it's not so simple. I've got a processing job I run each week that costs $6 of CPU time because I pay just for what I need. I wouldnt want to run it on a dedi because it's a beast.
  - huggyface 14y ago
    Your anecdote is interesting, but not relevant. The parent rightly observes that the whole point of a service like RDS is that you don't have to babysit it. If you still do then it's all of the disadvantages of your own box, plus more disadvantages.

EwanToo 14y ago

RDS should not have lost data, and if I were a user of it, I'd be annoyed too. At the same time, if you've not spotted by now that EBS (elastic block storage, which powers RDS) is not reliable and not to be trusted, then you have to look at yourself too. EBS is by far the worst product AWS offer, you simply should not use it without a very good reason, and if you do need to use it, you have to assume any given drive image will disappear at any moment - as it did here. Beyond that, any time you're running a database, no matter who the provider is, if you're not doing backups every day or hour, then you're not doing things right.

JohnHaugeland 14y ago
They go to great effort to tell prospective customers that it's extremely reliable, providing claims of obscene numbers of nines. A real engineer should know better, but otherwise, it's people trusting what a major company claims. If it was a fly by night organization I would totally agree with you, but Amazon is a major multinational. It seems to me as reasonable for an outsider to trust the claims they make as it is for me, a car industry outsider, to trust the claims that Chevrolet makes about my car. Do you know how to properly quality evaluate everything in your life? Hope you don't need a doctor, lawyer, or plumber soon.
- efsavage 14y ago
  I don't remember ever seeing "obscene numbers of nines" claimed by Amazon. For S3, yes, but not EBS. The '9' character doesn't even appear once on http://aws.amazon.com/ebs/ http://aws.amazon.com/ebs/. What it does say is this: "As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume. Which to any engineer, "real" or otherwise, should be a pretty strong signal that something bad will happen.
  - andrewvc 14y ago
    Yep, I've been telling this to people for years, and they keep sticking their fingers in their ears because they think they're entitled to magic uptime by amazon. The key to EBS is frequent S3 snapshots. The reliability issues of EBS only appears to the delta since your last S3 snapshot. Still... people use EBS w/o frequent S3 backups, amazingly.

debacle 14y ago

But but...the cloud.

mschalle 14y ago

Always assume Murphy's law will hold, regardless of what service provider you use. If you were running your own database, you surely would have had rigorous backups because the responsibility was on you. Assume that if a service can fail, it will. If data can be lost, it will be. Then, plan accordingly. EDIT: grammar

bananashake 14y ago

Why do you think the "Restore to Point in Time" failed to work? That puzzles me the most in this catastrophe and no has addressed it. In theory with Point-in-Time restoration you should not lose data from a failure on just the storage where the InnoDB is stored.