7 ms·

Hold off on AWS Aurora and Postgres until further notice

67 points by sudhirj 9y ago

We've had a sudden crash of the underlying storage (apparently) on one of our production workloads - and because the crash is in the underlying storage the MultiAZ setting makes no difference. Neither the primary nor the replica work. The point in time restore can't complete a restore to the last available point, so this looks like data loss to me. Points a earlier in time work fine, so might be an issue with a recent patch or failure. I'm currently in conversation with support, will update when I have more information.

stocktech 9y ago
It's the friday before a holiday curse. I'm sorry for your loss.
acranox 9y ago
The cloud is not immune from failure. Moving to the cloud just moves the line of responsibility for who manages a service, but it does not remove the risks. Hardware and software fail. Amazon does a better job mitigating the damage from failures than me, but their still still fails. I'm sorry for your data loss. I hope the recovery isn't too painful. Live and learn.
- deleted 9y ago
  [deleted]
BallinBige 9y ago
Sorry
kurttheviking 9y ago
We've been using Postgres on RDS for years and were considering moving to Aurora. I am very curious to know more about the fix and planned remedy.
- sudhirj 9y ago
  Current suspects are GIN index problems, but nothing confirmed yet.
  - kurttheviking 9y ago
    Ugh, that's bad -- sorry to hear that. We GIN index several JSONB fields as well.
  - petergeoghegan 9y ago
    Per my remarks here, I think that there is an obscure bug in GIN vacuuming that the community has yet to identify: https://postgr.es/m/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aMGEKLKv95csQ@mail.gmail.com https://postgr.es/m/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aM... It's possible that this upstream PostgreSQL issue affected you. What is known for sure is that there was a bugfix for GIN, that just missed the latest round of point releases. See: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=3b2787e1f8f1eeeb6bd9565288ab210262705b56 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... This fix alone could have made all the difference here (though this is a guess). I assume you won't have this fix with Aurora just yet (not sure -- I don't work for Amazon). But, I'd mention this to AWS support if it seems at all relevant. Hopefully they'll get in touch with the community if that's what it is. Good luck
    - sudhirj 9y ago
      Thanks, will pass this on.
deckarep 9y ago
That’s a tough situation to be in. Hopefully Amazon is working on a solution for you. Cloud is definitely not bullet-proof and risks are always there. I think it generally still wins however because as Cloud solutions get better, with better SLAs...companies can get better at what they specialize and offer. At least that is the goal and how I understand it.
jaydestro 9y ago
sorry to hear this, good luck. always sucks on a fri.
nolite 9y ago
" because the crash is in the underlying storage the MultiAZ setting makes no difference." @OP - I don't understand this line... can you explain more?
- vhold 9y ago
  Aurora read replicas are just compute heads decoupled from the storage. They increase availability and read performance, but don't do anything for the reliability of the storage. https://aws.amazon.com/rds/aurora/faqs/ https://aws.amazon.com/rds/aurora/faqs/ "Each 10GB chunk of your database volume is replicated six ways, across three Availability Zones." So, if there has been a storage layer failure, most likely it was a software bug in Aurora itself.
tyingq 9y ago
Very interested in what Amazon's final position on this is. I assumed part of the value proposition was that they handled redundancy and recovery.
QuinnyPig 9y ago
For what it’s worth, this appears localized to your account. Sorry for your situation, though.
fjania 9y ago
I don't see anything on the status page. I know the status page lags an actual incident, but I'd imagine something would be up by now. Is this a widespread problem?
joelbondurant 9y ago
LOL, outsourcing a closed source database to the McDonald's of IT. #wcpgw
CaliforniaKarl 9y ago
Hello! Was this ever resolved? It seems like there was one update from the poster, but then nothing.
- sudhirj 9y ago
  Resolved, managed to get the data back, don't have a postmortem yet.
sudhirj 9y ago
The data has been recovered, no post mortem yet.
- mrep 9y ago
  That's lucky. Was your service down for the entire time until the data was recovered?
  - sudhirj 9y ago
    We moved to a new DB as a clean slate, took fresh orders, and then merged the old ones in as they became available. Was able to do that because all primary keys were uuids and the order history API was being served somewhere else.
    - mrep 9y ago
      How did you handle other tables like users and sessions?
      - sudhirj 9y ago
        This microservice didn't have them, but even they're all keyed by UUID, so it's possible to start a clean slate pretty fast. Filling in existing users later should also be easy, the problems will occur if and when users try to sign up again with the same key (email / FBid etc).

67 points by sudhirj 9y ago

stocktech 9y ago

It's the friday before a holiday curse. I'm sorry for your loss.

acranox 9y ago

The cloud is not immune from failure. Moving to the cloud just moves the line of responsibility for who manages a service, but it does not remove the risks. Hardware and software fail. Amazon does a better job mitigating the damage from failures than me, but their still still fails. I'm sorry for your data loss. I hope the recovery isn't too painful. Live and learn.

deleted 9y ago
[deleted]

BallinBige 9y ago

Sorry

kurttheviking 9y ago

We've been using Postgres on RDS for years and were considering moving to Aurora. I am very curious to know more about the fix and planned remedy.

sudhirj 9y ago
Current suspects are GIN index problems, but nothing confirmed yet.
- kurttheviking 9y ago
  Ugh, that's bad -- sorry to hear that. We GIN index several JSONB fields as well.
- petergeoghegan 9y ago
  Per my remarks here, I think that there is an obscure bug in GIN vacuuming that the community has yet to identify: https://postgr.es/m/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aMGEKLKv95csQ@mail.gmail.com https://postgr.es/m/CAH2-Wz=GTnAPzEEZqYELOv3h1Fxpo5xhMrBP6aM... It's possible that this upstream PostgreSQL issue affected you. What is known for sure is that there was a bugfix for GIN, that just missed the latest round of point releases. See: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=3b2787e1f8f1eeeb6bd9565288ab210262705b56 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... This fix alone could have made all the difference here (though this is a guess). I assume you won't have this fix with Aurora just yet (not sure -- I don't work for Amazon). But, I'd mention this to AWS support if it seems at all relevant. Hopefully they'll get in touch with the community if that's what it is. Good luck
  - sudhirj 9y ago
    Thanks, will pass this on.

deckarep 9y ago

That’s a tough situation to be in. Hopefully Amazon is working on a solution for you. Cloud is definitely not bullet-proof and risks are always there. I think it generally still wins however because as Cloud solutions get better, with better SLAs...companies can get better at what they specialize and offer. At least that is the goal and how I understand it.

jaydestro 9y ago

sorry to hear this, good luck. always sucks on a fri.

nolite 9y ago

" because the crash is in the underlying storage the MultiAZ setting makes no difference." @OP - I don't understand this line... can you explain more?

vhold 9y ago
Aurora read replicas are just compute heads decoupled from the storage. They increase availability and read performance, but don't do anything for the reliability of the storage. https://aws.amazon.com/rds/aurora/faqs/ https://aws.amazon.com/rds/aurora/faqs/ "Each 10GB chunk of your database volume is replicated six ways, across three Availability Zones." So, if there has been a storage layer failure, most likely it was a software bug in Aurora itself.

tyingq 9y ago

Very interested in what Amazon's final position on this is. I assumed part of the value proposition was that they handled redundancy and recovery.

QuinnyPig 9y ago

For what it’s worth, this appears localized to your account. Sorry for your situation, though.

fjania 9y ago

I don't see anything on the status page. I know the status page lags an actual incident, but I'd imagine something would be up by now. Is this a widespread problem?

joelbondurant 9y ago

LOL, outsourcing a closed source database to the McDonald's of IT. #wcpgw

CaliforniaKarl 9y ago

Hello! Was this ever resolved? It seems like there was one update from the poster, but then nothing.

sudhirj 9y ago
Resolved, managed to get the data back, don't have a postmortem yet.

sudhirj 9y ago

The data has been recovered, no post mortem yet.

mrep 9y ago
That's lucky. Was your service down for the entire time until the data was recovered?
- sudhirj 9y ago
  We moved to a new DB as a clean slate, took fresh orders, and then merged the old ones in as they became available. Was able to do that because all primary keys were uuids and the order history API was being served somewhere else.
  - mrep 9y ago
    How did you handle other tables like users and sessions?
    - sudhirj 9y ago
      This microservice didn't have them, but even they're all keyed by UUID, so it's possible to start a clean slate pretty fast. Filling in existing users later should also be easy, the problems will occur if and when users try to sign up again with the same key (email / FBid etc).