4 ms·

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident

by mkl95 3y ago

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really, really know what you are doing.

speedgoose 3y ago
I would also advice to have some backups available outside of your favourite cloud provider. Your cloud provider may fuckup, it has happen many times. You may encounter issues such as your account deleted or temporarily unavailable. Or the whole datacenter and many zones are on fire.
- jrockway 3y ago
  For example, the "delete" button on GCP deletes the backups along with the database you're deleting. I would highly recommend not relying on those backups for anything mission-critical. Transfering the backups to another cloud provider is essential if your risk model involves losing your cloud account somehow.
marcus_holmes 3y ago
We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.
- sofixa 3y ago
  That sounds unlike my experience with AWS RDS, and I don't think there would be many users of it if it were generally true. When was this and was it AWS?
  - marcus_holmes 3y ago
    Copied the email we got from Amazon verbatim: ____ Hello Marcus, Hope you are keeping well and safe. Thank you for reaching back out to me. I can completely understand that this is not the behavior you expected with regards to engine version upgrades with multi-AZ configuration. By design, when performing any version upgrade on a multi-AZ instance, the engine version is upgraded on the primary and the secondary at the same time resulting in both the primary and standby being unavailable. Unfortunately, it is not possible for the instance to failover in this scenario since the database level changes needs to be applied at the same time. As mentioned in my previously correspondence, you do have the ability to opt-out of auto minor version upgrades if this does not suit your requirements [1]. By doing this, minor version updates will no longer be applied automatically and will have to be manually maintained. As additional information, note that multi-AZ deployment does reduce the downtime in certain activities, however database engine version upgrade does not come under this. Multi-AZ deployments helps RDS automatically perform a failover in the event of any of the following: 1. An Availability Zone outage 2. The primary DB instance fails 3. The DB instance's server type is changed 4. The operating system of the DB instance is undergoing software patching 5. A manual failover of the DB instance was initiated using Reboot with failover For more information on High Availability (Multi-AZ) deployments, kindly refer the below document for your reference [2]. I sincerely hope I was able to address your concerns Marcus. Please don't hesitate to reach back out to me and I will be happy to assist further as best I can. Thank you for your time. Wishing you a great weekend ahead, and stay safe. References: [1] Automatically upgrading the minor engine version https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Upgrading.html#USER_UpgradeDBInstance.Upgrading.AutoMinorVersionUpgrades https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_... [2] Failover Process for Amazon RDS http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html#Concepts.MultiAZ.Failover http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep... We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.
- kikimora 3y ago
  AFAIK if there is a binary replication between two instances then you cannot upgrade without downtime.
- evanelias 3y ago
  Was it a major version upgrade? Postgres WAL replication is a physical replication stream which doesn't work across major versions. I don't know much about RDS Postgres but I'd assume this is the reason. Logical replication can solve this, but has some important limitations (e.g. no replication of DDL), so it's understandable that RDS can't leverage this generically and automatically. I would expect that RDS MySQL doesn't have this problem since MySQL's built-in replication is logical rather than physical.

by mkl95 3y ago

speedgoose 3y ago

I would also advice to have some backups available outside of your favourite cloud provider. Your cloud provider may fuckup, it has happen many times. You may encounter issues such as your account deleted or temporarily unavailable. Or the whole datacenter and many zones are on fire.

jrockway 3y ago
For example, the "delete" button on GCP deletes the backups along with the database you're deleting. I would highly recommend not relying on those backups for anything mission-critical. Transfering the backups to another cloud provider is essential if your risk model involves losing your cloud account somehow.

marcus_holmes 3y ago

We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.

sofixa 3y ago
That sounds unlike my experience with AWS RDS, and I don't think there would be many users of it if it were generally true. When was this and was it AWS?
- marcus_holmes 3y ago
  Copied the email we got from Amazon verbatim: ____ Hello Marcus, Hope you are keeping well and safe. Thank you for reaching back out to me. I can completely understand that this is not the behavior you expected with regards to engine version upgrades with multi-AZ configuration. By design, when performing any version upgrade on a multi-AZ instance, the engine version is upgraded on the primary and the secondary at the same time resulting in both the primary and standby being unavailable. Unfortunately, it is not possible for the instance to failover in this scenario since the database level changes needs to be applied at the same time. As mentioned in my previously correspondence, you do have the ability to opt-out of auto minor version upgrades if this does not suit your requirements [1]. By doing this, minor version updates will no longer be applied automatically and will have to be manually maintained. As additional information, note that multi-AZ deployment does reduce the downtime in certain activities, however database engine version upgrade does not come under this. Multi-AZ deployments helps RDS automatically perform a failover in the event of any of the following: 1. An Availability Zone outage 2. The primary DB instance fails 3. The DB instance's server type is changed 4. The operating system of the DB instance is undergoing software patching 5. A manual failover of the DB instance was initiated using Reboot with failover For more information on High Availability (Multi-AZ) deployments, kindly refer the below document for your reference [2]. I sincerely hope I was able to address your concerns Marcus. Please don't hesitate to reach back out to me and I will be happy to assist further as best I can. Thank you for your time. Wishing you a great weekend ahead, and stay safe. References: [1] Automatically upgrading the minor engine version https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Upgrading.html#USER_UpgradeDBInstance.Upgrading.AutoMinorVersionUpgrades https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_... [2] Failover Process for Amazon RDS http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html#Concepts.MultiAZ.Failover http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep... We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.
kikimora 3y ago
AFAIK if there is a binary replication between two instances then you cannot upgrade without downtime.
evanelias 3y ago
Was it a major version upgrade? Postgres WAL replication is a physical replication stream which doesn't work across major versions. I don't know much about RDS Postgres but I'd assume this is the reason. Logical replication can solve this, but has some important limitations (e.g. no replication of DDL), so it's understandable that RDS can't leverage this generically and automatically. I would expect that RDS MySQL doesn't have this problem since MySQL's built-in replication is logical rather than physical.