3 ms·

> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery metho

> k8s should run already with automatic node upgrades This is difficult to impossible to do with databases; even if your database has a built-in recovery method for when a primary is taken offline, in such a way that allows for zero-downtime in theory, the reality is that such mechanisms depend on the secondary staying online until the failover mechanism is complete. If you turn over control of node upgrades to the cluster provider, the node under the secondary will get rebooted in the middle of the failover process, and you will get downtime at best, data loss at worst. What kubernetes teaches us is that databases aren't tied to the literal VM they're running on (which is now cattle), but rather on the availability of that node. If you run databases on kubernetes, you need to have a mechanism to slow down node upgrades. Source: helped run hundreds of Elasticsearch and Kafka nodes on kubernetes in production at one point in my career

AtlasBarfed 4y ago
Online lossless zero downtime upgrades? I've done it with Cassandra...and yeah Kafka can do it I've heard. But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready. Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.
- anecdotal1 4y ago
  Add a new RDS replica, wait for it to sync, promote it to master? Zero downtime achieved
  - AtlasBarfed 4y ago
    I didn't do the upgrade, I'm not a postgres MySQL person, but the best they could do was a third party tool that dropped it to a couple minutes.
redrove 4y ago
Your claim about needing the primary online to failiover to the secondary is untrue, at least not for all Postgres operators. Cloudnative PG rebuilds the secondary during failiover from the streamed WAL to an S3 endpoint. No primary needed.
GauntletWizard 4y ago
Kubernetes does have this capability - Pod Disruption Budgets. They're underutilized and under tested, but at least the default cluster autoscaler respects them and will avoid destroying nodes that would break that constraint.
- smarterclayton 4y ago
  They are pretty well tested as of today (now that multiple vendors respect it during node upgrade), but now they’re relatively under featured for the next set of problems: 1. No way to signal that the workload is ready to accept traffic but not ready for disruption (such as an async background refill) 2. No way to provide backpressure over a time window, vs just instantaneous (ie during a rolling node upgrade PDB doesn’t prevent you from being moved multiple times) 3. No way for an admin and workload owner to coordinate on the amount of disruption a particular disruption will cause and discriminate between light and heavy (restart on node, recreate on node, move across node) But yes, everyone with a nominally HA workload should have a PDB.

AtlasBarfed 4y ago

Online lossless zero downtime upgrades? I've done it with Cassandra...and yeah Kafka can do it I've heard. But those can be 30 hour operations even with you ducks in a row, and you better have backup strategies ready. Fun story, Amazon said rds would be always be zero downtime upgrades. But then came a major version upgrade and .... Surprise it wasn't.

anecdotal1 4y ago
Add a new RDS replica, wait for it to sync, promote it to master? Zero downtime achieved
- AtlasBarfed 4y ago
  I didn't do the upgrade, I'm not a postgres MySQL person, but the best they could do was a third party tool that dropped it to a couple minutes.

redrove 4y ago

Your claim about needing the primary online to failiover to the secondary is untrue, at least not for all Postgres operators. Cloudnative PG rebuilds the secondary during failiover from the streamed WAL to an S3 endpoint. No primary needed.

GauntletWizard 4y ago

Kubernetes does have this capability - Pod Disruption Budgets. They're underutilized and under tested, but at least the default cluster autoscaler respects them and will avoid destroying nodes that would break that constraint.

smarterclayton 4y ago
They are pretty well tested as of today (now that multiple vendors respect it during node upgrade), but now they’re relatively under featured for the next set of problems: 1. No way to signal that the workload is ready to accept traffic but not ready for disruption (such as an async background refill) 2. No way to provide backpressure over a time window, vs just instantaneous (ie during a rolling node upgrade PDB doesn’t prevent you from being moved multiple times) 3. No way for an admin and workload owner to coordinate on the amount of disruption a particular disruption will cause and discriminate between light and heavy (restart on node, recreate on node, move across node) But yes, everyone with a nominally HA workload should have a PDB.