3 ms·
Any replication / backup / restore process that is not regularly tested to verify that it works is fragile. Even more so when it's a series of manual steps to s
by koolba 10y ago
Any replication / backup / restore process that is not regularly tested to verify that it works is fragile. Even more so when it's a series of manual steps to stonith, promote, etc. There's nothing particularly fragile about Postgres and there's a wealth of backup options ranging from logical (pg_dump), to replication (built-in WAL based), to physical archiving (pg_basebackup + WAL archiving). There are a number of cloud DBaaS vendors that run huge fleets of Postgres databases (Heroku and Amazon RDS come to mind). This is a solved problem.
The root of this clusterfuck for GitLab is that they didn't test anything. Looks like there was some cargo culting of "Yah we need backups so let's run something ... yada yada S3 ..." but no verification of anything actually being backed up, or error alerts for missed backups, let alone testing a full restoration from said backups.