Lovely! We just migrated from postgres 14 to 16 for 3 postgres clusters (servers) on RDS containing about 2TB of data across 8 databases. We were down from 00:0

Lovely! We just migrated from postgres 14 to 16 for 3 postgres clusters (servers) on RDS containing about 2TB of data across 8 databases. We were down from 00:00 to 04:00. Steps we took: * enabled our fallback "maintenance mode" site. It's a super lightweight version of our site running on CF workers. * scaled down all apps using the db to 0 in terraform * hit the upgrade button in the aws web ui, which runs pg_upgrade. 14->15 * waited for it to finish * hit the upgrade button again. 15->16 * waited for the dbs to start accepting connections (they do before they're marked as ready, I think aws does more stuff than pg_upgrade) * Started `VACUUM ANALYZE; REINDEX DATABASE CONCURRENTLY`. The idea is to avoid performance issues between versions and make use of performance improvements from new versions. * Started spinning up the apps again * Waited until all apps had a handful of containers running * Started accepting traffic (disabled maintenance site) * Went to bed The REINDEX CONCURRENTLY happily chugged along for the biggest db for another 18 hours without blocking anything. Next time we're doing aws blue/green deploys to avoid downtime. We didn't this time since we weren't on 14.9 yet (the minimum minor version of 14 supported by blue green). If I was doing this myself I wouldn't pay the AWS tax, instead do blue/green myself with logical replication and a load balancer.