Amazing document. Thank you for sharing. Taking it back to my company to make sure we can learn from it and to know what to check (like our logical backups...

Amazing document. Thank you for sharing. Taking it back to my company to make sure we can learn from it and to know what to check (like our logical backups... I know we've seen issues with our 9.5 servers and RHEL7 defaulting to 9.1 or 9.2 on our host where we take the backups from! Verifying exit code here we come...) @sytse, I noticed you _do_ use streaming WAL replication, but I didn't notice any mention of attempting Point In Time Recovery. Have you taken a look into archiving the WAL files in S3? Those, along with frequent pg_basebackup's (frequent because replaying a WAL file has been painfully slow for us) could allow you to point in time recover to either a timestamp, or a transaction (and before or after). https://www.postgresql.org/docs/9.6/static/continuous-archiving.html https://www.postgresql.org/docs/9.6/static/continuous-archiv... We use https://github.com/wal-e/wal-e https://github.com/wal-e/wal-e to manage our uploading to swift (no S3 at our company heh) and then inhouse tooling to build a recovery.conf. Note we actually have our asynchronous followers work off of this system too so they're not taking bandwidth from the primary. (note this is can lead to ~ 1 WAL file of data loss, but is acceptable for us.) I doubt I could be of any help, since reading the report definitely shows y'all having an up on me with pg knowledge, but if there's anything I can do / talk about feel free to reach out.