This is why you have lots of copies of your data. Matt Raney has a great talk about designing for failure that includes details on Uber's "worst outage ever." I

This is why you have lots of copies of your data. Matt Raney has a great talk about designing for failure that includes details on Uber's "worst outage ever." It too involved postgresql replication and mistaking on host for another, but they didn't lose any data because they had more than a dozen live copies of their database. https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s This isn't an alternative to working backups, of course, but it is an additional safety net. Plus it can give you a lot more options when handling an incident.