A very similar thing happened to living social during their brightest years, but the replication and the backups had failed too. The oldest valid backup was abo

A very similar thing happened to living social during their brightest years, but the replication and the backups had failed too. The oldest valid backup was about a week old. It took the whole company offline for 2 days. It took a team of 10 people and some extra consultants to come up with a half backed version of the latest database based on elastic cache instances, redis caches and other "not meant to be a backup" services. It was insane walking in an office that had hundreds of employees and see they all gone while we rebuild this cobbled together db. At one point someone called it "good enough" and they basically had to honor the customer word if they had purchased something and it wasn't there. It was a mess. It was on all major news, and it was really bad press. In the end, they actually had a massive bump in their sales afterwards. Everyone went to checkout their own purchases and ended up buying something else, and the news was like free ads. https://www.washingtonpost.com/business/capitalbusiness/the-download-livingsocial-goes-down-for-nearly-48-hours-after-critical-database-error/2013/11/15/2012bc4e-4c9b-11e3-9890-a1e0997fb0c0_story.html https://www.washingtonpost.com/business/capitalbusiness/the-...