2 ms·
I was involved a combination of 4 and 6 once: Production deployment had a problem so the ops team rolled back the code and restored the db from backup. Except t
by WantonQuantum 4y ago
I was involved a combination of 4 and 6 once: Production deployment had a problem so the ops team rolled back the code and restored the db from backup. Except the backup that was restored was a day old. By the time we realised what had happened we had new data in the prod database so there was no correct version of db any more.
- cratermoon 4y agoSame, or nearly so. Also saw the entire site DB get corrupted. Backups had been broken for some time. All hands, we found some way to restore from the raw DB files, I don't recall the details. I know it took something like 18 hours, and we we lucky. I honestly don't think Twitter will make it through Thanksgiving and the World Cup starting in the same week. Even if they do, The World Cup final is Dec 18th, the Sunday before Christmas. And then Christmas and New Years.