4 ms·

Reading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accid

by jefftk 6y ago

Reading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accidentally was also cleaning up long-term storage. Increasing the redundancy of their storage wouldn't have helped.

chongli 6y ago
If you have regular (daily, hourly) tape backups that are shipped off-site then you greatly mitigate the potential damage of software bugs. It sounds like they weren’t making backups and just treating failover replica servers as a “backup” (which they clearly aren’t).
- StillBored 6y ago
  This sounds like a case of a bug in their software uncovered a huge gaping hole in their architecture. Put another way, while they seem to have fixed the immediate problem, no word on whether they are putting real BACKUP plans in place to avoid a similar problem happening and whipping out everything.
- deepstack 6y ago
  Always have a back that is write once! Storage are cheap now days. If this is your core business get your act together.