3 ms·

How hard can it be to have a set of data, make 2 copies and store the in different locations.

• vidarh 6y ago
  If you're storing a large enough set of data for long enough, 2 copies will be woefully insufficient to ensure you don't suffer a loss of a given subset of the data.
• jefftk 6y ago
  Reading their description of the problem, it wasn't that they lost one of their redundant servers, it was that their code that cleans up temporary storage accidentally was also cleaning up long-term storage. Increasing the redundancy of their storage wouldn't have helped.
  • chongli 6y ago
    If you have regular (daily, hourly) tape backups that are shipped off-site then you greatly mitigate the potential damage of software bugs. It sounds like they weren’t making backups and just treating failover replica servers as a “backup” (which they clearly aren’t).
    • StillBored 6y ago
      This sounds like a case of a bug in their software uncovered a huge gaping hole in their architecture. Put another way, while they seem to have fixed the immediate problem, no word on whether they are putting real BACKUP plans in place to avoid a similar problem happening and whipping out everything.
    • deepstack 6y ago
      Always have a back that is write once! Storage are cheap now days. If this is your core business get your act together.