5 ms·

That is an amazing story. Who knew that putting everything in a database without backups was risky?

by WJW 6y ago

brailsafe 6y ago
Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.
- pedrovhb 6y ago
  Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
  - arethuza 6y ago
    I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.
  - regularfry 6y ago
    It's up there with "unless you test power supply failover, don't rely on that generator to save you". I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
    - arethuza 6y ago
      I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up. When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
      - grumple 6y ago
        That sounds like their testing worked out for them. Better than a random failure.
        myself248 6y ago
        Yup. Have the problem when all the right people are awake and on-site to handle it. I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment. If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
- ewindal 6y ago
  Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1] [1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlh5q/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
- M2Ys4U 6y ago
  They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.
  - brailsafe 6y ago
    Ah, didn't spot that
PeterisP 6y ago
They explicitly mention that they have backups, but that the "backups are fucked", whatever that means. Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.
- fuzzy2 6y ago
  They backups are probably fine (as in “the data is in there”) but there may be no plans on how to restore them. Or they may be incomplete. I doubt any external factors are to blame.

That is an amazing story. Who knew that putting everything in a database without backups was risky?

by WJW 6y ago

That is an amazing story. Who knew that putting everything in a database without backups was risky?

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

pedrovhb 6y ago
Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
- arethuza 6y ago
  I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.
- regularfry 6y ago
  It's up there with "unless you test power supply failover, don't rely on that generator to save you". I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
  - arethuza 6y ago
    I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up. When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
    - grumple 6y ago
      That sounds like their testing worked out for them. Better than a random failure.
      - myself248 6y ago
        Yup. Have the problem when all the right people are awake and on-site to handle it. I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment. If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
ewindal 6y ago
Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1] [1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlh5q/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
M2Ys4U 6y ago
They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.
- brailsafe 6y ago
  Ah, didn't spot that

PeterisP 6y ago

They explicitly mention that they have backups, but that the "backups are fucked", whatever that means. Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.

fuzzy2 6y ago
They backups are probably fine (as in “the data is in there”) but there may be no plans on how to restore them. Or they may be incomplete. I doubt any external factors are to blame.