3 ms·

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only wi

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

pedrovhb 6y ago
Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
- arethuza 6y ago
  I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.
- regularfry 6y ago
  It's up there with "unless you test power supply failover, don't rely on that generator to save you". I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
  - arethuza 6y ago
    I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up. When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
    - grumple 6y ago
      That sounds like their testing worked out for them. Better than a random failure.
      - myself248 6y ago
        Yup. Have the problem when all the right people are awake and on-site to handle it. I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment. If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
ewindal 6y ago
Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1] [1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlh5q/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
M2Ys4U 6y ago
They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.
- brailsafe 6y ago
  Ah, didn't spot that

pedrovhb 6y ago

Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.

arethuza 6y ago
I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.
regularfry 6y ago
It's up there with "unless you test power supply failover, don't rely on that generator to save you". I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
- arethuza 6y ago
  I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up. When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
  - grumple 6y ago
    That sounds like their testing worked out for them. Better than a random failure.
    - myself248 6y ago
      Yup. Have the problem when all the right people are awake and on-site to handle it. I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment. If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.

ewindal 6y ago

Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1] [1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqlh5q/ https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

M2Ys4U 6y ago

They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.

brailsafe 6y ago
Ah, didn't spot that