6 ms·

I will continue to stand on my soapbox and proclaim that backup has to include an offline component (preferably one that's verified independently). An attacker

I will continue to stand on my soapbox and proclaim that backup has to include an offline component (preferably one that's verified independently). An attacker has to be willing to bring "kinetic" means to bear if you have offline backup media.

hsk0823 8y ago
How do we know that this isn't the case here and said hacker went after those records as well? This all seems like an inside job and thus even offline backups are at risk if a disgruntled employee wanted to go after it.
arethuza 8y ago
Also, a backup that you don't fully restore somewhere to check it actually fully works isn't really much of a backup at all.
- dsfyu404ed 8y ago
  As long as you know the 1s and 0s were properly written to a disk that's now sitting in a storage locker somewhere that's infinitely better than nothing. If shit really hits the fan you can figure out how to restore from it but the data actually needs to be there in the first place. Sure, practice restoring is ideal but anything is better than nothing. It's the difference between not being able to do anything for a week and probably having to close your business entirely. Edit: Why is this getting downvoted? I'm not saying this is a "real" backup solution. Call it an archive if you want. It's still better than not having any copy of you data in offline storage off site. Defense in depth people, there's no reason you can't both have a real backup system and shove old disks in a storage tote.
  - haukilup 8y ago
    I can't downvote so I can't say why. My first impression though is that your post doesn't offer much to the conversation. Sure, that's better than nothing. People should start the process of making backups. But when the backups are important (as we're discussing here), you need to verify that a backup can be restored - otherwise you're ignoring a huge risk.
  - scardine 8y ago
    I guess you are getting downvoted because it would be really really hard to know if the all of the right 1s and 0s you need are there without restoring.
  - benjohnson 8y ago
    My experience may shed some light on this: I use pgdump to make backups of an important PostgreSQL server and I thought I was done. Then the server died. I then used the standard import... but it failed. This was a while ago, but pgdump didn't preserve/escape some of the weird quoting in my strings or I (most likely) didn't understand hot to use pgdump properly. I hand to manually figure this out and fix the strings in the file before it would import. If I had tested the full restore, then I would have not have had to panic. Thankfully pgdump didn't throw up it's hands and not emit the strings - they would have been a decent and reasonably sane choice as well. I got lucky.
    - luckylion 8y ago
      You'd have to test every backup though, because initially, your DB might not contain any strings that trigger the problem.
    - sigotirandolas 8y ago
      It’s good to test PostgreSQL backups (or basically everything that is constantly doing reads and writes)... some pitfalls in my small experience with a RPi and some cheap SD card: * If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools. * Similarly, if the underlying file system experiments corruption, it will also seem to keep working for a while before, and after that you may be left with a bad database and backup. You have to be able to restore multiple time points or ensure your backup is good.
      - DKnoll 8y ago
        > * If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools. This is true for any sort of mounted database. Always dump the database separately from any FS backups. Another good idea (but not a replacement for backups) is to replicate the DB to a read-only slave which you can either make the master or recreate your master from.
  - Bjartr 8y ago
    You are getting downvoted because there is more to it than just "practice". If you haven't restored from a backup before, you don't know if it's possible to restore from that backup. > As long as you know the 1s and 0s were properly written ... How do you know? Did it actually write properly? If so, was the data you chose to write the data you actually need to fully restore? You might be able to do certain tests or checks to determine this to a high degree of probability, but there's only one test that will tell you with certainty, and that's actually performing a restore successfully.
  - toomuchtodo 8y ago
    > Sure, practice restoring is ideal but anything is better than nothing. Practicing restores is mandatory to confirm integrity. Any other form of backup is ceremony.
- yellowapple 8y ago
  I feel like these are self-contradictory unless you're really going to go through the effort of periodically restoring each of your offline tapes to a server (which I'm sure some organizations end up doing, to be fair, but that doesn't make it any less of a pain in the ass).
  - AznHisoka 8y ago
    I also hear people say that all the time. Usually they're managing measly small 1-100 GB databases, not multiple ones > 1 TB in size. Restoring them in a regular basis is easy in that case. But I'd love to hear how people practice restoring all their backups on a regular basis when it requires having an enormous amount of space. Practically, I just restore a few backups every month, and if it works, I just assume the others work as well (which I know is a false assumption, but what are you going to do?)
    - Intermernet 8y ago
      My DR procedure was to automatically restore the mail server, DB server and file server once a month with automated fail back to the live servers if they failed. These would happen outside heavy operation hours, and if successful, the restored servers would become the live servers in most instances. These were done using scripted vmware instances, mostly restoring from StorageCraft Shadow Protect backups. File server was the most resource intensive and the most prone to complaints about downtime, but usually the trade off was fine. DB server was usually fine due to the out of hours restore, and the ability to master - slave - replicate - handover. Mail server was usually transparent as you could set up smtp transfer rules for incoming, and outgoing didn't really care. The trickier ones were things like IBM I Series servers that usually require periodic full system saves, restores and tests. These couldn't be done automatically without excessive licensing fees and much manual baby sitting. If you get to the point where you have automatic scheduled DR, you can be pretty confident your backups are working. It the automated DR fails, you roll back, learn whatever lesson the failure taught you, and change the system so it doesn't happen again. Summary: schedule DR, not backups.
    - Dylan16807 8y ago
- benj111 8y ago
  Ha ha yes! One of my old employers dutifully did tape backups every day, and one of the bosses took them home. During an upgrade it was revealed that the drive wasn't working, and hadn't been working for at least a year (I think that's how far back the backups went).
- dredmorbius 8y ago
  People don't want backups. People want restores.
deleted 8y ago
[deleted]
xoa 8y ago
>that backup has to include an offline component I'd balance this (and arethuza's reply) with the basic fact that we have to balance all backup advice with practicalities of a given individual or organization's situation too. A backup is also no good if it's not used, consistently, or if someone is told "if you don't do X might as well not even bother" and gets discouraged because X requires more capital expenditure or expertise then they're actually capable of. Any decent backup is in fact better then nothing and can help with at least some failure scenarios. More expansive backups can deal with a wider range of threats, but tend to be more expensive too. I don't want anyone to ever be discouraged from getting the ball rolling. Just using a direct simple cloud solution or plug-in USB drive even is definitely objectively better then nothing. There is also room for hybrids and considering where, exactly, the offline bit should come in. It doesn't necessarily have to be the physical storage. There are cloud storage providers that offer WORM capability, or can separate delete/over write permissions from append. So it's possible to have a system that is backing up to that independent service through a restricted permission account and then have all management done from a minimal dedicated set of notebooks that are generally kept locked in safes. Another dedicated isolated system could be responsible for pulling a sample of backup files down and doing some simple watchdogging (even just a tripwire or entropy check). The backup is automatic and offsite, but it'll still have some additional resistance to remote attacks on the main site by virtue of system independence even if it's not tapes in a vault. And it's more practical for a small or less technically competent org. I don't deny the value of tapes in a vault or similar, just recognize the capex and the difficulty of many places in making sure it's sufficiently consistent. Resist the finger wagging urge is the main thing, at least not without considering the cost/value equation for the specific situation. Edit: to be clear, in this specific case there is definitely a reasonable argument that after decades and with a reasonable number of users they should have built up to something heavier duty. It did have paying users apparently after all. Even if it took a few years to save up for a better backup system that should have been done. But then again even just using Amazon Glacier and creating some custom IAM policies for API restriction along with Vault locks may have been enough to prevent this particular attack.

hsk0823 8y ago

How do we know that this isn't the case here and said hacker went after those records as well? This all seems like an inside job and thus even offline backups are at risk if a disgruntled employee wanted to go after it.

arethuza 8y ago

Also, a backup that you don't fully restore somewhere to check it actually fully works isn't really much of a backup at all.

dsfyu404ed 8y ago
As long as you know the 1s and 0s were properly written to a disk that's now sitting in a storage locker somewhere that's infinitely better than nothing. If shit really hits the fan you can figure out how to restore from it but the data actually needs to be there in the first place. Sure, practice restoring is ideal but anything is better than nothing. It's the difference between not being able to do anything for a week and probably having to close your business entirely. Edit: Why is this getting downvoted? I'm not saying this is a "real" backup solution. Call it an archive if you want. It's still better than not having any copy of you data in offline storage off site. Defense in depth people, there's no reason you can't both have a real backup system and shove old disks in a storage tote.
- haukilup 8y ago
  I can't downvote so I can't say why. My first impression though is that your post doesn't offer much to the conversation. Sure, that's better than nothing. People should start the process of making backups. But when the backups are important (as we're discussing here), you need to verify that a backup can be restored - otherwise you're ignoring a huge risk.
- scardine 8y ago
  I guess you are getting downvoted because it would be really really hard to know if the all of the right 1s and 0s you need are there without restoring.
- benjohnson 8y ago
  My experience may shed some light on this: I use pgdump to make backups of an important PostgreSQL server and I thought I was done. Then the server died. I then used the standard import... but it failed. This was a while ago, but pgdump didn't preserve/escape some of the weird quoting in my strings or I (most likely) didn't understand hot to use pgdump properly. I hand to manually figure this out and fix the strings in the file before it would import. If I had tested the full restore, then I would have not have had to panic. Thankfully pgdump didn't throw up it's hands and not emit the strings - they would have been a decent and reasonably sane choice as well. I got lucky.
  - luckylion 8y ago
    You'd have to test every backup though, because initially, your DB might not contain any strings that trigger the problem.
  - sigotirandolas 8y ago
    It’s good to test PostgreSQL backups (or basically everything that is constantly doing reads and writes)... some pitfalls in my small experience with a RPi and some cheap SD card: * If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools. * Similarly, if the underlying file system experiments corruption, it will also seem to keep working for a while before, and after that you may be left with a bad database and backup. You have to be able to restore multiple time points or ensure your backup is good.
    - DKnoll 8y ago
      > * If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools. This is true for any sort of mounted database. Always dump the database separately from any FS backups. Another good idea (but not a replacement for backups) is to replicate the DB to a read-only slave which you can either make the master or recreate your master from.
- Bjartr 8y ago
  You are getting downvoted because there is more to it than just "practice". If you haven't restored from a backup before, you don't know if it's possible to restore from that backup. > As long as you know the 1s and 0s were properly written ... How do you know? Did it actually write properly? If so, was the data you chose to write the data you actually need to fully restore? You might be able to do certain tests or checks to determine this to a high degree of probability, but there's only one test that will tell you with certainty, and that's actually performing a restore successfully.
- toomuchtodo 8y ago
  > Sure, practice restoring is ideal but anything is better than nothing. Practicing restores is mandatory to confirm integrity. Any other form of backup is ceremony.
yellowapple 8y ago
I feel like these are self-contradictory unless you're really going to go through the effort of periodically restoring each of your offline tapes to a server (which I'm sure some organizations end up doing, to be fair, but that doesn't make it any less of a pain in the ass).
- AznHisoka 8y ago
  I also hear people say that all the time. Usually they're managing measly small 1-100 GB databases, not multiple ones > 1 TB in size. Restoring them in a regular basis is easy in that case. But I'd love to hear how people practice restoring all their backups on a regular basis when it requires having an enormous amount of space. Practically, I just restore a few backups every month, and if it works, I just assume the others work as well (which I know is a false assumption, but what are you going to do?)
  - Intermernet 8y ago
    My DR procedure was to automatically restore the mail server, DB server and file server once a month with automated fail back to the live servers if they failed. These would happen outside heavy operation hours, and if successful, the restored servers would become the live servers in most instances. These were done using scripted vmware instances, mostly restoring from StorageCraft Shadow Protect backups. File server was the most resource intensive and the most prone to complaints about downtime, but usually the trade off was fine. DB server was usually fine due to the out of hours restore, and the ability to master - slave - replicate - handover. Mail server was usually transparent as you could set up smtp transfer rules for incoming, and outgoing didn't really care. The trickier ones were things like IBM I Series servers that usually require periodic full system saves, restores and tests. These couldn't be done automatically without excessive licensing fees and much manual baby sitting. If you get to the point where you have automatic scheduled DR, you can be pretty confident your backups are working. It the automated DR fails, you roll back, learn whatever lesson the failure taught you, and change the system so it doesn't happen again. Summary: schedule DR, not backups.
  - Dylan16807 8y ago
benj111 8y ago
Ha ha yes! One of my old employers dutifully did tape backups every day, and one of the bosses took them home. During an upgrade it was revealed that the drive wasn't working, and hadn't been working for at least a year (I think that's how far back the backups went).
dredmorbius 8y ago
People don't want backups. People want restores.

deleted 8y ago

[deleted]

xoa 8y ago

>that backup has to include an offline component I'd balance this (and arethuza's reply) with the basic fact that we have to balance all backup advice with practicalities of a given individual or organization's situation too. A backup is also no good if it's not used, consistently, or if someone is told "if you don't do X might as well not even bother" and gets discouraged because X requires more capital expenditure or expertise then they're actually capable of. Any decent backup is in fact better then nothing and can help with at least some failure scenarios. More expansive backups can deal with a wider range of threats, but tend to be more expensive too. I don't want anyone to ever be discouraged from getting the ball rolling. Just using a direct simple cloud solution or plug-in USB drive even is definitely objectively better then nothing. There is also room for hybrids and considering where, exactly, the offline bit should come in. It doesn't necessarily have to be the physical storage. There are cloud storage providers that offer WORM capability, or can separate delete/over write permissions from append. So it's possible to have a system that is backing up to that independent service through a restricted permission account and then have all management done from a minimal dedicated set of notebooks that are generally kept locked in safes. Another dedicated isolated system could be responsible for pulling a sample of backup files down and doing some simple watchdogging (even just a tripwire or entropy check). The backup is automatic and offsite, but it'll still have some additional resistance to remote attacks on the main site by virtue of system independence even if it's not tapes in a vault. And it's more practical for a small or less technically competent org. I don't deny the value of tapes in a vault or similar, just recognize the capex and the difficulty of many places in making sure it's sufficiently consistent. Resist the finger wagging urge is the main thing, at least not without considering the cost/value equation for the specific situation. Edit: to be clear, in this specific case there is definitely a reasonable argument that after decades and with a reasonable number of users they should have built up to something heavier duty. It did have paying users apparently after all. Even if it took a few years to save up for a better backup system that should have been done. But then again even just using Amazon Glacier and creating some custom IAM policies for API restriction along with Vault locks may have been enough to prevent this particular attack.