3 ms·

Nice writeup... Although I'm missing a few points... In my opinion a good backup (system) is only good, if it has been tested to be restorable as fast as possi

Nice writeup... Although I'm missing a few points... In my opinion a good backup (system) is only good, if it has been tested to be restorable as fast as possible and the procedure is clear (like in documented). How often have I heard or seen backups that "work great" and "oh, no problem we have them" only to see them fail or take ages to restore, when the disaster has happened (2 days can be an expensive amount of time in a production environment). Quite too often only parts could be restored. Another missing aspect is within the snapshots section... I like restic, which provides repository based backup with deduplicated snapshots for FILES (not filesystems). It's pretty much what you want if you don't have ZFS (or other reliable snapshot based filesystems) to keep different versions of your files that have been deleted on the filesystem. The last aspect is partly mentioned, the better PULL than PUSH part. Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything... So you could either use readonly media (like Blurays) or PULL is mandatory. It is also helpful to have auto-snapshotting on ZFS via zfs-auto-snapshot, zrepl or sanoid to go back in time to where the ransomware has started its journey.

sgc 1y ago
Since you mentioned restic, is there something wrong with using restic append-only with occasional on-server pruning instead of pulling? I thought this was the recommended way of avoiding ransomware problems using restic.
- sandreas 1y ago
  There are several methods... there is also restic rest-server (https://github.com/restic/rest-server https://github.com/restic/rest-server). I personally use ZFS with pull via ssh...
TacticalCoder 1y ago
> So you could either use readonly media (like Blurays) or PULL is mandatory. Or like someone already commented you can use a server that allows push but doesn't allow to mess with older files. You can for example restrict ssh to only the scp command and the ssh server can moreover offer a chroot'ed environment to which scp shall copy the backups. And the server can for example daily rotate that chroot. The push can then push one thing: daily backups. It cannot log in. It cannot overwrite older backups. Short of a serious SSH exploit where the ransomware could both re-configure the server to accept all ssh (and not just scp) and escape the chroot box, the ransomware is simply not destroying data from before the ransomware found its way on the system. My backup procedure does that for the one backup server that I have on a dedicated server: a chroot'ed ssh server that only accepts scp and nothing else. It's of course just one part of the backup procedure, not the only thing I rely on for backups. P.S: it's not incompatible with also using read-only media
- anonymars 1y ago
  I don't understand why this is dead..is it wrong advice? Is there some hidden flaw? Is it simply because the content is repeated elsewhere? On the face of it "append-only access (no changes)" seems sound to me
  - quesera 1y ago
    TacticalCoder's comments appear to be auto-deaded for the last week or so. I did not see a likely reason in a quick review of their comment history. You can view a comment directly by following the "... ago" link, and from there you can use the "vouch" link to revive the comment. I vouched for a few of TacticalCoder's recent comments.
- immibis 1y ago
  Pull-only mode is about reducing that chance of SSH exploits even further.
marcusb 1y ago
> Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything That depends on how you have access to your backup servers configured. I'm comfortable with append-only backup enforcement for push backups[0] with Borg and Restic via SSH, although I do use offline backup drive rotation as a last line of defense for my local backup set. YMMV. 0 - https://marcusb.org/posts/2024/07/ransomware-resistant-backups-with-borg-and-restic/ https://marcusb.org/posts/2024/07/ransomware-resistant-backu...
- guillem_lefait 1y ago
  Could you elaborate on your strategy to rotate your disks ?
  - marcusb 1y ago
    It's pretty simple: the backup host has the backup disk attached via a usb cradle. There's a file in the root directory of the backup disk file system that gets touched when the drive is rotated. A cron jobs emails me if this file is more than 3 months old. When I rotate the disk, I format the new disk and recreate the restic repos for the remote hosts. I then move the old disk into a fireproof safe. I keep four drives in rotation, so at any given point in time I have the online drive plus three with progressively older backup sets in the safe.
    - guillem_lefait 1y ago
      And then, after a year what do you do with the oldest hard drive ? Does it enter the cycle again, do you destruct it or do you use it in a failsafe environnement ? The procedure looks OK and I would like to make it more organised myself, just trying to find the right balance.
      - marcusb 1y ago
        The drive enters the cycle again. I use the drives until they show signs of failure (SMART monitoring/testing), or until I need to upgrade for capacity reasons. I'm using "recertified" (really, used) drives that I've written about here: https://marcusb.org/posts/2024/03/used-hard-drives-from-tech-on-tech/ https://marcusb.org/posts/2024/03/used-hard-drives-from-tech.... They are inexpensive and, so far, have been very reliable. (And, yes, I've done restores from the backup sets.)
KPGv2 1y ago
> tested to be restorable as fast as possible That depends on your goal, right? If it took me six months to recover my family photo backups, that'd be fine by me.

sgc 1y ago

Since you mentioned restic, is there something wrong with using restic append-only with occasional on-server pruning instead of pulling? I thought this was the recommended way of avoiding ransomware problems using restic.

sandreas 1y ago
There are several methods... there is also restic rest-server (https://github.com/restic/rest-server https://github.com/restic/rest-server). I personally use ZFS with pull via ssh...

TacticalCoder 1y ago

> So you could either use readonly media (like Blurays) or PULL is mandatory. Or like someone already commented you can use a server that allows push but doesn't allow to mess with older files. You can for example restrict ssh to only the scp command and the ssh server can moreover offer a chroot'ed environment to which scp shall copy the backups. And the server can for example daily rotate that chroot. The push can then push one thing: daily backups. It cannot log in. It cannot overwrite older backups. Short of a serious SSH exploit where the ransomware could both re-configure the server to accept all ssh (and not just scp) and escape the chroot box, the ransomware is simply not destroying data from before the ransomware found its way on the system. My backup procedure does that for the one backup server that I have on a dedicated server: a chroot'ed ssh server that only accepts scp and nothing else. It's of course just one part of the backup procedure, not the only thing I rely on for backups. P.S: it's not incompatible with also using read-only media

anonymars 1y ago
I don't understand why this is dead..is it wrong advice? Is there some hidden flaw? Is it simply because the content is repeated elsewhere? On the face of it "append-only access (no changes)" seems sound to me
- quesera 1y ago
  TacticalCoder's comments appear to be auto-deaded for the last week or so. I did not see a likely reason in a quick review of their comment history. You can view a comment directly by following the "... ago" link, and from there you can use the "vouch" link to revive the comment. I vouched for a few of TacticalCoder's recent comments.
immibis 1y ago
Pull-only mode is about reducing that chance of SSH exploits even further.

marcusb 1y ago

> Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything That depends on how you have access to your backup servers configured. I'm comfortable with append-only backup enforcement for push backups[0] with Borg and Restic via SSH, although I do use offline backup drive rotation as a last line of defense for my local backup set. YMMV. 0 - https://marcusb.org/posts/2024/07/ransomware-resistant-backups-with-borg-and-restic/ https://marcusb.org/posts/2024/07/ransomware-resistant-backu...

guillem_lefait 1y ago
Could you elaborate on your strategy to rotate your disks ?
- marcusb 1y ago
  It's pretty simple: the backup host has the backup disk attached via a usb cradle. There's a file in the root directory of the backup disk file system that gets touched when the drive is rotated. A cron jobs emails me if this file is more than 3 months old. When I rotate the disk, I format the new disk and recreate the restic repos for the remote hosts. I then move the old disk into a fireproof safe. I keep four drives in rotation, so at any given point in time I have the online drive plus three with progressively older backup sets in the safe.
  - guillem_lefait 1y ago
    And then, after a year what do you do with the oldest hard drive ? Does it enter the cycle again, do you destruct it or do you use it in a failsafe environnement ? The procedure looks OK and I would like to make it more organised myself, just trying to find the right balance.
    - marcusb 1y ago
      The drive enters the cycle again. I use the drives until they show signs of failure (SMART monitoring/testing), or until I need to upgrade for capacity reasons. I'm using "recertified" (really, used) drives that I've written about here: https://marcusb.org/posts/2024/03/used-hard-drives-from-tech-on-tech/ https://marcusb.org/posts/2024/03/used-hard-drives-from-tech.... They are inexpensive and, so far, have been very reliable. (And, yes, I've done restores from the backup sets.)

KPGv2 1y ago

> tested to be restorable as fast as possible That depends on your goal, right? If it took me six months to recover my family photo backups, that'd be fine by me.