5 ms·

Disaster Recovery with ZFS and Zrepl

32 points by aargh_aargh 2y ago

l2dy 2y ago
> I don’t back up my drives, I replicate them. Don't take this as a general advice. For important data, it's important to have multiple backups and validate their effectiveness routinely.
- chromakode 2y ago
  It's part of the 3-2-1 rule: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ My mid term goal is to trade offsite replication with a friend for automatically replicating the offsite portion of important things.
  - toastal 2y ago
    That works assuming you have the budget & technical friends
    - chromakode 2y ago
      Residential fiber internet is a lovely thing :D
- diarrhea 2y ago
  I feel like a filesystem with native snapshot support (zfs, btrfs), replicated once (also native in zfs) obsoletes conventional 3-2-1 backup systems. It’s technically only 2 versions, but you’re protected from all the same failure modes.
  - aborsy 2y ago
    If there is a bug in file system, both source and its replication can be lost. I will add to this a different backup software to an offsite or cloud. I use restic.
    - raverbashing 2y ago
      Especially in a fs that falls apart if you look at it wrong like ZFS
      - sshine 2y ago
        How does zfs fall apart? I’ve run it beneath Ubuntu for several years, and it’s only saved my ass. I didn’t replicate, only used the apt snapshots.
        justinclift 2y ago
        It's quite possible to get ZFS into weird states without too much effort when you're screwing around with the underlying devices (adding, deleting, changing things). This seems to crop up at really inconvenient times too, like when you're trying to do something during a scheduled outage. :( That kind of thing aside though, it's been pretty solid in my use for actual data storage. Just don't use ZFS's native encryption + ZFS snapshots + send/recv. Reportedly that combination is a cause of data corruption: https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt
      - stiray 2y ago
        Heh, you must really have foul look, I am using it on FreeBSD since 2007ish and never had any issues. Same pool, no longer the same disks as they were all replaced since. Also the hardware around it was replaced and FreeBSD 8 updated to all new versions, currently running 14.0. Never lost any data. There isn't much software that could claim that. Meanwhile I lost (I had backups) all my data twice on btrfs, I know it is more stable now, but I certainly wont ever use it again. Even HAMMER1 (I would love to use HAMMER2, but until my server dies, I will stay with FreeBSD) lost it only once and even in that case, after debugging irc session with Matt Dillon, I was able to recover most files. The only thing that pisses me off is that Kubuntu doesnt support it trough installer (yes I could do it manually but been there with Fedora and I am sick of tracking if initfs was updated or finish with unbootable - easy solvable, but annoying - situation ) and I am now forced to use Ubuntu with KDE on workstation. But this is not zfs.
  - tjoff 2y ago
    That doesn't make any sense. Native snapshots are fantastic, but are merely an effective way to do backups. One where you trade some complexity from the backup software to the filesystem. (only talking backups, snapshots of course have utility beyond that usecase as well)
  - prmoustache 2y ago
    Hmmm from what I understand zrepl can do copy of source to backup both in a push or pull mode. I would say the push mode is still a very fragile way to do backups. Imagine source machine is compromised and attacker decide to delete/encrypt your data, and see there is a backup mechanism connecting to the backup machine, what prevents him from using the deleting/encrypting the backups as well? You'd definitely want the backup machine to pull the snapshots, and have no way to connect to it from source machine directly with a user that would have access to the data or an admin account. That means no ssh keys on the source machine, no password kept in a password manager that would be loaded on the source machine either. Another strong method would involve 3 machines: source --push--> replica1 <--pull-- replica2 Where source and replica1 would have ZFS filesystem and snapshots while replica2 is using a different filesystem (LVM + ext4 ) and snapshots to safeguard from replicating bugs that lead to data not being available. ZFS snapshots could be saved as individual files on this filesystem.
    - justinclift 2y ago
      > ... what prevents him from using the deleting/encrypting the backups as well? With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. > That means no ssh keys on the source machine ... Typically for non-user logins (eg script access and similar) you do the extra step of configuring the receiving ssh to only allow a specific command for a given key. It's a configurable ssh thing, where you add extra info to the .ssh/authorized_keys file on the destination server. With that approach, it doesn't allow general user logins while still allowing the source machine to send the data.
      - prmoustache 2y ago
        > With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. Only and only if you apply best practicies mentionned above in the second point of your post. Anyway I'd rather protect my backup as much as I can and not allow the source machine to have any direct access to an account on the backup server because of possible security. Security is hard sometimes and you never know when some bugs might increase the possibilities. I like my backup servers to not have any open port. The caveat is that it is only possible if your primary backup server is locally and physically accessible. If you are travelling you might want to be able to access it without being physically present. An option might be to just have ssh disabled when you are at home and you enable the service when you know you won't be at home for a long enough period to make it a problem if you need to restore data.
- aargh_aargh 2y ago
  One case where this might go wrong is if you accidentally delete a file, the deletion replicates and you're left without a backup. Replication is nice but it is not a replacement for backup.
  - mhio 2y ago
    The setup in the article replicates regular incremental snapshots rather than file system changes. ZFS snapshots allow you to roll back to the point in time of each snapshot, up to the amount of disk space you can maintain for storing the changes.
chromakode 2y ago
Funny story -- when I was working on the xkcd Machine comic, I actually used the ZFS snapshots to rescue data. I accidentally blew away some early physics prototype code and fished it out of /.zfs/snapshot.
- sshine 2y ago
  Thanks for advertising the machine. I hadn’t seen it yet. It’s amazing!
  - aargh_aargh 2y ago
    And here's the dev blogpost, hot off the press! https://news.ycombinator.com/item?id=40300454 https://news.ycombinator.com/item?id=40300454
totetsu 2y ago
My face palm moment of this year was accidentally restoring a zfs snapshot of my root pool form a week ago, but it was actually a year and a week ago. Didn’t lose any of my data, but suddenly I had some offering to format their version mismatched databases.
deleted 2y ago
[deleted]
tie-in 2y ago
You don't actually need a dedicated ZFS backup program. A simple cron script will handle incremental backups just fine. If anyone is interested, the script we use to backup our multi-TB PostgreSQL database can be found here: https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po...
justinclift 2y ago
It sounds like the author is using ZFS encryption. If that's the case, then doing so with replicated ZFS snapshots is probably not a good idea. That specific scenario (ZFS encryption -> replication of encrypted snapshots) is a known cause of ZFS corruption. :( https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt Unfortunately it doesn't seem to be widely known about, though there is a suggestion to make it official: https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/openzfs-docs/issues/494
justinclift 2y ago
For ZFS use it's probably a good idea to avoid WD NVMe drives too. There are a large number of people who've reported problems with ZFS (and btrfs) with WD SN770 and a few other WD models: • https://github.com/openzfs/zfs/discussions/14793 https://github.com/openzfs/zfs/discussions/14793
kkfx 2y ago
Well... The issue with encrypted zfs + raw send is that a pool encrypted with a common key for all volume became an individual key per volume, a non-RAW send means your target read your files. If you use a keyfile this is a non-issue. If you type your key, well, you import all the old volumes, create a new pool and send them re-encrypting them with a common key. Very raw but doable at home scale setups.

32 points by aargh_aargh 2y ago

l2dy 2y ago

> I don’t back up my drives, I replicate them. Don't take this as a general advice. For important data, it's important to have multiple backups and validate their effectiveness routinely.

chromakode 2y ago
It's part of the 3-2-1 rule: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ My mid term goal is to trade offsite replication with a friend for automatically replicating the offsite portion of important things.
- toastal 2y ago
  That works assuming you have the budget & technical friends
  - chromakode 2y ago
    Residential fiber internet is a lovely thing :D
diarrhea 2y ago
I feel like a filesystem with native snapshot support (zfs, btrfs), replicated once (also native in zfs) obsoletes conventional 3-2-1 backup systems. It’s technically only 2 versions, but you’re protected from all the same failure modes.
- aborsy 2y ago
  If there is a bug in file system, both source and its replication can be lost. I will add to this a different backup software to an offsite or cloud. I use restic.
  - raverbashing 2y ago
    Especially in a fs that falls apart if you look at it wrong like ZFS
    - sshine 2y ago
      How does zfs fall apart? I’ve run it beneath Ubuntu for several years, and it’s only saved my ass. I didn’t replicate, only used the apt snapshots.
      - justinclift 2y ago
        It's quite possible to get ZFS into weird states without too much effort when you're screwing around with the underlying devices (adding, deleting, changing things). This seems to crop up at really inconvenient times too, like when you're trying to do something during a scheduled outage. :( That kind of thing aside though, it's been pretty solid in my use for actual data storage. Just don't use ZFS's native encryption + ZFS snapshots + send/recv. Reportedly that combination is a cause of data corruption: https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt
    - stiray 2y ago
      Heh, you must really have foul look, I am using it on FreeBSD since 2007ish and never had any issues. Same pool, no longer the same disks as they were all replaced since. Also the hardware around it was replaced and FreeBSD 8 updated to all new versions, currently running 14.0. Never lost any data. There isn't much software that could claim that. Meanwhile I lost (I had backups) all my data twice on btrfs, I know it is more stable now, but I certainly wont ever use it again. Even HAMMER1 (I would love to use HAMMER2, but until my server dies, I will stay with FreeBSD) lost it only once and even in that case, after debugging irc session with Matt Dillon, I was able to recover most files. The only thing that pisses me off is that Kubuntu doesnt support it trough installer (yes I could do it manually but been there with Fedora and I am sick of tracking if initfs was updated or finish with unbootable - easy solvable, but annoying - situation ) and I am now forced to use Ubuntu with KDE on workstation. But this is not zfs.
- tjoff 2y ago
  That doesn't make any sense. Native snapshots are fantastic, but are merely an effective way to do backups. One where you trade some complexity from the backup software to the filesystem. (only talking backups, snapshots of course have utility beyond that usecase as well)
- prmoustache 2y ago
  Hmmm from what I understand zrepl can do copy of source to backup both in a push or pull mode. I would say the push mode is still a very fragile way to do backups. Imagine source machine is compromised and attacker decide to delete/encrypt your data, and see there is a backup mechanism connecting to the backup machine, what prevents him from using the deleting/encrypting the backups as well? You'd definitely want the backup machine to pull the snapshots, and have no way to connect to it from source machine directly with a user that would have access to the data or an admin account. That means no ssh keys on the source machine, no password kept in a password manager that would be loaded on the source machine either. Another strong method would involve 3 machines: source --push--> replica1 <--pull-- replica2 Where source and replica1 would have ZFS filesystem and snapshots while replica2 is using a different filesystem (LVM + ext4 ) and snapshots to safeguard from replicating bugs that lead to data not being available. ZFS snapshots could be saved as individual files on this filesystem.
  - justinclift 2y ago
    > ... what prevents him from using the deleting/encrypting the backups as well? With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. > That means no ssh keys on the source machine ... Typically for non-user logins (eg script access and similar) you do the extra step of configuring the receiving ssh to only allow a specific command for a given key. It's a configurable ssh thing, where you add extra info to the .ssh/authorized_keys file on the destination server. With that approach, it doesn't allow general user logins while still allowing the source machine to send the data.
    - prmoustache 2y ago
      > With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. Only and only if you apply best practicies mentionned above in the second point of your post. Anyway I'd rather protect my backup as much as I can and not allow the source machine to have any direct access to an account on the backup server because of possible security. Security is hard sometimes and you never know when some bugs might increase the possibilities. I like my backup servers to not have any open port. The caveat is that it is only possible if your primary backup server is locally and physically accessible. If you are travelling you might want to be able to access it without being physically present. An option might be to just have ssh disabled when you are at home and you enable the service when you know you won't be at home for a long enough period to make it a problem if you need to restore data.
aargh_aargh 2y ago
One case where this might go wrong is if you accidentally delete a file, the deletion replicates and you're left without a backup. Replication is nice but it is not a replacement for backup.
- mhio 2y ago
  The setup in the article replicates regular incremental snapshots rather than file system changes. ZFS snapshots allow you to roll back to the point in time of each snapshot, up to the amount of disk space you can maintain for storing the changes.

chromakode 2y ago

Funny story -- when I was working on the xkcd Machine comic, I actually used the ZFS snapshots to rescue data. I accidentally blew away some early physics prototype code and fished it out of /.zfs/snapshot.

sshine 2y ago
Thanks for advertising the machine. I hadn’t seen it yet. It’s amazing!
- aargh_aargh 2y ago
  And here's the dev blogpost, hot off the press! https://news.ycombinator.com/item?id=40300454 https://news.ycombinator.com/item?id=40300454

totetsu 2y ago

My face palm moment of this year was accidentally restoring a zfs snapshot of my root pool form a week ago, but it was actually a year and a week ago. Didn’t lose any of my data, but suddenly I had some offering to format their version mismatched databases.

deleted 2y ago

[deleted]

tie-in 2y ago

You don't actually need a dedicated ZFS backup program. A simple cron script will handle incremental backups just fine. If anyone is interested, the script we use to backup our multi-TB PostgreSQL database can be found here: https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po...

justinclift 2y ago

It sounds like the author is using ZFS encryption. If that's the case, then doing so with replicated ZFS snapshots is probably not a good idea. That specific scenario (ZFS encryption -> replication of encrypted snapshots) is a known cause of ZFS corruption. :( https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt Unfortunately it doesn't seem to be widely known about, though there is a suggestion to make it official: https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/openzfs-docs/issues/494

justinclift 2y ago

For ZFS use it's probably a good idea to avoid WD NVMe drives too. There are a large number of people who've reported problems with ZFS (and btrfs) with WD SN770 and a few other WD models: • https://github.com/openzfs/zfs/discussions/14793 https://github.com/openzfs/zfs/discussions/14793

kkfx 2y ago

Well... The issue with encrypted zfs + raw send is that a pool encrypted with a common key for all volume became an individual key per volume, a non-RAW send means your target read your files. If you use a keyfile this is a non-issue. If you type your key, well, you import all the old volumes, create a new pool and send them re-encrypting them with a common key. Very raw but doable at home scale setups.