4 ms·

> I don’t back up my drives, I replicate them. Don't take this as a general advice. For important data, it's important to have multiple backups and validate th

by l2dy 2y ago

> I don’t back up my drives, I replicate them. Don't take this as a general advice. For important data, it's important to have multiple backups and validate their effectiveness routinely.

chromakode 2y ago
It's part of the 3-2-1 rule: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ My mid term goal is to trade offsite replication with a friend for automatically replicating the offsite portion of important things.
- toastal 2y ago
  That works assuming you have the budget & technical friends
  - chromakode 2y ago
    Residential fiber internet is a lovely thing :D
diarrhea 2y ago
I feel like a filesystem with native snapshot support (zfs, btrfs), replicated once (also native in zfs) obsoletes conventional 3-2-1 backup systems. It’s technically only 2 versions, but you’re protected from all the same failure modes.
- aborsy 2y ago
  If there is a bug in file system, both source and its replication can be lost. I will add to this a different backup software to an offsite or cloud. I use restic.
  - raverbashing 2y ago
    Especially in a fs that falls apart if you look at it wrong like ZFS
    - sshine 2y ago
      How does zfs fall apart? I’ve run it beneath Ubuntu for several years, and it’s only saved my ass. I didn’t replicate, only used the apt snapshots.
      - justinclift 2y ago
        It's quite possible to get ZFS into weird states without too much effort when you're screwing around with the underlying devices (adding, deleting, changing things). This seems to crop up at really inconvenient times too, like when you're trying to do something during a scheduled outage. :( That kind of thing aside though, it's been pretty solid in my use for actual data storage. Just don't use ZFS's native encryption + ZFS snapshots + send/recv. Reportedly that combination is a cause of data corruption: https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt
    - stiray 2y ago
      Heh, you must really have foul look, I am using it on FreeBSD since 2007ish and never had any issues. Same pool, no longer the same disks as they were all replaced since. Also the hardware around it was replaced and FreeBSD 8 updated to all new versions, currently running 14.0. Never lost any data. There isn't much software that could claim that. Meanwhile I lost (I had backups) all my data twice on btrfs, I know it is more stable now, but I certainly wont ever use it again. Even HAMMER1 (I would love to use HAMMER2, but until my server dies, I will stay with FreeBSD) lost it only once and even in that case, after debugging irc session with Matt Dillon, I was able to recover most files. The only thing that pisses me off is that Kubuntu doesnt support it trough installer (yes I could do it manually but been there with Fedora and I am sick of tracking if initfs was updated or finish with unbootable - easy solvable, but annoying - situation ) and I am now forced to use Ubuntu with KDE on workstation. But this is not zfs.
- tjoff 2y ago
  That doesn't make any sense. Native snapshots are fantastic, but are merely an effective way to do backups. One where you trade some complexity from the backup software to the filesystem. (only talking backups, snapshots of course have utility beyond that usecase as well)
- prmoustache 2y ago
  Hmmm from what I understand zrepl can do copy of source to backup both in a push or pull mode. I would say the push mode is still a very fragile way to do backups. Imagine source machine is compromised and attacker decide to delete/encrypt your data, and see there is a backup mechanism connecting to the backup machine, what prevents him from using the deleting/encrypting the backups as well? You'd definitely want the backup machine to pull the snapshots, and have no way to connect to it from source machine directly with a user that would have access to the data or an admin account. That means no ssh keys on the source machine, no password kept in a password manager that would be loaded on the source machine either. Another strong method would involve 3 machines: source --push--> replica1 <--pull-- replica2 Where source and replica1 would have ZFS filesystem and snapshots while replica2 is using a different filesystem (LVM + ext4 ) and snapshots to safeguard from replicating bugs that lead to data not being available. ZFS snapshots could be saved as individual files on this filesystem.
  - justinclift 2y ago
    > ... what prevents him from using the deleting/encrypting the backups as well? With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. > That means no ssh keys on the source machine ... Typically for non-user logins (eg script access and similar) you do the extra step of configuring the receiving ssh to only allow a specific command for a given key. It's a configurable ssh thing, where you add extra info to the .ssh/authorized_keys file on the destination server. With that approach, it doesn't allow general user logins while still allowing the source machine to send the data.
    - prmoustache 2y ago
      > With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. Only and only if you apply best practicies mentionned above in the second point of your post. Anyway I'd rather protect my backup as much as I can and not allow the source machine to have any direct access to an account on the backup server because of possible security. Security is hard sometimes and you never know when some bugs might increase the possibilities. I like my backup servers to not have any open port. The caveat is that it is only possible if your primary backup server is locally and physically accessible. If you are travelling you might want to be able to access it without being physically present. An option might be to just have ssh disabled when you are at home and you enable the service when you know you won't be at home for a long enough period to make it a problem if you need to restore data.
aargh_aargh 2y ago
One case where this might go wrong is if you accidentally delete a file, the deletion replicates and you're left without a backup. Replication is nice but it is not a replacement for backup.
- mhio 2y ago
  The setup in the article replicates regular incremental snapshots rather than file system changes. ZFS snapshots allow you to roll back to the point in time of each snapshot, up to the amount of disk space you can maintain for storing the changes.

by l2dy 2y ago

> I don’t back up my drives, I replicate them. Don't take this as a general advice. For important data, it's important to have multiple backups and validate their effectiveness routinely.

chromakode 2y ago

It's part of the 3-2-1 rule: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ My mid term goal is to trade offsite replication with a friend for automatically replicating the offsite portion of important things.

toastal 2y ago
That works assuming you have the budget & technical friends
- chromakode 2y ago
  Residential fiber internet is a lovely thing :D

diarrhea 2y ago

I feel like a filesystem with native snapshot support (zfs, btrfs), replicated once (also native in zfs) obsoletes conventional 3-2-1 backup systems. It’s technically only 2 versions, but you’re protected from all the same failure modes.

aborsy 2y ago
If there is a bug in file system, both source and its replication can be lost. I will add to this a different backup software to an offsite or cloud. I use restic.
- raverbashing 2y ago
  Especially in a fs that falls apart if you look at it wrong like ZFS
  - sshine 2y ago
    How does zfs fall apart? I’ve run it beneath Ubuntu for several years, and it’s only saved my ass. I didn’t replicate, only used the apt snapshots.
    - justinclift 2y ago
      It's quite possible to get ZFS into weird states without too much effort when you're screwing around with the underlying devices (adding, deleting, changing things). This seems to crop up at really inconvenient times too, like when you're trying to do something during a scheduled outage. :( That kind of thing aside though, it's been pretty solid in my use for actual data storage. Just don't use ZFS's native encryption + ZFS snapshots + send/recv. Reportedly that combination is a cause of data corruption: https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt
  - stiray 2y ago
    Heh, you must really have foul look, I am using it on FreeBSD since 2007ish and never had any issues. Same pool, no longer the same disks as they were all replaced since. Also the hardware around it was replaced and FreeBSD 8 updated to all new versions, currently running 14.0. Never lost any data. There isn't much software that could claim that. Meanwhile I lost (I had backups) all my data twice on btrfs, I know it is more stable now, but I certainly wont ever use it again. Even HAMMER1 (I would love to use HAMMER2, but until my server dies, I will stay with FreeBSD) lost it only once and even in that case, after debugging irc session with Matt Dillon, I was able to recover most files. The only thing that pisses me off is that Kubuntu doesnt support it trough installer (yes I could do it manually but been there with Fedora and I am sick of tracking if initfs was updated or finish with unbootable - easy solvable, but annoying - situation ) and I am now forced to use Ubuntu with KDE on workstation. But this is not zfs.
tjoff 2y ago
That doesn't make any sense. Native snapshots are fantastic, but are merely an effective way to do backups. One where you trade some complexity from the backup software to the filesystem. (only talking backups, snapshots of course have utility beyond that usecase as well)
prmoustache 2y ago
Hmmm from what I understand zrepl can do copy of source to backup both in a push or pull mode. I would say the push mode is still a very fragile way to do backups. Imagine source machine is compromised and attacker decide to delete/encrypt your data, and see there is a backup mechanism connecting to the backup machine, what prevents him from using the deleting/encrypting the backups as well? You'd definitely want the backup machine to pull the snapshots, and have no way to connect to it from source machine directly with a user that would have access to the data or an admin account. That means no ssh keys on the source machine, no password kept in a password manager that would be loaded on the source machine either. Another strong method would involve 3 machines: source --push--> replica1 <--pull-- replica2 Where source and replica1 would have ZFS filesystem and snapshots while replica2 is using a different filesystem (LVM + ext4 ) and snapshots to safeguard from replicating bugs that lead to data not being available. ZFS snapshots could be saved as individual files on this filesystem.
- justinclift 2y ago
  > ... what prevents him from using the deleting/encrypting the backups as well? With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. > That means no ssh keys on the source machine ... Typically for non-user logins (eg script access and similar) you do the extra step of configuring the receiving ssh to only allow a specific command for a given key. It's a configurable ssh thing, where you add extra info to the .ssh/authorized_keys file on the destination server. With that approach, it doesn't allow general user logins while still allowing the source machine to send the data.
  - prmoustache 2y ago
    > With ZFS snapshots the older snapshots would still be present on the target server, in their unencrypted form. Only and only if you apply best practicies mentionned above in the second point of your post. Anyway I'd rather protect my backup as much as I can and not allow the source machine to have any direct access to an account on the backup server because of possible security. Security is hard sometimes and you never know when some bugs might increase the possibilities. I like my backup servers to not have any open port. The caveat is that it is only possible if your primary backup server is locally and physically accessible. If you are travelling you might want to be able to access it without being physically present. An option might be to just have ssh disabled when you are at home and you enable the service when you know you won't be at home for a long enough period to make it a problem if you need to restore data.

aargh_aargh 2y ago

One case where this might go wrong is if you accidentally delete a file, the deletion replicates and you're left without a backup. Replication is nice but it is not a replacement for backup.

mhio 2y ago
The setup in the article replicates regular incremental snapshots rather than file system changes. ZFS snapshots allow you to roll back to the point in time of each snapshot, up to the amount of disk space you can maintain for storing the changes.