4 ms·

LVM snapshots are copy on write and can be used the same way.

by lmz 2y ago

magicalhippo 2y ago
Any backup software that utilizes LVM in this way? Ie automatically creates a snapshot and sends the incremental changes since previous snapshot to a backup destination like a NAS or S3 blob storage.
- lmz 2y ago
  I don't think the diffs are usable that way. They're actually more like an "undo log" in that the snapshot space is taken by "old blocks" when the actual volume is taking writes. It's useful for the same reasons as volume shadow copy: a consistent snapshot of the block device. (Also this can be very bad for write performance as any writes are doubled - to snapshot and to to the real device)
  - magicalhippo 2y ago
    Yeah ok, that makes sense. Write performance is a concern, but usually the backups run when there's little activity.
- _flux 2y ago
  I think block-level snapshots would be very difficult to use this way. I just make a full dedupped backups from LVM snapshots with kopia, but I've set that up only on one system, on others I just use kopia as-is. It takes some time, but that's fine for me. Previous backup of 25 GB an hour ago took 20 minutes. I suppose if it only walked files it knew were changed it would be a lot faster.
  - magicalhippo 2y ago
    Thanks, sounds interesting. So you create a snapshot, then let kopia process that snapshot rather than the live filesystem, and then remove the snapshot? > I suppose if it only walked files it knew were changed it would be a lot faster. Right, for me I'd want to set it up to do the full disk, so could be millions of files and hundreds of GB. But this trick should work with other backups software, so perhaps it's a viable option.
    - _flux 2y ago
      Exactly so. Here's the script, should it be of benefit to someone, even if it of course needs to be modified: #!/bin/sh success=false teardown() { umount /mnt/backup/var/lib/docker || true umount /mnt/backup/root/.cache || true umount /mnt/backup/ || true for lv in root docker-data; do lvremove --yes /dev/hass-vg/$lv-snapshot || true done if [ "$1" != "no-exit" ]; then $success exit $? fi } set -x set -e teardown no-exit trap teardown EXIT for lv in root docker-data; do lvcreate --snapshot -L 1G -n $lv-snapshot /dev/hass-vg/$lv done mount /dev/hass-vg/root-snapshot /mnt/backup mount /dev/hass-vg/docker-data-snapshot /mnt/backup/var/lib/docker mount /root/.cache /mnt/backup/root/.cache -o bind chroot /mnt/backup kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create / /var/lib/docker kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create /boot /boot/efi success=true
      - magicalhippo 2y ago
        Awesome, thanks!
- abbbi 2y ago
  wyng backup does this. It uses the device mappers thin_dump tools to allow for incremental backups between snapshots, too: https://github.com/tasket/wyng-backup https://github.com/tasket/wyng-backup edit: requires lvm thin provisioned volumes There is also thin-send-recv which basically does the same as zfs send/recv just with lvm: https://github.com/LINBIT/thin-send-recv https://github.com/LINBIT/thin-send-recv it uses the same functions of the device mapper to allow incremental sync of lvm thin volumes.
  - magicalhippo 2y ago
    Thanks for the pointers, looks very relevant. It's just such a low-effort peace of mind. Just a few clicks and I know that regardless what happens to my disk or my system, I can be up and running in very little time with very little effort. On Linux it's always a bit more work, but backups and restore is one of those things I prefer is not too complicated, as stress level is usually high enough when you need to do restore to worry about forgetting some incantation steps.
    - abbbi 2y ago
      it depends. Doing a complete disaster recovery of a windows system IMHO can be a real struggle. Especially if you have to restore a system to different hardware, which the system state backup that microsoft offers does not support afaik. Backing up a linux system in combination with REAR: https://github.com/rear/rear https://github.com/rear/rear and a backup utility of your choice for the regular backup has never failed me so far. I used it to restore linux systems to complete different hardware without any troubles.
      - magicalhippo 2y ago
        For my cases it's been quite easy, but then I've mostly had quite plain hardware so didn't need vendor drivers to recover. While I've had to recover in anger twice, I've used the same procedure to migrate to new hardware many times. Just restore to the new disk in the new machine, and let Windows reboot a few times and off I went. REAR looks useful, hadn't seen that before.

LVM snapshots are copy on write and can be used the same way.

by lmz 2y ago

LVM snapshots are copy on write and can be used the same way.

Any backup software that utilizes LVM in this way? Ie automatically creates a snapshot and sends the incremental changes since previous snapshot to a backup destination like a NAS or S3 blob storage.

lmz 2y ago
I don't think the diffs are usable that way. They're actually more like an "undo log" in that the snapshot space is taken by "old blocks" when the actual volume is taking writes. It's useful for the same reasons as volume shadow copy: a consistent snapshot of the block device. (Also this can be very bad for write performance as any writes are doubled - to snapshot and to to the real device)
- magicalhippo 2y ago
  Yeah ok, that makes sense. Write performance is a concern, but usually the backups run when there's little activity.
_flux 2y ago
I think block-level snapshots would be very difficult to use this way. I just make a full dedupped backups from LVM snapshots with kopia, but I've set that up only on one system, on others I just use kopia as-is. It takes some time, but that's fine for me. Previous backup of 25 GB an hour ago took 20 minutes. I suppose if it only walked files it knew were changed it would be a lot faster.
- magicalhippo 2y ago
  Thanks, sounds interesting. So you create a snapshot, then let kopia process that snapshot rather than the live filesystem, and then remove the snapshot? > I suppose if it only walked files it knew were changed it would be a lot faster. Right, for me I'd want to set it up to do the full disk, so could be millions of files and hundreds of GB. But this trick should work with other backups software, so perhaps it's a viable option.
  - _flux 2y ago
    Exactly so. Here's the script, should it be of benefit to someone, even if it of course needs to be modified: #!/bin/sh success=false teardown() { umount /mnt/backup/var/lib/docker || true umount /mnt/backup/root/.cache || true umount /mnt/backup/ || true for lv in root docker-data; do lvremove --yes /dev/hass-vg/$lv-snapshot || true done if [ "$1" != "no-exit" ]; then $success exit $? fi } set -x set -e teardown no-exit trap teardown EXIT for lv in root docker-data; do lvcreate --snapshot -L 1G -n $lv-snapshot /dev/hass-vg/$lv done mount /dev/hass-vg/root-snapshot /mnt/backup mount /dev/hass-vg/docker-data-snapshot /mnt/backup/var/lib/docker mount /root/.cache /mnt/backup/root/.cache -o bind chroot /mnt/backup kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create / /var/lib/docker kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create /boot /boot/efi success=true
    - magicalhippo 2y ago
      Awesome, thanks!
abbbi 2y ago
wyng backup does this. It uses the device mappers thin_dump tools to allow for incremental backups between snapshots, too: https://github.com/tasket/wyng-backup https://github.com/tasket/wyng-backup edit: requires lvm thin provisioned volumes There is also thin-send-recv which basically does the same as zfs send/recv just with lvm: https://github.com/LINBIT/thin-send-recv https://github.com/LINBIT/thin-send-recv it uses the same functions of the device mapper to allow incremental sync of lvm thin volumes.
- magicalhippo 2y ago
  Thanks for the pointers, looks very relevant. It's just such a low-effort peace of mind. Just a few clicks and I know that regardless what happens to my disk or my system, I can be up and running in very little time with very little effort. On Linux it's always a bit more work, but backups and restore is one of those things I prefer is not too complicated, as stress level is usually high enough when you need to do restore to worry about forgetting some incantation steps.
  - abbbi 2y ago
    it depends. Doing a complete disaster recovery of a windows system IMHO can be a real struggle. Especially if you have to restore a system to different hardware, which the system state backup that microsoft offers does not support afaik. Backing up a linux system in combination with REAR: https://github.com/rear/rear https://github.com/rear/rear and a backup utility of your choice for the regular backup has never failed me so far. I used it to restore linux systems to complete different hardware without any troubles.
    - magicalhippo 2y ago
      For my cases it's been quite easy, but then I've mostly had quite plain hardware so didn't need vendor drivers to recover. While I've had to recover in anger twice, I've used the same procedure to migrate to new hardware many times. Just restore to the new disk in the new machine, and let Windows reboot a few times and off I went. REAR looks useful, hadn't seen that before.