3 ms·

> I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. No. I need

by jusssi 6y ago

> I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. No. I need a backup that consistently finishes before I close the lid to put my laptop in my backpack at the end of the day. Too slow -> no backup.

_flux 6y ago
Unless daily work datasets are huge, you would be hard pressed to find a backup system that is unable to pull that off, though it's unclear from your description when you consider your backup to start. My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.
- jusssi 6y ago
  Many dev environments come in virtual machine images. On a good day they're throwaway Vagrant builds, but sometimes there's a significant manual setup required and you want to include them in a backup. Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.
  - wila 6y ago
    There's backup tools that can make a complete backup of your VM while it is still running. At least for VMware virtual machines [0] You can schedule the backup to start some time before you leave at the end of the day. [0] https://www.vimalin.com https://www.vimalin.com disclaimer: I am the author of vimalin
  - _flux 6y ago
    I think Kopia should deal fine with this, though I don't think I have many such large files that I modify. The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks. Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..) So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.
  - derekp7 6y ago
    Why would you sync the VM image instead of doing a file level backup from within the VM? File-level backups lets you control include/exclude lists (no need to back up /var/tmp for example, or exclude *.dbf files as they should be backed up with an appropriate database plugin that does a hot base backup, then grabs transaction logs afterwards).
    - jusssi 6y ago
      The main purpose of my backup is for getting back to doing invoiceable hours after the statistically inevitable hardware failure or misplacement (I've yet to lose a laptop, fingers crossed). It's not that the data on those VMs is irreplaceable, any significant work output will be in a VCS outside my laptop anyway. It's that rebuilding it from pieces in case of loss requires work that no client will pay for. That being the case, I much prefer being able to re-image my home dir partition into a new laptop/ssd and get back to paying work, over to having to fiddle with individual VMs that only exist as partial backups. Edit: I've actually considered the opposite of what you suggest, just doing a raw partition image for backup. SSD is really fast, ethernet is fast, and the runtime would be consistent. Just haven't got it out of me to try it in practice.

by jusssi 6y ago

_flux 6y ago

Unless daily work datasets are huge, you would be hard pressed to find a backup system that is unable to pull that off, though it's unclear from your description when you consider your backup to start. My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.

jusssi 6y ago
Many dev environments come in virtual machine images. On a good day they're throwaway Vagrant builds, but sometimes there's a significant manual setup required and you want to include them in a backup. Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.
- wila 6y ago
  There's backup tools that can make a complete backup of your VM while it is still running. At least for VMware virtual machines [0] You can schedule the backup to start some time before you leave at the end of the day. [0] https://www.vimalin.com https://www.vimalin.com disclaimer: I am the author of vimalin
- _flux 6y ago
  I think Kopia should deal fine with this, though I don't think I have many such large files that I modify. The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks. Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..) So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.
- derekp7 6y ago
  Why would you sync the VM image instead of doing a file level backup from within the VM? File-level backups lets you control include/exclude lists (no need to back up /var/tmp for example, or exclude *.dbf files as they should be backed up with an appropriate database plugin that does a hot base backup, then grabs transaction logs afterwards).
  - jusssi 6y ago
    The main purpose of my backup is for getting back to doing invoiceable hours after the statistically inevitable hardware failure or misplacement (I've yet to lose a laptop, fingers crossed). It's not that the data on those VMs is irreplaceable, any significant work output will be in a VCS outside my laptop anyway. It's that rebuilding it from pieces in case of loss requires work that no client will pay for. That being the case, I much prefer being able to re-image my home dir partition into a new laptop/ssd and get back to paying work, over to having to fiddle with individual VMs that only exist as partial backups. Edit: I've actually considered the opposite of what you suggest, just doing a raw partition image for backup. SSD is really fast, ethernet is fast, and the runtime would be consistent. Just haven't got it out of me to try it in practice.