3 ms·

I need a backup system in which files are never deleted, even if they are deleted locally. These files should also be easy and efficient to search for; you shou

by sa1 5y ago

I need a backup system in which files are never deleted, even if they are deleted locally. These files should also be easy and efficient to search for; you shouldn't have to go through all previous snapshots, trying to guess which snapshot a file might be in. Does restic do this?

colordrops 5y ago
Yes on deletes, backups are incremental. Not sure about how easy to find files but I'd be surprised if there weren't commands for doing this.
- sa1 5y ago
  I'm not sure, but seems to be an expensive operation on snapshot based architectures - have to search through all snapshots.
  - lucb1e 5y ago
    If you know the path, it would be $ restic mount --repo /mnt/backupdrive/myrepo /mnt/backedup $ echo /mnt/backedup/snapshots/*/path/to/file And it would print which snapshots it is contained in. If you don't know the path, I don't know how long `find -type f` takes so you might be right about that being inefficient. It is certainly not a use-case that I think restic ever had in mind. I also don't know of "backup" software to solve this, it seems a bit out of scope for most of them (they are meant to have a backup copy of your disk, not be a file manager with history). You might have more luck with tools like rsync (or Toucan, from the good old days where I used Windows and Portable Apps), those can certainly create only new files and never delete deleted files, but then you typically don't get encryption and deduplication.
    - sa1 5y ago
      Thanks, I’ve also thought that something built on rclone or rsync might be more suitable. I haven’t yet moved to restic, so it might be that find -f is good enough for me.
donatzsky 5y ago
It's a backup tool, so it won't delete anything unless you tell it to.
- sa1 5y ago
  Yes, I should have specified that the emphasis is on the use-case of finding and restoring old files. Yes, a backup tool never deletes by itself, but the standard way of using such tools is that you keep the last n snapshots - this is just delayed sync. I don't know anything about backup architectures, but is it possible to be more efficient with storage and indexing(for search) if your use-case includes finding files from 5 years ago, but not necessarily recreate the whole filesystem as it was on some date?
  - donatzsky 5y ago
    > Yes, a backup tool never deletes by itself, but the standard way of using such tools is that you keep the last n snapshots - this is just delayed sync. Restic, Kopia and most other modern backup solutions allow you to to define the retention policy. Usually by specifying how many hourly/daily/weekly/monthly/yearly snapshots to keep. They don't just remove all snapshots that are older than n. They also usually let you restore any given file, from any given snapshot, without having to restore everything. And as long as there's a functional index, this shouldn't be terribly slow either.
    - sa1 5y ago
      Thanks. If a file existed for 3 days and was then deleted, it might make it to 3 daily snapshots, and no weekly, monthly or yearly snapshot. Even if you’re lucky and the snapshots are not deleted, if you don’t remember when the file existed, currently you need to search through all snapshots. There’s no index or any data structure to speed this search up. I agree, restoring is fast once you’ve found the file. Currently I use backblaze, and it does have infinite retention and snapshots, and restoring is fast. But I’m looking to migrate to something that gives me search as well.
johntash 5y ago
My backup script runs a 'restic diff' command after every snapshot it takes and puts it in a log file. It's basically a list of every file added, removed, or modified in that snapshot. Since I have a directory full of these logs, I can search for deleted files by doing something like 'grep "^- " restic-diff-*.log' I'm not sure if there's a better way to do this, but it works pretty well for me.
- sa1 5y ago
  Thanks, this might work.

by sa1 5y ago

colordrops 5y ago

Yes on deletes, backups are incremental. Not sure about how easy to find files but I'd be surprised if there weren't commands for doing this.

sa1 5y ago
I'm not sure, but seems to be an expensive operation on snapshot based architectures - have to search through all snapshots.
- lucb1e 5y ago
  If you know the path, it would be $ restic mount --repo /mnt/backupdrive/myrepo /mnt/backedup $ echo /mnt/backedup/snapshots/*/path/to/file And it would print which snapshots it is contained in. If you don't know the path, I don't know how long `find -type f` takes so you might be right about that being inefficient. It is certainly not a use-case that I think restic ever had in mind. I also don't know of "backup" software to solve this, it seems a bit out of scope for most of them (they are meant to have a backup copy of your disk, not be a file manager with history). You might have more luck with tools like rsync (or Toucan, from the good old days where I used Windows and Portable Apps), those can certainly create only new files and never delete deleted files, but then you typically don't get encryption and deduplication.
  - sa1 5y ago
    Thanks, I’ve also thought that something built on rclone or rsync might be more suitable. I haven’t yet moved to restic, so it might be that find -f is good enough for me.

donatzsky 5y ago

It's a backup tool, so it won't delete anything unless you tell it to.

sa1 5y ago
Yes, I should have specified that the emphasis is on the use-case of finding and restoring old files. Yes, a backup tool never deletes by itself, but the standard way of using such tools is that you keep the last n snapshots - this is just delayed sync. I don't know anything about backup architectures, but is it possible to be more efficient with storage and indexing(for search) if your use-case includes finding files from 5 years ago, but not necessarily recreate the whole filesystem as it was on some date?
- donatzsky 5y ago
  > Yes, a backup tool never deletes by itself, but the standard way of using such tools is that you keep the last n snapshots - this is just delayed sync. Restic, Kopia and most other modern backup solutions allow you to to define the retention policy. Usually by specifying how many hourly/daily/weekly/monthly/yearly snapshots to keep. They don't just remove all snapshots that are older than n. They also usually let you restore any given file, from any given snapshot, without having to restore everything. And as long as there's a functional index, this shouldn't be terribly slow either.
  - sa1 5y ago
    Thanks. If a file existed for 3 days and was then deleted, it might make it to 3 daily snapshots, and no weekly, monthly or yearly snapshot. Even if you’re lucky and the snapshots are not deleted, if you don’t remember when the file existed, currently you need to search through all snapshots. There’s no index or any data structure to speed this search up. I agree, restoring is fast once you’ve found the file. Currently I use backblaze, and it does have infinite retention and snapshots, and restoring is fast. But I’m looking to migrate to something that gives me search as well.

johntash 5y ago

My backup script runs a 'restic diff' command after every snapshot it takes and puts it in a log file. It's basically a list of every file added, removed, or modified in that snapshot. Since I have a directory full of these logs, I can search for deleted files by doing something like 'grep "^- " restic-diff-*.log' I'm not sure if there's a better way to do this, but it works pretty well for me.

sa1 5y ago
Thanks, this might work.