8 ms·

Duplicity: Encrypted bandwidth-efficient backup

143 points by GTP 3y ago

diekhans 3y ago
I have been having great luck with incremental backups with the very similar named Duplicacy https://duplicacy.com/ https://duplicacy.com/
AdmiralAsshat 3y ago
Brilliant name, if you think about it. If they ever decided to start doing shady shit, they'd have a perfect legal shield. No one would be able to convincingly argue in court that they were being duplicitous.
- mlyle 3y ago
  If no one can argue they're duplicitous, then it's a case of false advertising...
longwave 3y ago
I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/ https://www.borgbackup.org/
- giamma 3y ago
  Same for me. Also, on MacOs duplicity was consuming much more CPU than Borg and was causing my fan to spin loudly. Eventually I moved to timemachine, but I still consider Borg a very good option.
- zzzeek 3y ago
  I have an overnight cron that flattens my duplicity backups from many incremental backups made over the course of one day to a single full backup file, that becomes the new backup. then subsequent backups over the course of the day do incremental on that file. So I always have full backups for each individual day with only a dozen or so incremental backups tacked onto it. that said will give Borg a look
- tussa 3y ago
  I did the same. I had some weird path issues with Duplicity. Borg is now my holy backup grail. Wish I could backup incrementally to AWS glacier storage but that just me sounding like an ungrateful begger. I'm incredibly grateful and happy with Borg!
- sigio 3y ago
  Agree completely... used duplicity many years ago, but switched to Borg and never looked back. Currently doing borg-backups of quite a lot of systems, many every 6 hours, and some, like my main shell-host every 2 hours. It's quick, tiny and easy... and restores are the easiest, just mount the backup, browse the snapshot, and copy files where needed.
  - stavros 3y ago
    After Borg, I switched to Restic: https://restic.net/ https://restic.net/ AFAIK, the only difference is that Restic doesn't require Restic installed on the remote server, so you can efficiently backup to things like S3 or FTP. Other than that, both are fantastic.
    - pdimitar 3y ago
      Technically Borg doesn't require it either, you can backup to a local directory and then use `rclone` to upload the repo wherever. Not practical for huge backups but it works for me as I'm backing up my machines configuration and code directories only. ~60MB, and that includes a lot of code and some data (SQL, JSON et. al.)
      - stavros 3y ago
        Sure, but rsync requires the server to support rsync.
        pdimitar 3y ago
        Does rclone require rsync? Haven't checked.
        stavros 3y ago
        Oh sorry, brain fart, I thought you said rsync. I think rclone uploads everything if you don't have rclone on the server, but I'm not sure.
- shaunpud 3y ago
  Works a treat with borgmatic https://torsion.org/borgmatic https://torsion.org/borgmatic
- GTP 3y ago
  Also duplicity let's you automatically delete backups older than a certain amount of time, what is the difference?
jwr 3y ago
Excellent piece of software, and relatively simple to use with gpg encryption. I've been using it for many years. My only complaint is that, like a lot of software written in Python, it has no regard for traditional UNIX behavior (keep quiet unless you have something meaningful to say), so I have to live with cron reporting stuff like: "/usr/lib/python2.7/dist-packages/paramiko/rsakey.py:99: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. algorithm=hashes.SHA1()" along with stuff I actually do (or might) care about. Oh well.
- tynorf 3y ago
  You can try setting `export PYTHONWARNINGS="ignore"` to suppress warnings.
- duskwuff 3y ago
  > /usr/lib/python2.7/ You are using an old version of Duplicity. It dropped all support for Python 2 in 2022: https://git.launchpad.net/duplicity/commit/setup.py?id=5505fb51a489a8306901a14d93c7aaec5dec41b2 https://git.launchpad.net/duplicity/commit/setup.py?id=5505f...
65 3y ago
If you're using S3 to back up your files, it's easier to write a shell script with the AWS CLI. For example here's a script I wrote that I run automatically to back up my computer to S3. I have an exclude array to exclude certain folders. It's simpler than downloading software and more customizable. # $1 # local folder # $2 # bucket declare -a exclude=( "node_modules" "Applications" "Public" ) args="" for item in "${exclude[@]}"; do args+=" --exclude '*/$item/*' --exclude '$item/*'"; done cmd="aws s3 sync '$1' 's3://$2$1' --include '*' $args" eval "$cmd"
- rakoo 3y ago
  Your script doesn't do the same thing as duplicity. Your script mirrors the local directory with your bucket. It loses all history. Duplicity does backups (ie with history) but not just that, it does differential backups to not upload everything all the time.
  - 65 3y ago
    S3 has bucket versioning if you want to have multiple backups. The S3 sync command also does differential backups; if you for example try to run the script over and over it will only upload new/different files.
    - res0nat0r 3y ago
      The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/timestamp has changed. Something like borgbackup is much faster as it uses smarter caching to skip unchanged directories etc.
      - 65 3y ago
        It's possible to find probable duplicate files with the S3 CLI based on size and tags - I was working a script to do just that but I haven't finished it yet. Alternatively if you want exact backups of your computer you can use the --delete flag which will delete files in the bucket that aren't in the source. I agree this is not the absolute most optimized solution but it does work quite well for me and is easily extendible with other scripts and S3 CLI commands. Theoretically if Borgbackup or Duplicity are backing up to S3 they're using all the same commands as the S3 CLI/SDK. Besides, shell scripting is fun!
mrich 3y ago
If you don't need incremental backups (thus saving space for the signatures) and want to store to S3 Deep Glacier, take a look at https://github.com/mrichtarsky/glacier_deep_archive_backup https://github.com/mrichtarsky/glacier_deep_archive_backup
_flux 3y ago
I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full backup again periodically, unless you want to have an ever-growing sequence of increments from the day you started doing backups. Content-addressed backups avoid that, because all snapshots are complete (even if the backup process itself is incremental), but their content blobs are shared and eventually garbage collected when no references exist to them. My tool of choice is kopia. Also borgbackup does similar things (though borgbackup is still unable to back up to the same repo from multiple hosts at the same time, though I haven't checked this for a while). Both do encryption, but its symmetric, so the client will have keys to opening the backups as well. If you require asymmetric encryption then these tools are not for you—though I guess this is not a technical requirement for this approach, so maybe one day a content-addressed backup tool with asymmetric encryption will appear?
- mizzao 3y ago
  Content-addressed backups sound something like how git stores data, is that the best way to think about them? And if so, what would be the main differences between just committing to a git repo for example?
  - KMag 3y ago
    The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication. Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide which blocks to store as diffs of others. Microsoft had some enterprise product that performed distribution somewhat like this, but also recursively using similarity hashes to see if the diffs were similar to existing files on the far machine.
dveeden2 3y ago
https://apps.gnome.org/DejaDup/ https://apps.gnome.org/DejaDup/ is using this as backend. It also has a experimental option to use https://github.com/restic/restic https://github.com/restic/restic instead of duplicity.
MallocVoidstar 3y ago
I've had issues with Duplicity not performing backups when a single backend is down (despite it being set to continue on error). In my case an SFTP server lost its IPv6 routing so Duplicity couldn't connect to it; rather than give up on that backend and only use the other server it just gave up entirely.
dash2 3y ago
[flagged]
darrmit 3y ago
I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.
- monkey26 3y ago
  I've been using Restic since 2017 without issue. Tried Kopia for a while, but its backup size ballooned on me, maybe it wasn't quite ready.
  - pdimitar 3y ago
    Kopia supports zstd compression, I found it to be within +10% to -5% of the size of my Borg repo. It also has extensive support for ignoring stuff and it works very well. I still use Borg because its policy of expiring older snapshots is more useful for me, but Kopia is extremely solid and I would use it any day if I didn't care that it doesn't actually keep one monthly backup for the last 3 months as Borg does (it decides which older snapshots to keep with another algorithm; it's documented on their website).
- tmalsburg2 3y ago
  +1 for restic. I tried various solutions and restic is the best by far. So fast, so reliable. https://restic.net/ https://restic.net/
- muppetman 3y ago
  I agree. restic with it's simple ability to mount (using fusefs) a backup so you can just copy out the file(s) you need, is so wonderful. A single binary that you just download and can SCP around the place to any device etc. It's fantastic to have so many great open source backup solutions. I investigated many and settled on restic. It still brings me joy to actually use it, it's so simple and hassle free.
- baal80spam 3y ago
  +1 for rclone. What a great piece of software.
lgunsch 3y ago
Years ago I used a very simple bash script font-end for Duplicity called Duply. It worked very well for the half-dozen years or so I used it.
tzs 3y ago
Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ https://www.duplicati.com/ [2] https://duplicacy.com/ https://duplicacy.com/
- jszymborski 3y ago
  While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner Storage Box. I've restored many times with no issue. Remember folks, test your backups. [0] https://forum.duplicati.com/t/several-days-and-no-restore-feedback/5396 https://forum.duplicati.com/t/several-days-and-no-restore-fe...
  - johnchristopher 3y ago
    > Remember folks, test your backups. Since you mention it, I am seizing the opportunity to ask: how should borg backup be tested ? Can it be automated ?
    - jszymborski 3y ago
      It's actually pretty simple using the check command [0]! borg check --verify-data REPOSITORY_OR_ARCHIVE You can add that to a cron job. Alternatively, I think the Vorta GUI also has a way to easily schedule it[1]. I'll add that one thing I like to do once in a blue-moon is to spin-up a VM and try to recover a few random files. While the check command checks that the data is there and theoretically recoverable, nothing really beats proving to yourself that you can, in a clean environment, recover your files. [0] https://borgbackup.readthedocs.io/en/stable/usage/check.html https://borgbackup.readthedocs.io/en/stable/usage/check.html [1] https://vorta.borgbase.com/ https://vorta.borgbase.com/
      - 3y ago
BeetleB 3y ago
Duply is a good frontend for Duplicity. https://duply.net/Main_Page https://duply.net/Main_Page
igtztorrero 3y ago
Try kopia.io is very good
- green-salt 3y ago
  Seconding this, its saved me several times.
deleted 3y ago
[deleted]
thedanbob 3y ago
PSA for anyone else as stupid as me: when doing selective restores be very careful about how you set target_directory. "duplicity restore --path-to-restore some_file source_url ~" does not mean "restore some_file to my home directory", it means "replace my home directory with some_file".
- kiwijamo 3y ago
  I usually make a folder specifically for restores and target anything there -- this avoids this issue.
pengaru 3y ago
Whatever you do, don't use rdiff-backup.
- sliken 3y ago
  Why? I've had no issues.
  - pengaru 3y ago
    For starters it has a tendency to paint itself into a corner on ENOSPC situations. You won't even be able to perform a restore if a backup was started but unfinished because it ran out of space. There's this process of "regressing" the repo [0] which must occur before you can do practically anything after an interrupted/failed backup. What this actually must do is undo the partial forward progress, by performing what's effectively a restore of the files that got pushed into the future relative to the rest of the repository, which requires more space for any actually modified files. Unless you have/can create free space to do these things, it can become wedged... and if it's a dedicated backup system where you've intentionally filled disks up with restore points, you can find yourself having to throw out backups just to make things functional again. That's the most obvious glaring problem, beyond that it's just kind of garbage in terms of the amount of space and time it requires to perform restores. Especially restores of files having many reverse-differential increments leading back to the desired restore point. It can require ~2X a given file's size in spare space to assemble the desired version, while it iteratively reconstructs all the intermediate versions in arriving at the desired version. Unless someone improved this since I last had to deal with it, which is possible, it's been years. Source: Ages ago I worked for a startup[1] that shipped a backup appliance originally implemented by contractors using rdiff-backup behind the scenes. Writing a replacement that didn't suck but was compatible with rdiff-backup's repos while adding newfangled stuff like transactional backups with no need for "regress", direct read-only FUSE access of restore points without needing space, and synthetic virtual-NTFS style access for booting VMs off restore points consumed several years of my life... There are far better options in 2024. [0] https://github.com/rdiff-backup/rdiff-backup/blob/master/src/rdiffbackup/locations/_repo_shadow.py#L933 https://github.com/rdiff-backup/rdiff-backup/blob/master/src... [1] https://www.crunchbase.com/organization/axcient https://www.crunchbase.com/organization/axcient

143 points by GTP 3y ago

I have been having great luck with incremental backups with the very similar named Duplicacy https://duplicacy.com/ https://duplicacy.com/

AdmiralAsshat 3y ago

Brilliant name, if you think about it. If they ever decided to start doing shady shit, they'd have a perfect legal shield. No one would be able to convincingly argue in court that they were being duplicitous.

mlyle 3y ago
If no one can argue they're duplicitous, then it's a case of false advertising...

longwave 3y ago

I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/ https://www.borgbackup.org/

giamma 3y ago
Same for me. Also, on MacOs duplicity was consuming much more CPU than Borg and was causing my fan to spin loudly. Eventually I moved to timemachine, but I still consider Borg a very good option.
zzzeek 3y ago
I have an overnight cron that flattens my duplicity backups from many incremental backups made over the course of one day to a single full backup file, that becomes the new backup. then subsequent backups over the course of the day do incremental on that file. So I always have full backups for each individual day with only a dozen or so incremental backups tacked onto it. that said will give Borg a look
tussa 3y ago
I did the same. I had some weird path issues with Duplicity. Borg is now my holy backup grail. Wish I could backup incrementally to AWS glacier storage but that just me sounding like an ungrateful begger. I'm incredibly grateful and happy with Borg!
sigio 3y ago
Agree completely... used duplicity many years ago, but switched to Borg and never looked back. Currently doing borg-backups of quite a lot of systems, many every 6 hours, and some, like my main shell-host every 2 hours. It's quick, tiny and easy... and restores are the easiest, just mount the backup, browse the snapshot, and copy files where needed.
- stavros 3y ago
  After Borg, I switched to Restic: https://restic.net/ https://restic.net/ AFAIK, the only difference is that Restic doesn't require Restic installed on the remote server, so you can efficiently backup to things like S3 or FTP. Other than that, both are fantastic.
  - pdimitar 3y ago
    Technically Borg doesn't require it either, you can backup to a local directory and then use `rclone` to upload the repo wherever. Not practical for huge backups but it works for me as I'm backing up my machines configuration and code directories only. ~60MB, and that includes a lot of code and some data (SQL, JSON et. al.)
    - stavros 3y ago
      Sure, but rsync requires the server to support rsync.
      - pdimitar 3y ago
        Does rclone require rsync? Haven't checked.
        stavros 3y ago
        Oh sorry, brain fart, I thought you said rsync. I think rclone uploads everything if you don't have rclone on the server, but I'm not sure.
shaunpud 3y ago
Works a treat with borgmatic https://torsion.org/borgmatic https://torsion.org/borgmatic
GTP 3y ago
Also duplicity let's you automatically delete backups older than a certain amount of time, what is the difference?

jwr 3y ago

Excellent piece of software, and relatively simple to use with gpg encryption. I've been using it for many years. My only complaint is that, like a lot of software written in Python, it has no regard for traditional UNIX behavior (keep quiet unless you have something meaningful to say), so I have to live with cron reporting stuff like: "/usr/lib/python2.7/dist-packages/paramiko/rsakey.py:99: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. algorithm=hashes.SHA1()" along with stuff I actually do (or might) care about. Oh well.

tynorf 3y ago
You can try setting `export PYTHONWARNINGS="ignore"` to suppress warnings.
duskwuff 3y ago
> /usr/lib/python2.7/ You are using an old version of Duplicity. It dropped all support for Python 2 in 2022: https://git.launchpad.net/duplicity/commit/setup.py?id=5505fb51a489a8306901a14d93c7aaec5dec41b2 https://git.launchpad.net/duplicity/commit/setup.py?id=5505f...

65 3y ago

If you're using S3 to back up your files, it's easier to write a shell script with the AWS CLI. For example here's a script I wrote that I run automatically to back up my computer to S3. I have an exclude array to exclude certain folders. It's simpler than downloading software and more customizable. # $1 # local folder # $2 # bucket declare -a exclude=( "node_modules" "Applications" "Public" ) args="" for item in "${exclude[@]}"; do args+=" --exclude '*/$item/*' --exclude '$item/*'"; done cmd="aws s3 sync '$1' 's3://$2$1' --include '*' $args" eval "$cmd"

rakoo 3y ago
Your script doesn't do the same thing as duplicity. Your script mirrors the local directory with your bucket. It loses all history. Duplicity does backups (ie with history) but not just that, it does differential backups to not upload everything all the time.
- 65 3y ago
  S3 has bucket versioning if you want to have multiple backups. The S3 sync command also does differential backups; if you for example try to run the script over and over it will only upload new/different files.
  - res0nat0r 3y ago
    The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/timestamp has changed. Something like borgbackup is much faster as it uses smarter caching to skip unchanged directories etc.
    - 65 3y ago
      It's possible to find probable duplicate files with the S3 CLI based on size and tags - I was working a script to do just that but I haven't finished it yet. Alternatively if you want exact backups of your computer you can use the --delete flag which will delete files in the bucket that aren't in the source. I agree this is not the absolute most optimized solution but it does work quite well for me and is easily extendible with other scripts and S3 CLI commands. Theoretically if Borgbackup or Duplicity are backing up to S3 they're using all the same commands as the S3 CLI/SDK. Besides, shell scripting is fun!

mrich 3y ago

If you don't need incremental backups (thus saving space for the signatures) and want to store to S3 Deep Glacier, take a look at https://github.com/mrichtarsky/glacier_deep_archive_backup https://github.com/mrichtarsky/glacier_deep_archive_backup

_flux 3y ago

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full backup again periodically, unless you want to have an ever-growing sequence of increments from the day you started doing backups. Content-addressed backups avoid that, because all snapshots are complete (even if the backup process itself is incremental), but their content blobs are shared and eventually garbage collected when no references exist to them. My tool of choice is kopia. Also borgbackup does similar things (though borgbackup is still unable to back up to the same repo from multiple hosts at the same time, though I haven't checked this for a while). Both do encryption, but its symmetric, so the client will have keys to opening the backups as well. If you require asymmetric encryption then these tools are not for you—though I guess this is not a technical requirement for this approach, so maybe one day a content-addressed backup tool with asymmetric encryption will appear?

mizzao 3y ago
Content-addressed backups sound something like how git stores data, is that the best way to think about them? And if so, what would be the main differences between just committing to a git repo for example?
- KMag 3y ago
  The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication. Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide which blocks to store as diffs of others. Microsoft had some enterprise product that performed distribution somewhat like this, but also recursively using similarity hashes to see if the diffs were similar to existing files on the far machine.

dveeden2 3y ago

https://apps.gnome.org/DejaDup/ https://apps.gnome.org/DejaDup/ is using this as backend. It also has a experimental option to use https://github.com/restic/restic https://github.com/restic/restic instead of duplicity.

MallocVoidstar 3y ago

I've had issues with Duplicity not performing backups when a single backend is down (despite it being set to continue on error). In my case an SFTP server lost its IPv6 routing so Duplicity couldn't connect to it; rather than give up on that backend and only use the other server it just gave up entirely.

dash2 3y ago

[flagged]

darrmit 3y ago

I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.

monkey26 3y ago
I've been using Restic since 2017 without issue. Tried Kopia for a while, but its backup size ballooned on me, maybe it wasn't quite ready.
- pdimitar 3y ago
  Kopia supports zstd compression, I found it to be within +10% to -5% of the size of my Borg repo. It also has extensive support for ignoring stuff and it works very well. I still use Borg because its policy of expiring older snapshots is more useful for me, but Kopia is extremely solid and I would use it any day if I didn't care that it doesn't actually keep one monthly backup for the last 3 months as Borg does (it decides which older snapshots to keep with another algorithm; it's documented on their website).
tmalsburg2 3y ago
+1 for restic. I tried various solutions and restic is the best by far. So fast, so reliable. https://restic.net/ https://restic.net/
muppetman 3y ago
I agree. restic with it's simple ability to mount (using fusefs) a backup so you can just copy out the file(s) you need, is so wonderful. A single binary that you just download and can SCP around the place to any device etc. It's fantastic to have so many great open source backup solutions. I investigated many and settled on restic. It still brings me joy to actually use it, it's so simple and hassle free.
baal80spam 3y ago
+1 for rclone. What a great piece of software.

lgunsch 3y ago

Years ago I used a very simple bash script font-end for Duplicity called Duply. It worked very well for the half-dozen years or so I used it.

tzs 3y ago

Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ https://www.duplicati.com/ [2] https://duplicacy.com/ https://duplicacy.com/

jszymborski 3y ago
While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner Storage Box. I've restored many times with no issue. Remember folks, test your backups. [0] https://forum.duplicati.com/t/several-days-and-no-restore-feedback/5396 https://forum.duplicati.com/t/several-days-and-no-restore-fe...
- johnchristopher 3y ago
  > Remember folks, test your backups. Since you mention it, I am seizing the opportunity to ask: how should borg backup be tested ? Can it be automated ?
  - jszymborski 3y ago
    It's actually pretty simple using the check command [0]! borg check --verify-data REPOSITORY_OR_ARCHIVE You can add that to a cron job. Alternatively, I think the Vorta GUI also has a way to easily schedule it[1]. I'll add that one thing I like to do once in a blue-moon is to spin-up a VM and try to recover a few random files. While the check command checks that the data is there and theoretically recoverable, nothing really beats proving to yourself that you can, in a clean environment, recover your files. [0] https://borgbackup.readthedocs.io/en/stable/usage/check.html https://borgbackup.readthedocs.io/en/stable/usage/check.html [1] https://vorta.borgbase.com/ https://vorta.borgbase.com/
    - 3y ago

BeetleB 3y ago

Duply is a good frontend for Duplicity. https://duply.net/Main_Page https://duply.net/Main_Page

igtztorrero 3y ago

Try kopia.io is very good

green-salt 3y ago
Seconding this, its saved me several times.

deleted 3y ago

[deleted]

thedanbob 3y ago

PSA for anyone else as stupid as me: when doing selective restores be very careful about how you set target_directory. "duplicity restore --path-to-restore some_file source_url ~" does not mean "restore some_file to my home directory", it means "replace my home directory with some_file".

kiwijamo 3y ago
I usually make a folder specifically for restores and target anything there -- this avoids this issue.

pengaru 3y ago

Whatever you do, don't use rdiff-backup.

sliken 3y ago
Why? I've had no issues.
- pengaru 3y ago
  For starters it has a tendency to paint itself into a corner on ENOSPC situations. You won't even be able to perform a restore if a backup was started but unfinished because it ran out of space. There's this process of "regressing" the repo [0] which must occur before you can do practically anything after an interrupted/failed backup. What this actually must do is undo the partial forward progress, by performing what's effectively a restore of the files that got pushed into the future relative to the rest of the repository, which requires more space for any actually modified files. Unless you have/can create free space to do these things, it can become wedged... and if it's a dedicated backup system where you've intentionally filled disks up with restore points, you can find yourself having to throw out backups just to make things functional again. That's the most obvious glaring problem, beyond that it's just kind of garbage in terms of the amount of space and time it requires to perform restores. Especially restores of files having many reverse-differential increments leading back to the desired restore point. It can require ~2X a given file's size in spare space to assemble the desired version, while it iteratively reconstructs all the intermediate versions in arriving at the desired version. Unless someone improved this since I last had to deal with it, which is possible, it's been years. Source: Ages ago I worked for a startup[1] that shipped a backup appliance originally implemented by contractors using rdiff-backup behind the scenes. Writing a replacement that didn't suck but was compatible with rdiff-backup's repos while adding newfangled stuff like transactional backups with no need for "regress", direct read-only FUSE access of restore points without needing space, and synthetic virtual-NTFS style access for booting VMs off restore points consumed several years of my life... There are far better options in 2024. [0] https://github.com/rdiff-backup/rdiff-backup/blob/master/src/rdiffbackup/locations/_repo_shadow.py#L933 https://github.com/rdiff-backup/rdiff-backup/blob/master/src... [1] https://www.crunchbase.com/organization/axcient https://www.crunchbase.com/organization/axcient